Santhe — Tomato Price Prediction Pipeline

District-wise daily tomato price and arrival forecasting for Indian markets, with a focus on Karnataka. Combines 10 data sources into 150+ ML features, trains LightGBM/XGBoost/Random Forest models, and generates multi-horizon price forecasts with confidence intervals.

---

What It Does

10 Data Sources ──> SQLite DB ──> 150+ Features ──> 3 Model Types ──> Price Forecasts
   (scrapers)       (database)     (feature eng.)    (walk-forward CV)   (1/7/14/30 day)

Data Type	Source	Frequency	API Key?
Mandi prices + arrivals	agmarknet.gov.in	Daily	No
Weather + soil moisture	Open-Meteo	Daily	No
Diesel/fuel prices	goodreturns.in / data.gov.in	Daily	No
INR/USD exchange rate	frankfurter.app	Daily	No
Vegetation health (NDVI proxy)	NASA POWER	Daily	No
Reservoir water levels	data.gov.in / seasonal model	Weekly	No
CPI (vegetables/food)	data.gov.in / MOSPI reference	Monthly	Yes*
WPI (wholesale prices)	data.gov.in / OEA reference	Monthly	Yes*
Fertilizer prices	Government gazette (config)	Quarterly	No
Trade policy events	DGFT notifications (config)	Event-based	No

*data.gov.in key is already configured. All other sources are completely free with no keys required.

---

Quick Start

1. Install

cd santhe-scraper
pip install -r requirements.txt

Dependencies: requests, pandas, numpy, beautifulsoup4, lightgbm, xgboost, scikit-learn, python-dateutil

2. Run the Full Pipeline

# Everything: scrape all data -> engineer features -> train models -> forecast
python3 run_pipeline.py --threads 4

# This will take a while on first run (fetching ~4 years of data).
# Subsequent runs are incremental — duplicates are skipped automatically.

3. Check Results

# Forecasts are printed to console and saved to:
cat output/forecasts.csv

# Database stats
python3 main.py --stats

# Evaluate model accuracy
python3 model_trainer.py --evaluate

Common Workflows

# Already have data? Just retrain + forecast
python3 run_pipeline.py --skip-scrape --threads 4

# Already trained? Just forecast with saved models
python3 run_pipeline.py --skip-scrape --skip-train

# Quick test: LightGBM only, skip slow NDVI fetch
python3 run_pipeline.py --skip-ndvi --models lightgbm --no-quantile

# Forecast specific regions
python3 run_pipeline.py --skip-scrape --regions Kolar Nashik Madanapalle

# Predict for a single district
python3 model_trainer.py --predict --district Kolar

# Scrape only specific data
python3 main.py --additional-data    # Macro data only
python3 main.py --weather-only --threads 4
python3 main.py --agmarknet --threads 4

---

Project Structure

santhe-scraper/
|
|-- run_pipeline.py              # One-command full pipeline (recommended entry point)
|-- main.py                      # CLI with granular control over individual steps
|-- model_trainer.py             # ML training, evaluation, prediction
|-- feature_engineer.py          # 150+ feature creation from raw data
|-- database.py                  # SQLite storage (11 tables, upsert/dedup)
|-- config.py                    # All settings, GPS coords, API keys, constants
|
|-- Scrapers:
|   |-- scraper_agmarknet_gov.py # Price + arrival (agmarknet.gov.in API)
|   |-- scraper_weather.py       # Weather + soil (Open-Meteo, free)
|   |-- scraper_diesel.py        # Diesel/fuel prices
|   |-- scraper_forex.py         # INR/USD exchange rate (frankfurter.app)
|   |-- scraper_ndvi.py          # Vegetation health (NASA POWER)
|   |-- scraper_reservoir.py     # Dam water levels (Karnataka)
|   |-- scraper_cpi.py           # Consumer Price Index
|   |-- scraper_wpi.py           # Wholesale Price Index
|   |-- scraper_fertilizer.py    # Fertilizer prices (config-driven)
|   |-- scraper_trade_policy.py  # Export/import bans (config-driven)
|   |-- scraper_agmarknet.py     # data.gov.in API (legacy)
|   |-- scraper_ceda.py          # CEDA Ashoka API (legacy)
|
|-- output/
|   |-- santhe_tomato.db         # SQLite database (all data)
|   |-- models/                  # Trained model pickles + metadata
|   |-- forecasts.csv            # Latest predictions
|   |-- tomato_prices_daily.csv  # Exported price data
|   |-- weather_daily.csv        # Exported weather data
|   |-- tomato_model_features.csv
|
|-- docs/
|   |-- DATA_SOURCES.md          # Detailed scraper documentation
|   |-- PIPELINE.md              # Feature engineering + model training details

---

CLI Reference

run_pipeline.py (Recommended)

Runs the complete pipeline in one command.

Flag	Description
--threads N	Parallel scraping threads (default: 1, max: 8)
--start-date	Start date YYYY-MM-DD (default: 2021-02-01)
--end-date	End date YYYY-MM-DD (default: today)
--regions X Y Z	Districts to forecast (default: Kolar, Chikkaballapur, Bangalore, Belagavi, Nashik)
--models	Model types: lightgbm, xgboost, random_forest (default: all)
--targets	Prediction targets: price, arrival (default: both)
--skip-scrape	Skip data scraping, use existing DB
--skip-train	Skip training, use saved models
--skip-ndvi	Skip NASA POWER fetch (slow)
--no-quantile	Skip quantile models (faster training)
--min-records N	Min records per district to include (default: 60)
--db PATH	Database path (default: output/santhe_tomato.db)
-v	Verbose debug logging

main.py (Granular Control)

Individual pipeline steps with mutually exclusive modes.

Mode	What It Does
--full	All states prices + weather + macro data + features
--karnataka	Karnataka + cross-state prices + weather + macro
--agmarknet	agmarknet.gov.in district-wise daily (Feb 2021+)
--prices-only	Prices for all 18 states
--weather-only	Weather for all configured districts
--weather-from-prices	Weather for all 400+ districts with price data
--additional-data	All macro data (diesel, CPI, WPI, NDVI, forex, reservoir)
--diesel-only	Diesel prices only
--macro-only	CPI + WPI + Forex + trade policy + fertilizer
--ndvi-only	NASA POWER vegetation data
--merge	Merge prices + weather, create feature CSV
--train-model	Train ML models
--export	Export DB to CSV files
--stats	Show database statistics
--state X	Single state prices

model_trainer.py

Mode	What It Does
--train	Full training with walk-forward CV
--evaluate	Load saved models, print evaluation report
--predict --district X	Generate predictions for a district
--models X	Which model types to train
--targets X	Which targets (price/arrival)
--no-quantile	Skip prediction intervals
--date YYYY-MM-DD	Specific date for prediction

---

Coverage

Karnataka — 17 Districts

District	Production Share	Importance
Kolar	28%	Primary — Karnataka's largest tomato hub
Chikkaballapur	15%	Primary — Chintamani market
Bangalore Rural	10%	Primary — Doddaballapura
Belagavi	10%	Primary — North Karnataka hub
Ramanagara	8%	Primary
Haveri, Mandya, Chitradurga, Chikkamagaluru, Tumakuru, Hassan, Shimoga, Chamarajanagar, Ballari, Kalaburagi	—	Secondary
Bangalore, Mysuru	—	Consumption centers

Cross-State Markets — 8 Locations

Market	State	Why It Matters
Nashik	Maharashtra	Asia's #1 tomato market — inverse season with Kolar
Madanapalle	Andhra Pradesh	Supplies alongside Kolar in Jun-Aug
Pune, Narayangaon	Maharashtra	Nashik supply belt
Azadpur (Delhi)	Delhi	India's largest mandi — national price setter
Koyambedu (Chennai)	Tamil Nadu	South India consumption hub
Solan	Himachal Pradesh	Summer supply to North India
Shivpuri	Madhya Pradesh	Central India supply belt

---

Database

SQLite database at output/santhe_tomato.db with 11 tables:

Table	Key Columns	Unique On
tomato_prices	date, state, district, market, modal_price, arrival_qty	date + state + district + market + variety
weather_daily	date, district, temperature, precipitation, soil_moisture	date + district + location
diesel_prices	date, city, diesel_price, petrol_price	date + city
forex_data	date, inr_usd_rate	date
cpi_data	date, cpi_vegetables, cpi_food, cpi_general	date
wpi_data	date, wpi_vegetables, wpi_tomato, wpi_food	date
ndvi_data	date, location, root_zone_wetness, solar_radiation	date + location
reservoir_data	date, dam_name, storage_mcft, capacity_pct	date + dam_name
fertilizer_prices	date, urea_price, dap_price, mop_price	date
trade_policy	start_date, end_date, policy_type, description	start_date + policy_type
scrape_log	scrape_type, target, records_fetched, status	(audit log)

All tables use INSERT OR IGNORE — running scrapers again skips existing records automatically.

---

Incremental Updates

# Daily cron job — fetch last 7 days to catch delayed reports
python3 run_pipeline.py --start-date $(date -d '7 days ago' +%Y-%m-%d) --skip-train

# Weekly retrain with latest data
python3 run_pipeline.py --skip-scrape

---

Troubleshooting

Issue	Fix
ModuleNotFoundError	Run pip install -r requirements.txt
No price data fetched	agmarknet.gov.in may be temporarily down; retry later
Weather API rate limit	Reduce --threads or wait; scraper auto-retries
NDVI fetch is very slow	Use --skip-ndvi; NASA POWER API has rate limits
Model training fails	Ensure sufficient data: at least 60 records per district
Prediction returns empty	District name must match DB exactly (case-sensitive)
PerformanceWarning from pandas	Harmless — just a warning about DataFrame fragmentation

---

Further Reading

• docs/DATA_SOURCES.md — Detailed documentation for all 10 data sources and scrapers
• docs/PIPELINE.md — Feature engineering, model training, and prediction deep-dive

Data Source References

• Agmarknet: https://agmarknet.gov.in
• data.gov.in: https://data.gov.in
• Open-Meteo: https://open-meteo.com/en/docs
• NASA POWER: https://power.larc.nasa.gov
• Frankfurter (forex): https://www.frankfurter.app
• MOSPI (CPI): https://mospi.gov.in
• OEA (WPI): https://eaindustry.nic.in