Agri Yield Pipeline

End-to-end system for ingesting, processing, visualizing, and modeling agricultural yield, NDVI, and weather data in Missouri.

Table of Contents

1. Prerequisites
2. Clone & Setup
3. Environment Configuration
4. Docker Services
5. Real-Time Data Ingestion
6. Stream Processing
7. Dashboard Access
8. API Endpoints
9. Troubleshooting Tips

Prerequisites

• Git (for cloning the repo)
• Docker & Docker Compose (v3.8+)
• Python 3.9+ (for running local scripts)
• Google Earth Engine CLI (earthengine)

Clone & Setup

bash git clone https://github.com/aurascoper/agri_yield_pipeline.git cd agri_yield_pipeline

Environment Configuration

Create a .env file in the project root with the following variables: ```dotenv NOAAAPITOKEN= USDAAPIKEY= INFLUXDBURL=http://localhost:8086 INFLUXDBTOKEN= INFLUXDBORG=<yourorg> INFLUXDBBUCKET=<yourbucket> POSTGRESUSER=user POSTGRESPASSWORD=password POSTGRESDB=alerts INFLUXDBINITUSERNAME=admin INFLUXDBINIT_PASSWORD=password

(Optional) Kafka settings:

KAFKABOOTSTRAPSERVERS=localhost:9092 KAFKAGROUPID=processor-group

(Optional) Redis URL:

REDIS_URL=redis://localhost:6379/0 Authenticate with Google Earth Engine: bash earthengine authenticate ```

Docker Services

Start core services: bash docker-compose up -d Services launched: - Zookeeper & Kafka - Redis - PostgreSQL - InfluxDB (initialized with your .env settings) - Stream Processor - Dash Dashboard

Check status and logs: bash docker-compose ps docker-compose logs -f

Real-Time Data Ingestion

Fetch NOAA weather and USDA yield data into InfluxDB and Kafka: bash python src/data_ingestion/live_ingestor.py \ --start-date 2021-01-01 \ --end-date 2021-12-31 \ --station-id GHCND:USW00003952 \ --year 2021

Stream Processing

Consumes raw Kafka topics (weather, yield), enriches data, and writes to: - Kafka output topic (enriched-yield) - InfluxDB (for dashboard queries) - Redis (for alerts cache)

Run via Docker Compose (already started above): bash docker-compose up -d stream-processor Or locally: bash python src/processing/stream_processor.py

Dashboard Access

The live Dash dashboard is available at: http://localhost:8050

To run locally: bash python src/visualization/live_dashboard.py

API Endpoints

Backend API with FastAPI (NDVI & weather): bash uvicorn api.main:app --reload - GET /ndvi/ - GET /weather/

Troubleshooting Tips

• Ensure .env is correctly configured and contains all required variables.
  
• Verify no port conflicts on 5432, 6379, 8086, 8050, and 9092.
  
• Use docker-compose ps and docker-compose logs <service> for diagnostics.
  
• Access InfluxDB UI at http://localhost:8086 (use credentials from .env).
  
• Test Redis with redis-cli -u redis://localhost:6379/0.
  
• Confirm Google Earth Engine authentication: earthengine authenticate.