Past Events

The system is organized into five formal layers, each with explicit responsibilities and codebase boundaries:

Data Layer:

TimescaleDB (docker-compose.yml: db) stores all persistent data: products, price history, external signals, features, forecasts, alerts, and explanations. The database schema is defined and managed in backend/app/models/entities.py and backend/app/db/init_db.py, with support for both PostgreSQL/TimescaleDB and SQLite for development and testing.
SQLAlchemy ORM (backend/app/models/entities.py) defines all data models, with relationships and constraints, including Product, PriceHistory, ExternalSignal, FeatureRecord, Forecast, Alert, and related entities. The schema is initialized and updated via backend/app/db/init_db.py, which also enables TimescaleDB hypertables for efficient time-series storage.

Processing Layer:

Data ingestion (backend/app/services/ingestion/service.py, providers.py) fetches and validates raw data from the DCS dashboard (for retail prices), FRED (for global economic indicators), Open-Meteo (for weather), and GDELT (for news sentiment). The ingestion logic is robust, with strict validation and error handling (see DataValidationError in backend/app/services/ingestion/errors.py).
Data processing (backend/app/services/processing/) cleans and normalizes raw data. While the processing logic is distributed across the pipeline, key normalization and validation steps are enforced in the orchestrator and feature engineering modules.
Feature engineering (backend/app/services/features/engineering.py) generates lagged, rolling, and macro signal features for each product-region series, using pandas groupby operations. The target variable (next-period price) is also computed here.