Skip to content

Data & Model Monitoring (Evidently)

Status: ✅ Implemented — feature drift detection pipeline, Airflow DAG, and REST endpoint are operational.

Feature drift detection runs via: - DVC stage (monitor_drift): src/pipelines/monitor_drift.py — runs Evidently against training reference and writes reports/drift/latest.json - Airflow DAG: airflow/dags/ml_monitor_drift_01.py — scheduled batch execution - REST endpoint: GET /monitoring/drift — reads reports/drift/latest.json and refreshes the model_feature_drift_score Prometheus gauge

Prediction drift and model performance monitoring require a ground truth feedback loop — see below.

Intended usage

Evidently will be used to: - detect feature distribution drift against the training baseline, - monitor prediction distribution changes over time, - generate offline batch reports as a first step.

Planned monitored signals

Data drift

  • Feature distribution shift (PSI / KS test) against training reference.
  • Missing or unexpected values.
  • Schema changes.

These connect to the data pipeline in src/data/ and the feature engineering layer in src/features/.

Prediction drift

  • Output probability distribution shift.
  • Class balance changes.

Ground truth evaluation

  • Match results arrive ~90 minutes after kick-off.
  • Full accuracy evaluation requires a ground truth feedback loop — planned separately from drift detection.

Offline-first plan

  1. Batch Evidently report on recent predictions vs training baseline (initial phase).
  2. Export key drift signals as Prometheus metrics (future phase).
  3. AlertManager rule on confirmed drift signal (future phase).