Skip to content

Common Failures & Troubleshooting

This runbook lists frequent failure scenarios and recommended investigation steps. Severity and recovery details are aligned with Failure Modes.

P1 — Full outage

RabbitMQ unavailable

Symptoms: all inference fails with 500/503; Celery workers lose connection.

Actions 1. Check pod status: kubectl get pods -n soccer-serving 2. Restart RabbitMQ pod: kubectl rollout restart deployment/rabbitmq -n soccer-serving 3. Celery workers reconnect automatically after broker recovery.

FastAPI pod crash

Symptoms: API endpoints unreachable; K8s readiness probe fails.

Actions 1. Check pod events: kubectl describe pod -l app=soccer-api -n soccer-serving 2. K8s restarts the pod automatically via restart policy. 3. If in CrashLoopBackOff — inspect logs: kubectl logs -l app=soccer-api -n soccer-serving --previous

P2 — Degraded serving

High latency on /predict

Symptoms: p99 latency well above 200 ms; occasional 504 timeouts.

Possible causes - worker queue backlog (too few workers), - worker cold start (freshly scaled pod loading the model), - Redis unavailable (all requests go to cache-miss path).

Actions 1. Check Celery queue depth: curl http://api.soccer.dmitryivanov.dev/monitoring/celery/queues 2. Check active workers: curl http://api.soccer.dmitryivanov.dev/monitoring/celery/workers 3. Scale workers if needed: kubectl scale deployment/celery-worker-ml --replicas=N -n soccer-serving 4. Inspect worker logs for model load errors.

Async queue backlog growing

Symptoms: task queue depth rising; async jobs not completing.

Possible causes - insufficient workers, - stuck or failing tasks (check DLQ).

Actions 1. Inspect worker logs: kubectl logs -l app=celery-worker-ml -n soccer-serving 2. Scale workers: kubectl scale deployment/celery-worker-ml --replicas=N -n soccer-serving 3. Check retry count in Celery task output; max retries = 2.

Redis unavailable

Symptoms: cache: false in all predict responses; latency increase (cache-miss path always taken); Redis errors in logs.

Actions 1. Check pod: kubectl get pods -l app=redis -n soccer-serving 2. Restart: kubectl rollout restart deployment/redis -n soccer-serving 3. Cache warms automatically on subsequent requests; no data loss.

Inference continues in degraded mode without Redis — this is not a full outage.

Stale cache after model promotion

Symptoms: predictions still use the old champion model version for up to TTL seconds after promotion.

Actions 1. If immediate consistency is required, flush Redis: kubectl exec -it <redis-pod> -n soccer-serving -- redis-cli FLUSHDB 2. Re-trigger inference for active matches. 3. Otherwise, stale entries expire automatically within the configured TTL.

MLflow unavailable

Symptoms: new Celery worker processes fail to load model on startup; existing in-flight workers unaffected.

Actions 1. Check pod: kubectl get pods -l app=mlflow -n soccer-mlflow 2. Restart: kubectl rollout restart deployment/mlflow -n soccer-mlflow 3. Workers reload the model on the next request after MLflow is back. 4. Do not promote new models while MLflow is unavailable.

P3 — Data pipeline blocked

WhoScored source unavailable

Symptoms: Airflow DAG marked failed; Celery scraping task fails with connection/HTTP error.

Actions 1. Check Airflow task logs at airflow.dmitryivanov.dev. 2. Retry via Airflow backfill once source recovers: airflow dags backfill -s <date> -e <date> scrape_livescores 3. Check for WhoScored layout changes if the error is a parse failure.

Scraper broken (layout change)

Symptoms: Celery task fails with parse/selector error; data gap visible in PostgreSQL.

Actions 1. Inspect scraping task logs in Airflow. 2. Update CSS selectors in src/data/scraper/driver.py. 3. Trigger manual backfill to fill the data gap.

Selenoid host unreachable

Symptoms: Celery task fails on browser session init; log shows connection refused to Selenoid host.

Actions 1. Verify Selenoid host is running (it is external to K8s — check the VPS directly). 2. Restart Selenoid service manually on the host. 3. Re-trigger the Airflow DAG task.

Raw export failed (PostgreSQL → MinIO)

Symptoms: DVC stage load_data_from_sources fails; no new parquet in MinIO.

Actions 1. Check DB connectivity: verify PostgreSQL pod is healthy. 2. Check MinIO credentials: kubectl get secret soccer-minio-secret -n soccer-data -o yaml 3. Re-run the stage: dvc repro load_data_from_sources

P4 — Offline pipeline blocked

Data contract failure (Great Expectations)

Symptoms: DVC stage validate_raw / validate_finished / validate_features exits non-zero; pipeline blocked.

Actions 1. Inspect the GE output in the DVC stage log. 2. If the change is intentional (upstream schema evolved): update the GE suite and document the change. 3. If unintentional: investigate the data source; do not bypass the gate. 4. Re-run: dvc repro validate_raw (or the appropriate gate stage).

DVC stage failure (any stage)

Symptoms: dvc repro exits non-zero; stage shown as failed.

Actions 1. Inspect the failed stage log: dvc repro <stage> 2>&1 | tail -50 2. Fix root cause (code, data, config). 3. Re-run: dvc repro

Model regression (metric drop)

Symptoms: MLflow comparison in classification_models stage shows metric below threshold.

Actions 1. Review MLflow run in mlflow.dmitryivanov.dev. 2. Do not promote to champion if metrics are below the acceptance threshold. 3. Roll back if already promoted: update MLflow alias back to the previous run. 4. Re-examine training data for quality issues or distribution shift.

Deployment / infrastructure

Secrets injection failure (SOPS / K8s)

Symptoms: pod fails to start; env vars missing; pod events show secret not found.

Actions 1. Re-run SOPS decryption in CI: make decrypt 2. Re-apply the K8s secret manifest: kubectl apply -f k8s/secrets/ 3. Restart affected pods.

Bad deployment config (Helm values error)

Symptoms: Helm upgrade fails in CI; pods in CrashLoopBackOff.

Actions 1. Roll back: helm rollback soccer-api -n soccer-serving 2. Fix the Helm values file. 3. Re-run CI pipeline.