Highlights
What we build
A production DriftDetector that monitors live inference distributions against a reference baseline using KS tests, Population Stability Index (PSI), and Jensen-Shannon divergence — all computed incrementally without reloading history.
A RetrainOrchestrator that wakes up on drift events, clones the latest labeled dataset snapshot, fine-tunes a Gemini-backed agent, and writes a versioned artifact to disk.
A Shadow Evaluation Gate that runs the candidate model in parallel with the champion, accumulates a held-out evaluation score, and auto-promotes only if the score delta clears a configurable threshold.
An AdaptiveAgentRouter that hot-swaps the champion model in the L65 FastAPI serving layer — zero downtime, no Kubernetes rollout needed.
A React CT Dashboard with live WebSocket feeds showing drift scores, retraining state-machine status, model lineage, and A/B traffic allocation.
Connection to L65: We plug directly into the inference server built in L65. The same Prometheus middleware now emits feature histograms. The batching queue feeds our reference window. The model registry abstraction gets a promote() method.
Enables L67: Every retrain snapshot creates a timestamped dataset artifact under artifacts/datasets/. L67’s DVC layer will version-control exactly these snapshots and the feature store will consume the same schema.
Architecture Context
Place in the 90-Lesson VAIA Path
Lessons 61–65 delivered the serve fast half of MLOps: optimized inference, caching, batching, and load testing. L66 closes the loop — the learn fast half. Without CT, a VAIA becomes stale the moment its training distribution diverges from production reality. Regulatory environments (finance, healthcare) explicitly require demonstrable model monitoring. This lesson is the pivot from static deployment to living systems.
Integration with L65 Components
L65 InferenceServer
├── /predict endpoint ──────────────► DriftDetector.observe(features, output)
├── Prometheus metrics ─────────────► DriftDashboard (existing Grafana)
└── ModelRegistry.get_champion() ◄──── AdaptiveAgentRouter.promote(version)
The ModelRegistry from L65 stored a single champion. We extend it with push_candidate(), score_candidate(), and promote_candidate() — three new methods that slot cleanly into the existing abstraction.
Module 5 Objectives Alignment
Preparing for a distributed systems interview?
→Download the free Interview Pack
→ Subscribe now to access source code repository - 200 + coding lessons
