Most AI agent tutorials stop at “it responds to a prompt.” This course starts where they stop.
You will engineer a system that blocks prompt injection in under 5ms, compresses context by 60% before the LLM call, detects its own quality regressions and improves automatically, scales on Kubernetes from zero to ten replicas without dropping a request, passes a SOC 2 audit, and handles a regional cloud outage in under 30 minutes — without human intervention.
The five agents you ship in the capstone are portfolio-ready. They demonstrate the layer of AI engineering that enterprise teams actually pay for — and that no LangChain tutorial has ever covered.
30 Lessons. 4 Modules. 5 Production Agents. Starts August 4, 2026.
Explore Lessons 1–7: Secure Agent Foundations. Continue your journey through the complete Production AI Engineering.
What You Build
Throughout this course, you'll build five production-ready AI agents that gradually evolve into a complete enterprise AI platform. You'll engineer secure agent architectures, multi-agent collaboration, real-time streaming, self-improving LLMOps pipelines, and Kubernetes-native deployments with enterprise security, compliance, disaster recovery, and FinOps.
Every project solves a real production challenge using the same architectural patterns found in enterprise AI systems. By the end, you'll have a portfolio that demonstrates not just how to use AI—but how to engineer AI systems that organizations can confidently deploy at scale.
Agent 1 — Secure Customer Service Agent
Every Week 1 control in one request: injection blocked at Layer 4 in <5ms, RBAC enforced at Layer 3, semantic cache at Layer 2 reducing token cost by 40%, output scanned for API keys and PII before delivery.
Agent 2 — Multi-Agent Research System
SupervisorAgent fans out to three parallel WorkerAgent instances via asyncio.gather(). One worker failure does not cancel the others. Total latency = slowest single worker, not the sum.
Agent 3 — Streaming Analytics Agent
StreamingLLM → SSE endpoint → EventSource client. First token under 500ms. The X-Accel-Buffering: no header that makes nginx stop buffering your stream.
Agent 4 — Self-Healing LLMOps Agent
EvalHarness blocks deploys on regression. ABRouter routes 10% of live traffic to a new prompt variant. CircuitBreaker trips after 3 failures and rejects requests in microseconds. PromptOptimiser runs weekly and commits winners to version control.
Agent 5 — Enterprise Kubernetes Deployment
KEDA scales on agent_request_queue_depth, not CPU — before latency degrades, not after. SOC 2 audit trail with chain-hash tamper detection. HIPAA PHI detector blocks protected data at the perimeter. Route 53 failover completes in under 30 minutes. FinOps allocator bills teams for exactly what they consumed.
Who This Course Is For
Backend engineers who have built agents that work in demos and want to understand why they fail at scale.
Platform engineers who need to deploy LLM workloads on Kubernetes with the same operational rigour as any other production service.
Tech leads who own the “we need SOC 2 compliance for our AI system” conversation and need to know what to build.
Founding engineers at AI startups where the next customer due diligence will ask about security architecture, compliance controls, and disaster recovery.
What Makes This Different
No frameworks until you understand the layer below. You implement the security perimeter before touching any orchestration library. You implement the token bucket before reading about rate-limiting middleware.
Real numbers, not analogies. 60% token reduction from context compression. 15 percentage point accuracy gain from multi-agent debate. 67% infrastructure cost reduction from KEDA scale-to-zero. Every number comes from the lesson that demonstrates it.
Failure first. Every module ends with a “what breaks without this” section. You learn the failure mode the lesson prevents before you learn how it prevents it.
Production checklist on every lesson. Not “nice to have” suggestions — the specific checks that get caught in SOC 2 audits, load tests, and security reviews.
Learning Objectives
By Phase 1: You can articulate the four-layer agent security model, explain why tool calls require a five-gate validation pipeline, and deploy a secure agent base with 8-signal observability.
By Phase 2: You can implement parallel multi-agent dispatch, achieve first-token-under-500ms streaming, build layered error recovery with a dead-letter queue, and reduce token costs by 60% through context compression.
By Phase 3: You can build a CI quality gate that blocks deployments on regression, run statistically-valid A/B experiments on live traffic, implement a self-healing circuit breaker, and close the LLMOps feedback loop automatically.
By Phase 4: You can deploy an AI agent on Kubernetes with zero-downtime rolling updates, queue-depth auto-scaling, SOC 2 audit controls, HIPAA compliance safeguards, multi-region disaster recovery, and FinOps cost allocation.
Key Topics
Security Architecture: 4-layer enforcement model, prompt injection detection (regex + structural analysis), RBAC with role inheritance, output scanning for secrets and PHI, sandboxed subprocess execution with import allowlist.
Memory & Cost: Conversation buffer, semantic similarity cache (55% hit rate → 40% cost reduction), vector store for cross-session recall, four-strategy context compressor (60% token reduction), real-time Prometheus cost dashboard.
Reliability Engineering: Token bucket rate limiting, four-class error taxonomy, exponential backoff with jitter, model fallback chain, dead-letter queue, three-state circuit breaker, SLOs with error budget tracking and automated runbooks.
LLMOps: Evaluation harness with CI gate, A/B testing with deterministic routing and statistical significance, prompt optimisation with full audit trail, multi-agent debate (+15pp accuracy), knowledge graph for compliance-queryable relationships, full continuous improvement pipeline.
Enterprise Infrastructure: Kubernetes Deployment + HPA + KEDA (queue-depth scaling, scale-to-zero), SOC 2 immutable audit trail with chain-hash tamper detection, HIPAA PHI detection and 15-minute auto-logoff, multi-region disaster recovery (RTO < 30 min, RPO < 5 min), FinOps cost allocation with team showback and 80%-threshold budget alerts.
Prerequisites
Python 3.11+
Comfortable reading and writing async Python (
asyncio,await)Basic Docker knowledge (build an image, run a container)
No prior AI/agent experience required — but familiarity with API calls helps
OpenAI API key optional — every lesson runs in stub mode without one
Course Structure
Phase 1 — Secure Agent Foundations
Lessons 1–7
The architecture that every production agent runs on. By Lesson 7 you have a complete, hardened agent base — security, tools, memory, rate limiting, observability, and sandboxed execution. This is the foundation Phases 2–4 build on without modification.
Lesson 1 — The 4-Layer Secure Agent Architecture
Every production agent enforces the same request path: Security Perimeter → Tool Orchestrator → Memory Manager → Core LLM. A request that fails Layer 4 never reaches Layer 1. You implement all four layers — including stubs for the ones Lessons 2–7 will fill — so the architecture is correct from Day 1. Core types: SecurityContext, AgentRequest, AgentResponse
Lesson 2 — Tool Execution & Validation
Five gates, in order: registry lookup, Pydantic schema validation, RBAC permission check, sandboxed subprocess execution, output filtering. Miss gate 3 and an unpermissioned caller reaches your database tool. Miss gate 5 and your agent returns API keys in its responses. Core classes: ToolDefinition, ToolOrchestrator
Lesson 3 — Memory Systems — Short-Term, Semantic Cache & Long-Term
Three layers with different jobs: ConversationBuffer keeps the last N turns. SemanticCache returns cached responses to near-identical queries — 55% hit rate in production means 40% cost reduction without touching a single prompt. LongTermMemory enables cross-session recall via a vector store interface. Core classes: ConversationBuffer, SemanticCache, LongTermMemory
Lesson 4 — Rate Limiting & Cost Control
The token bucket algorithm: capacity sets the burst ceiling, refill_rate sets the sustained limit. A request with a projected cost above PER_REQUEST_CAP is rejected before the LLM call. A user who crosses DAILY_BUDGET_USD is blocked until midnight. This is what prevents the $40,000 API bill. Core classes: Bucket, RateLimiter, CostTracker
Lesson 5 — Security Hardening — Prompt Injection, RBAC & Output Guardrails PromptGuard runs three layers: regex signatures, structural analysis, anomaly scoring. RBAC resolves role inheritance so admin implies power implies user implies guest. OutputGuard scans every response for API key patterns, SSNs, card numbers, and filesystem paths before the response leaves the agent. Core classes: PromptGuard, RBAC, OutputGuard
Lesson 6 — Observability — Metrics, Structured Logs & Distributed Traces
Eight production signals: four golden (traffic, latency, errors, saturation) and four AI-specific (cost per request, cache hit rate, security rejection rate, tool execution latency). Every layer boundary emits a structured JSON log line with request_id. Four Grafana alert rules. One Telemetry class that wires all three pillars. Core class: Telemetry
Lesson 7 — Sandboxed Tool Execution — Isolated, Resource-Capped & Audited SandboxExecutor spawns a subprocess, enforces RLIMIT_AS memory cap, kills on timeout, applies an import allowlist, and rejects any non-JSON output. SandboxAuditLog writes a tamper-evident record of every execution — args are hashed, never stored raw. An open import list is a filesystem deletion waiting to happen. Core classes: SandboxPolicy, SandboxExecutor, SandboxAuditLog
Phase 2 — Production Integration
Lessons 8–14
Patterns that make the Phase 1 foundation scale: parallel agents, streaming, layered error recovery, token compression, and production monitoring. By Lesson 14 the agent handles real concurrent load, streams output to the browser, recovers from failures without human intervention, and has a real-time dashboard showing spend per user per minute.
Lesson 8 — Multi-Agent Orchestration — The Supervisor Pattern SupervisorAgent dispatches to N WorkerAgent instances via asyncio.gather(). Total wall-clock time = slowest worker, not the sum. A failed worker returns a WorkerResult with success=False — the supervisor logs it, excludes it from the response, and returns the rest. One worker going down does not take the request down. Core classes: WorkerTask, WorkerResult, WorkerAgent, SupervisorAgent
Lesson 9 — Async Tool Pipelines — Fan-Out / Fan-In
Three sequential tool calls at 1,500ms become three parallel calls at 500ms — a 3× latency reduction with no change to the output. asyncio.gather(return_exceptions=True) ensures one failed tool never cancels the others. plan_tool_calls() groups dependent tools into sequential batches and independent tools into parallel batches. Core classes: AsyncToolOrchestrator, ToolPlan
Lesson 10 — Streaming Responses — SSE Token Pipeline
StreamingLLM calls OpenAI with stream=True and yields tokens as they arrive. The FastAPI route flushes every 5 tokens as an SSE data: frame — single-token flushes are noisy, 20-token flushes feel laggy. Two response headers prevent nginx from buffering the stream. First token under 500ms. Core class: StreamingLLM
Lesson 11 — Error Recovery — Retry, Fallback & Dead-Letter Queue
classify() maps any exception to one of four classes: TRANSIENT → retry with exponential backoff + jitter. MODEL → FallbackLLM tries the next model in the chain. TOOL → degrade gracefully. FATAL → DeadLetterQueue + alert. The jitter term is not optional: without it, all agents that hit a rate limit simultaneously retry in lockstep and re-hit it on every wave. Core classes: ErrorClass, FallbackLLM, DeadLetterQueue
Lesson 12 — Context Compression — 60% Token Reduction
Four strategies, applied cheapest-first: strip tool call metadata → remove near-duplicates → truncate system prompt to 800 chars → LLM-summarise the oldest half. Most requests exit after strategy 1 or 2. Strategy 4 (the LLM call) triggers only when the others are insufficient. 60% token reduction at 10k requests/day is a meaningful monthly cost difference. Core class: ContextCompressor
Lesson 13 — Real-Time Cost Dashboard
Cost-per-request trend and cache hit rate are the two metrics that catch regressions before the monthly invoice. Total spend is a lagging indicator. You build five PromQL queries, a per-user spend API endpoint (/admin/cost/users/{id}), and a Slack budget alert that fires at 90% daily consumption. Key function: user_summary()
Lesson 14 — SLOs, Error Budgets & Automated Runbooks
Three SLOs: availability 99.5%, latency p95 < 2s, cost per request < $0.005. Each has an error budget — the allowed failure allocation for the month. When 2% of the budget is consumed in one hour, AutomatedRunbook fires: reduce rate limits, extend cache TTL, route non-critical queries to a cheaper model. Idempotent actions only — the runbook may fire multiple times. Core classes: SLO, AutomatedRunbook
Phase 3 — LLMOps & Advanced Orchestration
Lessons 15–21
The feedback loop that makes your agent continuously improve without human intervention. By Lesson 21 the agent blocks quality regressions in CI, runs live A/B experiments, heals from failures in milliseconds, improves its own prompt accuracy on a weekly schedule, and uses structured relationship graphs for compliance-queryable audit trails.
Lesson 15 — Evaluation Harness — Automated Quality Gates
EvalCase defines input, must_contain assertions, must_not_contain assertions, and a max_cost_usd cap. EvalHarness runs a suite against a live agent, produces a PASS/FAIL report, and exits with code 1 — blocking the CI merge. A 5 percentage point quality regression is caught in minutes, not days. Core classes: EvalCase, EvalResult, EvalHarness
Lesson 16 — A/B Testing Framework — Ship Changes Without Guessing
ABRouter hashes user_id via MD5 to deterministically assign the same user to the same variant on every request. A variant that improves accuracy but introduces a must_not_contain failure is never eligible for promotion regardless of its pass rate. Minimum sample size enforced before any recommendation surfaces. Core classes: Variant, ABRouter
Lesson 17 — Self-Healing Agent Loop — Autonomous Recovery
CircuitBreaker is a three-state FSM: CLOSED (normal), OPEN (reject immediately), HALF_OPEN (probe with exactly one request). After failure_threshold consecutive failures, the breaker opens in milliseconds — no waiting for timeouts. One circuit per downstream service: LLM API, database, cache, each tool. HealthMonitor polls four signals against THRESHOLDS and calls AutomatedRunbook.respond() on breach. Core classes: State, CircuitBreaker, HealthMonitor
Lesson 18 — Prompt Optimisation — Systematic Improvement
PromptOptimiser runs every candidate through EvalHarness, records all results in OptimisationRun, selects the winner by pass rate, and commits the run ID to the audit log. “I tweaked the wording and it felt better” is not a reproducible process. The audit log makes every prompt change attributable, comparable, and reversible. Core classes: PromptCandidate, OptimisationRun, PromptOptimiser
Lesson 19 — Multi-Agent Debate — +15pp Accuracy via Disagreement
Generator produces an initial answer (1 LLM call). N critics run in parallel via asyncio.gather() — each identifies specific weaknesses in the generator’s response. Arbiter synthesises the final answer (1 LLM call). Result: +13–16 percentage point accuracy on complex reasoning tasks, at a 3–5× cost multiplier. A query classifier routes only high-stakes queries to the debate engine. Core class: DebateEngine
Lesson 20 — Knowledge Graph Integration — Structured Retrieval
“Which agents called the database tool in the last 7 days, and who authorised each call?” is a graph question. A vector store cannot answer it. AgentKnowledgeGraph (Neo4j) answers it in one Cypher query: (User)-[:TRIGGERED]->(AuditEvent)-[:INVOLVED]->(Tool). All writes are async — the graph never blocks the response path. Core class: AgentKnowledgeGraph
Lesson 21 — The Full LLMOps Pipeline — Closing the Loop
Six stages, fully automated: collect 500 production traces → build eval suite from top queries → run baseline → generate candidates → A/B route on live traffic → deploy winner when significance reached. LLMOpsPipeline runs on a weekly schedule and immediately after any quality alert. A 3 percentage point improvement threshold prevents noisy micro-improvements from deploying. Core class: LLMOpsPipeline
Phase 4 — Enterprise Deployment & Capstone
Lessons 22–30
Kubernetes, compliance, and five production agents. By Lesson 30 you have a deployment that a security team can audit, a compliance team can certify, and an SRE team can operate — and a portfolio of five agents that demonstrate every technique from Phases 1–3 working together.
Lesson 22 — Kubernetes Deployment — Production Manifests
maxUnavailable: 0 in the rolling update strategy means zero dropped requests during deploys. runAsNonRoot: true eliminates an entire class of container escape impact. The readiness probe means no traffic arrives until the pod signals ready. The preStop: sleep 5 closes the race condition between SIGTERM and the load balancer’s routing table update. Artifacts: k8s/namespace.yaml, k8s/deployment.yaml, k8s/hpa.yaml, k8s/configmap.yaml
Lesson 23 — Auto-Scaling — KEDA & Custom Metrics
CPU spikes after the LLM call completes. agent_request_queue_depth spikes when requests arrive — before any processing begins. KEDA scales on queue depth: proactively, before latency degrades. A cron trigger keeps minimum replicas non-zero during business hours and allows scale-to-zero overnight. Typical result: 60–70% infrastructure cost reduction on normal traffic patterns. Artifacts: k8s/keda-scaledobject.yaml, k8s/prometheus-adapter.yaml, scripts/scaling_savings.py
Lesson 24 — SOC 2 Compliance — Audit Trail & Access Controls
Each ImmutableAuditWriter entry includes a chain_hash — the SHA-256 of the previous entry’s JSON. Modify any entry and the chain breaks. generate_evidence_pack() produces structured output pointing to the actual evidence: S3 bucket ARN, IAM policy document, CloudWatch alert ARNs. Auditors need pointers to real evidence, not prose descriptions. Core classes: ImmutableAuditWriter · Key function: generate_evidence_pack()
Lesson 25 — HIPAA-Ready Agents — PHI Handling & Audit Requirements PHIDetector scans eight PHI categories before any input reaches the LLM API. HIPAASessionManager enforces a 900-second (15-minute) session timeout: assert_valid() rejects any session idle longer than that. The timeout is not configurable — it is a regulatory requirement under 45 CFR 164.312(a). No BAA with your LLM provider means no HIPAA compliance regardless of what you build. Core classes: PHIDetector, HIPAASessionManager
Lesson 26 — Disaster Recovery — Multi-Region Failover
Route 53 polls every 10 seconds. Three consecutive failures (30 seconds) trigger automatic failover — no human required, no runbook to locate, no pager to answer. The quarterly DR test script measures actual RTO against your SLA target and outputs a structured pass/fail result for your audit record. A DR plan never tested is a document, not a capability. Key function: dr_test.py
Lesson 27 — Cost Governance — FinOps for AI Agents
Three allocation tags on every LLM call: team_id, product_id, agent_type. CostAllocator produces a monthly showback report sorted by spend. Cache savings is tracked as a separate line item — the number that makes the FinOps conversation with leadership concrete. Budget alerts at 80% consumption, not 100%. Core classes: CostAllocation, CostAllocator
Lesson 28 — Capstone Part 1: Agents 1 & 2
Agent 1 — Secure Customer Service: a prompt injection attempt hits Layer 4 and is rejected in under 5ms. The LLM never sees it. Agent 2 — Multi-Agent Research: three specialist workers (research, analysis, writing) run in parallel. One worker’s failure does not cascade to the response.
Lesson 29 — Capstone Part 2: Agents 3 & 4
Agent 3 — Streaming Analytics: security runs synchronously, streaming begins only after all four layers clear. Agent 4 — Self-Healing LLMOps: CircuitBreaker, ABRouter, EvalHarness, and PromptOptimiser wired into a single feedback loop that monitors and improves the agent’s own quality — without human intervention.
Lesson 30 — Capstone Part 3: Agent 5 + Course Complete
Agent 5 — Enterprise Kubernetes Deployment: every component from Lessons 1–29 deployed as a single production system. SOC 2 audit trail active from the first request. HIPAA PHI detection at the perimeter. KEDA scaling on queue depth. Route 53 failover tested and documented. FinOps allocator tagging every LLM call.
This is the agent you put in your portfolio.
Beyond the Curriculum
Most AI tutorials stop once the model generates a response. Production AI Engineering begins where those tutorials end. Throughout this course, you'll learn how to build AI systems that remain secure under attack, scale under heavy traffic, recover from failures automatically, satisfy enterprise compliance requirements, and continuously improve through automated LLMOps pipelines.
Every lesson focuses on solving a real production engineering challenge using proven architectural patterns—not shortcuts or abstractions. By the final capstone, you won't just have five portfolio-ready AI agents—you'll understand how modern enterprise AI platforms are designed, operated, and trusted in production.
Ready to Build Production AI Systems?
If you’ve made it this far, you’ve already seen that this isn’t another “build an AI chatbot” course. It’s a complete roadmap to engineering secure, scalable, and enterprise-ready AI systems from the ground up.
Subscribe to unlock every lesson that demonstrates real Production AI Engineering skills.
s a SOC 2 audit, and handles a regional cloud outage in under 30 minutes — without human intervention.
The five agents you ship in the capstone are portfolio-ready. They demonstrate the layer of AI engineering that enterprise teams actually pay for — and that no LangChain tutorial has ever covered.
30 Lessons. 4 Modules. 5 Production Agents. Starts August 4, 2026.
Explore Lessons 1–7: Secure Agent Foundations. Continue your journey through the complete Production AI Engineering.
What You Build
Throughout this course, you'll build five production-ready AI agents that gradually evolve into a complete enterprise AI platform. You'll engineer secure agent architectures, multi-agent collaboration, real-time streaming, self-improving LLMOps pipelines, and Kubernetes-native deployments with enterprise security, compliance, disaster recovery, and FinOps.
Every project solves a real production challenge using the same architectural patterns found in enterprise AI systems. By the end, you'll have a portfolio that demonstrates not just how to use AI—but how to engineer AI systems that organizations can confidently deploy at scale.
Agent 1 — Secure Customer Service Agent
Every Week 1 control in one request: injection blocked at Layer 4 in <5ms, RBAC enforced at Layer 3, semantic cache at Layer 2 reducing token cost by 40%, output scanned for API keys and PII before delivery.
Agent 2 — Multi-Agent Research System
SupervisorAgent fans out to three parallel WorkerAgent instances via asyncio.gather(). One worker failure does not cancel the others. Total latency = slowest single worker, not the sum.
Agent 3 — Streaming Analytics Agent
StreamingLLM → SSE endpoint → EventSource client. First token under 500ms. The X-Accel-Buffering: no header that makes nginx stop buffering your stream.
Agent 4 — Self-Healing LLMOps Agent
EvalHarness blocks deploys on regression. ABRouter routes 10% of live traffic to a new prompt variant. CircuitBreaker trips after 3 failures and rejects requests in microseconds. PromptOptimiser runs weekly and commits winners to version control.
Agent 5 — Enterprise Kubernetes Deployment
KEDA scales on agent_request_queue_depth, not CPU — before latency degrades, not after. SOC 2 audit trail with chain-hash tamper detection. HIPAA PHI detector blocks protected data at the perimeter. Route 53 failover completes in under 30 minutes. FinOps allocator bills teams for exactly what they consumed.
Who This Course Is For
Backend engineers who have built agents that work in demos and want to understand why they fail at scale.
Platform engineers who need to deploy LLM workloads on Kubernetes with the same operational rigour as any other production service.
Tech leads who own the “we need SOC 2 compliance for our AI system” conversation and need to know what to build.
Founding engineers at AI startups where the next customer due diligence will ask about security architecture, compliance controls, and disaster recovery.
What Makes This Different
No frameworks until you understand the layer below. You implement the security perimeter before touching any orchestration library. You implement the token bucket before reading about rate-limiting middleware.
Real numbers, not analogies. 60% token reduction from context compression. 15 percentage point accuracy gain from multi-agent debate. 67% infrastructure cost reduction from KEDA scale-to-zero. Every number comes from the lesson that demonstrates it.
Failure first. Every module ends with a “what breaks without this” section. You learn the failure mode the lesson prevents before you learn how it prevents it.
Production checklist on every lesson. Not “nice to have” suggestions — the specific checks that get caught in SOC 2 audits, load tests, and security reviews.
Learning Objectives
By Phase 1: You can articulate the four-layer agent security model, explain why tool calls require a five-gate validation pipeline, and deploy a secure agent base with 8-signal observability.
By Phase 2: You can implement parallel multi-agent dispatch, achieve first-token-under-500ms streaming, build layered error recovery with a dead-letter queue, and reduce token costs by 60% through context compression.
By Phase 3: You can build a CI quality gate that blocks deployments on regression, run statistically-valid A/B experiments on live traffic, implement a self-healing circuit breaker, and close the LLMOps feedback loop automatically.
By Phase 4: You can deploy an AI agent on Kubernetes with zero-downtime rolling updates, queue-depth auto-scaling, SOC 2 audit controls, HIPAA compliance safeguards, multi-region disaster recovery, and FinOps cost allocation.
Key Topics
Security Architecture: 4-layer enforcement model, prompt injection detection (regex + structural analysis), RBAC with role inheritance, output scanning for secrets and PHI, sandboxed subprocess execution with import allowlist.
Memory & Cost: Conversation buffer, semantic similarity cache (55% hit rate → 40% cost reduction), vector store for cross-session recall, four-strategy context compressor (60% token reduction), real-time Prometheus cost dashboard.
Reliability Engineering: Token bucket rate limiting, four-class error taxonomy, exponential backoff with jitter, model fallback chain, dead-letter queue, three-state circuit breaker, SLOs with error budget tracking and automated runbooks.
LLMOps: Evaluation harness with CI gate, A/B testing with deterministic routing and statistical significance, prompt optimisation with full audit trail, multi-agent debate (+15pp accuracy), knowledge graph for compliance-queryable relationships, full continuous improvement pipeline.
Enterprise Infrastructure: Kubernetes Deployment + HPA + KEDA (queue-depth scaling, scale-to-zero), SOC 2 immutable audit trail with chain-hash tamper detection, HIPAA PHI detection and 15-minute auto-logoff, multi-region disaster recovery (RTO < 30 min, RPO < 5 min), FinOps cost allocation with team showback and 80%-threshold budget alerts.
Prerequisites
Python 3.11+
Comfortable reading and writing async Python (
asyncio,await)Basic Docker knowledge (build an image, run a container)
No prior AI/agent experience required — but familiarity with API calls helps
OpenAI API key optional — every lesson runs in stub mode without one
Course Structure
Phase 1 — Secure Agent Foundations
Lessons 1–7
The architecture that every production agent runs on. By Lesson 7 you have a complete, hardened agent base — security, tools, memory, rate limiting, observability, and sandboxed execution. This is the foundation Phases 2–4 build on without modification.
Lesson 1 — The 4-Layer Secure Agent Architecture
Every production agent enforces the same request path: Security Perimeter → Tool Orchestrator → Memory Manager → Core LLM. A request that fails Layer 4 never reaches Layer 1. You implement all four layers — including stubs for the ones Lessons 2–7 will fill — so the architecture is correct from Day 1. Core types: SecurityContext, AgentRequest, AgentResponse
Lesson 2 — Tool Execution & Validation
Five gates, in order: registry lookup, Pydantic schema validation, RBAC permission check, sandboxed subprocess execution, output filtering. Miss gate 3 and an unpermissioned caller reaches your database tool. Miss gate 5 and your agent returns API keys in its responses. Core classes: ToolDefinition, ToolOrchestrator
Lesson 3 — Memory Systems — Short-Term, Semantic Cache & Long-Term
Three layers with different jobs: ConversationBuffer keeps the last N turns. SemanticCache returns cached responses to near-identical queries — 55% hit rate in production means 40% cost reduction without touching a single prompt. LongTermMemory enables cross-session recall via a vector store interface. Core classes: ConversationBuffer, SemanticCache, LongTermMemory
Lesson 4 — Rate Limiting & Cost Control
The token bucket algorithm: capacity sets the burst ceiling, refill_rate sets the sustained limit. A request with a projected cost above PER_REQUEST_CAP is rejected before the LLM call. A user who crosses DAILY_BUDGET_USD is blocked until midnight. This is what prevents the $40,000 API bill. Core classes: Bucket, RateLimiter, CostTracker
Lesson 5 — Security Hardening — Prompt Injection, RBAC & Output Guardrails PromptGuard runs three layers: regex signatures, structural analysis, anomaly scoring. RBAC resolves role inheritance so admin implies power implies user implies guest. OutputGuard scans every response for API key patterns, SSNs, card numbers, and filesystem paths before the response leaves the agent. Core classes: PromptGuard, RBAC, OutputGuard
Lesson 6 — Observability — Metrics, Structured Logs & Distributed Traces
Eight production signals: four golden (traffic, latency, errors, saturation) and four AI-specific (cost per request, cache hit rate, security rejection rate, tool execution latency). Every layer boundary emits a structured JSON log line with request_id. Four Grafana alert rules. One Telemetry class that wires all three pillars. Core class: Telemetry
Lesson 7 — Sandboxed Tool Execution — Isolated, Resource-Capped & Audited SandboxExecutor spawns a subprocess, enforces RLIMIT_AS memory cap, kills on timeout, applies an import allowlist, and rejects any non-JSON output. SandboxAuditLog writes a tamper-evident record of every execution — args are hashed, never stored raw. An open import list is a filesystem deletion waiting to happen. Core classes: SandboxPolicy, SandboxExecutor, SandboxAuditLog
Phase 2 — Production Integration
Lessons 8–14
Patterns that make the Phase 1 foundation scale: parallel agents, streaming, layered error recovery, token compression, and production monitoring. By Lesson 14 the agent handles real concurrent load, streams output to the browser, recovers from failures without human intervention, and has a real-time dashboard showing spend per user per minute.
Lesson 8 — Multi-Agent Orchestration — The Supervisor Pattern SupervisorAgent dispatches to N WorkerAgent instances via asyncio.gather(). Total wall-clock time = slowest worker, not the sum. A failed worker returns a WorkerResult with success=False — the supervisor logs it, excludes it from the response, and returns the rest. One worker going down does not take the request down. Core classes: WorkerTask, WorkerResult, WorkerAgent, SupervisorAgent
Lesson 9 — Async Tool Pipelines — Fan-Out / Fan-In
Three sequential tool calls at 1,500ms become three parallel calls at 500ms — a 3× latency reduction with no change to the output. asyncio.gather(return_exceptions=True) ensures one failed tool never cancels the others. plan_tool_calls() groups dependent tools into sequential batches and independent tools into parallel batches. Core classes: AsyncToolOrchestrator, ToolPlan
Lesson 10 — Streaming Responses — SSE Token Pipeline
StreamingLLM calls OpenAI with stream=True and yields tokens as they arrive. The FastAPI route flushes every 5 tokens as an SSE data: frame — single-token flushes are noisy, 20-token flushes feel laggy. Two response headers prevent nginx from buffering the stream. First token under 500ms. Core class: StreamingLLM
Lesson 11 — Error Recovery — Retry, Fallback & Dead-Letter Queue
classify() maps any exception to one of four classes: TRANSIENT → retry with exponential backoff + jitter. MODEL → FallbackLLM tries the next model in the chain. TOOL → degrade gracefully. FATAL → DeadLetterQueue + alert. The jitter term is not optional: without it, all agents that hit a rate limit simultaneously retry in lockstep and re-hit it on every wave. Core classes: ErrorClass, FallbackLLM, DeadLetterQueue
Lesson 12 — Context Compression — 60% Token Reduction
Four strategies, applied cheapest-first: strip tool call metadata → remove near-duplicates → truncate system prompt to 800 chars → LLM-summarise the oldest half. Most requests exit after strategy 1 or 2. Strategy 4 (the LLM call) triggers only when the others are insufficient. 60% token reduction at 10k requests/day is a meaningful monthly cost difference. Core class: ContextCompressor
Lesson 13 — Real-Time Cost Dashboard
Cost-per-request trend and cache hit rate are the two metrics that catch regressions before the monthly invoice. Total spend is a lagging indicator. You build five PromQL queries, a per-user spend API endpoint (/admin/cost/users/{id}), and a Slack budget alert that fires at 90% daily consumption. Key function: user_summary()
Lesson 14 — SLOs, Error Budgets & Automated Runbooks
Three SLOs: availability 99.5%, latency p95 < 2s, cost per request < $0.005. Each has an error budget — the allowed failure allocation for the month. When 2% of the budget is consumed in one hour, AutomatedRunbook fires: reduce rate limits, extend cache TTL, route non-critical queries to a cheaper model. Idempotent actions only — the runbook may fire multiple times. Core classes: SLO, AutomatedRunbook
Phase 3 — LLMOps & Advanced Orchestration
Lessons 15–21
The feedback loop that makes your agent continuously improve without human intervention. By Lesson 21 the agent blocks quality regressions in CI, runs live A/B experiments, heals from failures in milliseconds, improves its own prompt accuracy on a weekly schedule, and uses structured relationship graphs for compliance-queryable audit trails.
Lesson 15 — Evaluation Harness — Automated Quality Gates
EvalCase defines input, must_contain assertions, must_not_contain assertions, and a max_cost_usd cap. EvalHarness runs a suite against a live agent, produces a PASS/FAIL report, and exits with code 1 — blocking the CI merge. A 5 percentage point quality regression is caught in minutes, not days. Core classes: EvalCase, EvalResult, EvalHarness
Lesson 16 — A/B Testing Framework — Ship Changes Without Guessing
ABRouter hashes user_id via MD5 to deterministically assign the same user to the same variant on every request. A variant that improves accuracy but introduces a must_not_contain failure is never eligible for promotion regardless of its pass rate. Minimum sample size enforced before any recommendation surfaces. Core classes: Variant, ABRouter
Lesson 17 — Self-Healing Agent Loop — Autonomous Recovery
CircuitBreaker is a three-state FSM: CLOSED (normal), OPEN (reject immediately), HALF_OPEN (probe with exactly one request). After failure_threshold consecutive failures, the breaker opens in milliseconds — no waiting for timeouts. One circuit per downstream service: LLM API, database, cache, each tool. HealthMonitor polls four signals against THRESHOLDS and calls AutomatedRunbook.respond() on breach. Core classes: State, CircuitBreaker, HealthMonitor
Lesson 18 — Prompt Optimisation — Systematic Improvement
PromptOptimiser runs every candidate through EvalHarness, records all results in OptimisationRun, selects the winner by pass rate, and commits the run ID to the audit log. “I tweaked the wording and it felt better” is not a reproducible process. The audit log makes every prompt change attributable, comparable, and reversible. Core classes: PromptCandidate, OptimisationRun, PromptOptimiser
Lesson 19 — Multi-Agent Debate — +15pp Accuracy via Disagreement
Generator produces an initial answer (1 LLM call). N critics run in parallel via asyncio.gather() — each identifies specific weaknesses in the generator’s response. Arbiter synthesises the final answer (1 LLM call). Result: +13–16 percentage point accuracy on complex reasoning tasks, at a 3–5× cost multiplier. A query classifier routes only high-stakes queries to the debate engine. Core class: DebateEngine
Lesson 20 — Knowledge Graph Integration — Structured Retrieval
“Which agents called the database tool in the last 7 days, and who authorised each call?” is a graph question. A vector store cannot answer it. AgentKnowledgeGraph (Neo4j) answers it in one Cypher query: (User)-[:TRIGGERED]->(AuditEvent)-[:INVOLVED]->(Tool). All writes are async — the graph never blocks the response path. Core class: AgentKnowledgeGraph
Lesson 21 — The Full LLMOps Pipeline — Closing the Loop
Six stages, fully automated: collect 500 production traces → build eval suite from top queries → run baseline → generate candidates → A/B route on live traffic → deploy winner when significance reached. LLMOpsPipeline runs on a weekly schedule and immediately after any quality alert. A 3 percentage point improvement threshold prevents noisy micro-improvements from deploying. Core class: LLMOpsPipeline
Phase 4 — Enterprise Deployment & Capstone
Lessons 22–30
Kubernetes, compliance, and five production agents. By Lesson 30 you have a deployment that a security team can audit, a compliance team can certify, and an SRE team can operate — and a portfolio of five agents that demonstrate every technique from Phases 1–3 working together.
Lesson 22 — Kubernetes Deployment — Production Manifests
maxUnavailable: 0 in the rolling update strategy means zero dropped requests during deploys. runAsNonRoot: true eliminates an entire class of container escape impact. The readiness probe means no traffic arrives until the pod signals ready. The preStop: sleep 5 closes the race condition between SIGTERM and the load balancer’s routing table update. Artifacts: k8s/namespace.yaml, k8s/deployment.yaml, k8s/hpa.yaml, k8s/configmap.yaml
Lesson 23 — Auto-Scaling — KEDA & Custom Metrics
CPU spikes after the LLM call completes. agent_request_queue_depth spikes when requests arrive — before any processing begins. KEDA scales on queue depth: proactively, before latency degrades. A cron trigger keeps minimum replicas non-zero during business hours and allows scale-to-zero overnight. Typical result: 60–70% infrastructure cost reduction on normal traffic patterns. Artifacts: k8s/keda-scaledobject.yaml, k8s/prometheus-adapter.yaml, scripts/scaling_savings.py
Lesson 24 — SOC 2 Compliance — Audit Trail & Access Controls
Each ImmutableAuditWriter entry includes a chain_hash — the SHA-256 of the previous entry’s JSON. Modify any entry and the chain breaks. generate_evidence_pack() produces structured output pointing to the actual evidence: S3 bucket ARN, IAM policy document, CloudWatch alert ARNs. Auditors need pointers to real evidence, not prose descriptions. Core classes: ImmutableAuditWriter · Key function: generate_evidence_pack()
Lesson 25 — HIPAA-Ready Agents — PHI Handling & Audit Requirements PHIDetector scans eight PHI categories before any input reaches the LLM API. HIPAASessionManager enforces a 900-second (15-minute) session timeout: assert_valid() rejects any session idle longer than that. The timeout is not configurable — it is a regulatory requirement under 45 CFR 164.312(a). No BAA with your LLM provider means no HIPAA compliance regardless of what you build. Core classes: PHIDetector, HIPAASessionManager
Lesson 26 — Disaster Recovery — Multi-Region Failover
Route 53 polls every 10 seconds. Three consecutive failures (30 seconds) trigger automatic failover — no human required, no runbook to locate, no pager to answer. The quarterly DR test script measures actual RTO against your SLA target and outputs a structured pass/fail result for your audit record. A DR plan never tested is a document, not a capability. Key function: dr_test.py
Lesson 27 — Cost Governance — FinOps for AI Agents
Three allocation tags on every LLM call: team_id, product_id, agent_type. CostAllocator produces a monthly showback report sorted by spend. Cache savings is tracked as a separate line item — the number that makes the FinOps conversation with leadership concrete. Budget alerts at 80% consumption, not 100%. Core classes: CostAllocation, CostAllocator
Lesson 28 — Capstone Part 1: Agents 1 & 2
Agent 1 — Secure Customer Service: a prompt injection attempt hits Layer 4 and is rejected in under 5ms. The LLM never sees it. Agent 2 — Multi-Agent Research: three specialist workers (research, analysis, writing) run in parallel. One worker’s failure does not cascade to the response.
Lesson 29 — Capstone Part 2: Agents 3 & 4
Agent 3 — Streaming Analytics: security runs synchronously, streaming begins only after all four layers clear. Agent 4 — Self-Healing LLMOps: CircuitBreaker, ABRouter, EvalHarness, and PromptOptimiser wired into a single feedback loop that monitors and improves the agent’s own quality — without human intervention.
Lesson 30 — Capstone Part 3: Agent 5 + Course Complete
Agent 5 — Enterprise Kubernetes Deployment: every component from Lessons 1–29 deployed as a single production system. SOC 2 audit trail active from the first request. HIPAA PHI detection at the perimeter. KEDA scaling on queue depth. Route 53 failover tested and documented. FinOps allocator tagging every LLM call.
This is the agent you put in your portfolio.
Beyond the Curriculum
Most AI tutorials stop once the model generates a response. Production AI Engineering begins where those tutorials end. Throughout this course, you'll learn how to build AI systems that remain secure under attack, scale under heavy traffic, recover from failures automatically, satisfy enterprise compliance requirements, and continuously improve through automated LLMOps pipelines.
Every lesson focuses on solving a real production engineering challenge using proven architectural patterns—not shortcuts or abstractions. By the final capstone, you won't just have five portfolio-ready AI agents—you'll understand how modern enterprise AI platforms are designed, operated, and trusted in production.
Ready to Build Production AI Systems?
If you’ve made it this far, you’ve already seen that this isn’t another “build an AI chatbot” course. It’s a complete roadmap to engineering secure, scalable, and enterprise-ready AI systems from the ground up.
Subscribe to unlock every lesson that demonstrates real Production AI Engineering skills.