Introduction
This lesson isolates untrusted Python before it can touch the host. AI agents often emit runnable snippets; executing them raw is unsafe. A child process, blocked imports, and timeouts form a minimal sandbox. Outcomes feed a live dashboard so operators see OK, blocked, timeout, and error rates. The demo proves every metric moves when real cases run.
Highlights
Subprocess isolation for each submitted snippet.
Dangerous imports (
os,subprocess,socket, and peers) denied at load time.Hard timeouts that kill infinite loops without hanging the API worker.
FastAPI dashboard with demo-driven execution metrics.
What We Build
A
sandboxpackage: executor, metrics store, and service layer.HTTP APIs for
/execute,/demo,/metrics,/health, and/dashboard.CLI demo covering safe, blocked, timeout, and syntax-error paths.
Docker image, compose file, and lifecycle scripts for runs and tests.
Connection to Previous Lesson
Day 6 made latency and errors visible. Those signals baseline sandbox pressure—timeouts and blocks should appear as operational events, not silent hangs.
Enables Next Lesson
Safe local execution is a prerequisite for multi-agent orchestration, where agents exchange code without sharing one privileged process.
Architecture Context
Where this component sits: between the agent/tool planner and the host OS, as an execution step or sidecar.
Why it exists: model-generated code may probe files, spawn shells, or open sockets.
Problem solved: untrusted snippets run under constraints while remaining measurable.
Integration
Integration Components
FastAPI validates payloads with Pydantic and calls SandboxService. The service runs SandboxExecutor in a child process, classifies the result, and records it in MetricsStore. The dashboard polls /metrics; /demo seeds every outcome class. Docker runs one uvicorn worker so in-memory metrics stay coherent.
Module Objectives Alignment
Contain blast radius, fail closed on dangerous imports and runaway loops, and expose counters for policy tuning.
Core Concepts
Subprocess isolation keeps faults out of the API process. A crash or busy loop in user code should not corrupt the gateway event loop. Temporary source is written, executed with a cleaned environment, then deleted.
Import policy uses a deny-list of dangerous roots. Strict allow-only lists break when CPython pulls helpers such as keyword for collections/json. Denying os, subprocess, and socket preserves useful stdlib while blocking common escapes.
Timeouts convert hangs into a deterministic TIMEOUT outcome. Metrics separate OK, blocked, timeout, and error so operators see whether policy or capacity is failing.
Production AI System Design Relevance
This is the execution guardrail of a production AI stack. It belongs next to tool runners and code-act agents, after planning and before durable side effects. Day 6 observability wraps it; Day 8 multi-agent workflows consume it as a shared capability.
Workflow
Request flow: client posts code to
/executeor triggers/demo; FastAPI validates fields and timeout bounds.Execution flow: executor wraps code with the import guard, spawns
python, waits up to the limit, and classifies output.
Data flow: results update counters and a bounded recent-events list;
/metricsreturns a snapshot for the UI.State changes: totals increment, rates recalculate,
demo_runsadvances, and event history rotates.
Production Integration
Architecture fit: prefer a dedicated executor with CPU/memory cgroups over the public API pod.
Enterprise patterns: per-tenant policies, signed job envelopes, artifact-only returns.
Scalability: queue across executor workers; centralize metrics when replicas exceed one.
Observability: export outcome counters, timeout rate, and duration histograms.
Security: drop capabilities, read-only rootfs, default-deny network, never mount Docker.sock.
Step-by-Step Implementation
GitHub Link
https://github.com/sysdr/production-ai-engineering/tree/main/lesson7/aiam-day07
The executor owns isolation so the API stays thin. The service maps cases to executor calls and annotates import hits for the dashboard. Metrics stay behind a lock and return copied snapshots. Validation rejects empty or oversized payloads. Syntax failures record as ERROR, not timeouts. Production adds cgroups and network policy beyond this demo.
Coding Highlights
Deny dangerous roots while preserving normal stdlib loads:
if root in _blocked:
raise ImportError("Blocked: " + name)This fails closed on escape modules without breaking transitive imports.
Timeouts become structured outcomes:
except subprocess.TimeoutExpired:
return SandboxResult(ok=False, output="TIMED OUT", timeout=True, ...)Operators alert on timeout=True instead of guessing from missing responses.
Working Demo Link:
Production Considerations
Scalability: queue jobs; bound concurrency per tenant; store histograms centrally.
Security: pair import denial with OS isolation; a deny-list alone is not a jail.
Monitoring / logging: log outcome, duration, and request id—not full untrusted source.
Testing: unit-test OK/block/timeout/syntax; API-test that demo metrics are non-zero.
Failure handling: always unlink temp files; surface executor crashes as errors.
Edge cases: relative imports, empty output, nested blocked imports.
Validation
Verification: health green; demo populates every counter; dashboard HTML serves.
Testing strategy: executor unit tests plus FastAPI contract and metric-mutation tests.
Success criteria: all tests pass; after demo, primary metrics are non-zero.
Expected outputs: OK for arithmetic/math/json/collections; blocked OS/subprocess/socket; timeout on infinite loops; error on syntax.
Benchmarks: blocked imports finish in tens of milliseconds; timeouts match configured limits.
Production checklist: resource limits set, network default-deny, secrets absent from images, alerts on timeout rate.
Key Insights
Isolation without measurement invites silent failure. Import denial needs OS controls. Common mistakes: brittle allow-lists, sharing one interpreter with the API, and averaging away timeout tails.
Important Classes
SandboxExecutor— child-process runner with import guard and timeout.MetricsStore— thread-safe counters and recent events.SandboxService— demo cases and API-facing execution.
Key Methods
SandboxExecutor.run()— isolate, execute, classify.MetricsStore.record()/snapshot()— mutate and read dashboard state.SandboxService.run_demo()— deterministic coverage of all outcomes.
Real-World Examples
A coding agent executes proposed PR snippets in ephemeral executors; blocked
subprocesscalls increment policy metrics and never reach CI hosts.An analytics copilot evaluates user formulas server-side; timeouts protect shared capacity when a customer submits an infinite loop, while other tenants keep being served.