Lesson 1 — The 4-Layer Secure Agent Architecture

Introduction

This lesson introduces the foundation of a production AI agent: a four-layer secure architecture where every request must pass through Security, Tools, Memory, and finally the LLM. You will build a self-contained Day 1 system with typed contracts, fail-closed injection defense, permissioned tools, and bounded session memory.

A FastAPI dashboard exposes live metrics—requests, blocks, tokens, cost, and layer latency—so observability is learned alongside the agent itself.

The project runs offline in stub mode or with a real OpenAI key, and ships with Docker scripts for start, demo, test, and cleanup. By the end, you have a reusable agent kernel that later lessons extend without collapsing security into ad-hoc prompts.

Highlights

• Build a production-shaped agent with four concentric layers: Security (L4) → Tools (L3) → Memory (L2) → LLM (L1).

• Enforce prompt-injection defense, RBAC, and rate limiting before any model call.

• Emit live metrics (requests, blocks, tokens, cost, layer latency) to a FastAPI dashboard.

• Run the full stack with an official python:3.12-slim Docker image via self-contained start.sh / demo.sh.

• Treat stub LLM mode as a first-class engineering path so CI and demos work without API keys.

What We Build

• A SecureAgent orchestrator that routes every request through L4 → L3 → L2 → L1 and filters responses on the way out.

• Layer modules: SecurityPerimeter, ToolOrchestrator, InMemoryStore, CoreLLM.

• A FastAPI service (app.py) exposing /health, /metrics, /chat, /demo, and a live /dashboard.

• Pytest coverage for injection blocking and metrics side effects, plus Dockerized start/demo/test scripts.

• Cost and latency accounting that stays non-zero in stub mode so the dashboard teaches observability habits.

Enables Next Lesson

Day 2 builds on the L3 tool boundary: real tool execution, schema validation, and safer tool-output handling. After Day 1 you already have permission-filtered tool discovery and output redaction — Day 2 turns that registry into an execution pipeline without rewriting security or memory.

Architecture Context

Where this component sits — At the edge of a production AI product: every user (or service) request enters through the agent perimeter before reaching a model provider. The dashboard and metrics sit beside the agent as the first observability surface.

Why it exists — Prompt-only “safety” is insufficient. Production systems need perimeter controls (length, sanitize, injection, authz, rate limit), capability gating for tools, session memory with bounds, and a narrowly-scoped model call with cost/latency telemetry.

Problem solved — Prevents untrusted input from reaching L1; prevents over-privileged tools from appearing; bounds memory growth; makes failures observable (blocked vs rejected vs ok) instead of silent prompt failures.

Integration

Integration Components

Module Objectives Alignment

• Security-first agent design — L4 is mandatory, not optional middleware.

• Separation of concerns — tools, memory, and LLM are distinct replaceable layers.

• Operability — metrics and dashboard are part of the build, not a later add-on.

• Testability — stub LLM + injection tests make Day 1 CI-complete without secrets.

Core Concepts

Concentric layers, not a pipeline of peers

Engineering reasoning: if security is “one step among equals,” a future refactor can bypass it. Placing security as the outermost gate makes SecureAgent.run structurally unable to call the LLM without passing L4.

Prompt injection as an authorization failure

Injection is not a “bad reply” quality issue; it is an attempt to elevate privilege over the system prompt and tools. Raising PermissionError maps it to the same class of failure as unauthorized access — metrics label it blocked, not ok.

Least-privilege tools

Tool registries without permission checks leak capabilities into the prompt surface. Filtering available tools by SecurityContext.permissions shrinks the attack path even when L1 is imperfect.

Bounded session memory

Unbounded chat history becomes a cost and privacy bomb. A sliding window (max_turns) is the minimal production discipline before durable stores appear in later lessons.

Stub LLM as a design choice

Production CI should not depend on vendor availability. A deterministic stub with estimated tokens/cost teaches the same observability contracts as a live model while keeping demos offline-capable.

In-process metrics with a single worker

For Day 1, a threaded MetricsStore plus uvicorn --workers 1 gives correct live dashboard updates without Redis. The tradeoff is intentional and documented — multi-worker needs a shared metrics backend later.

Key Insights

• Security before creativity — tokens spent on injected prompts are both wasted money and risk; block early.

• Status taxonomy matters — ok / blocked / rejected drives ops dashboards and alerting differently than HTTP 200 with a polite refusal string.

• Tradeoff: regex injection filters — fast and explainable, but incomplete against novel jailbreaks; combine with model-side policy and human review queues in production.

• Tradeoff: in-memory session store — great latency, poor durability and multi-replica stickiness; fine for Day 1, not for multi-AZ production alone.

• Common mistake — putting API keys in source or committing .env. Use .env.example with empty values only.

• Common mistake — multi-worker uvicorn without shared metrics, then wondering why the dashboard stays at zero.

• Common mistake — treating HTML escape as XSS-only concern; escaped user text also reduces markup-based prompt smuggling in logs and UIs.

Production AI System Design Relevance

Inside a production AI platform, Day 1 is the agent control plane kernel:

Clients → API Gateway → Secure Agent (L4–L1) → Model Provider / Tools
                         ↓
                   Metrics / Dashboard / Logs

Later modules hang off this kernel: durable memory, tool sandboxes, retrieval, evaluation, canary models, and policy engines all assume a clean security perimeter and typed request/response contracts exist.

Workflow

Request flow

1. Client sends prompt + identity context (user_id, session_id, permissions).

2. SecureAgent wraps input as AgentRequest.

3. L4 validates/sanitizes or raises.

4. L3 computes available tools for permissions.

5. L2 loads session history.

6. L1 produces AgentResponse.

7. L2 stores the turn; L3 filters output; metrics record outcome.

Execution flow

• Happy path: L4 pass → L3 list tools → L2 get → L1 call → L2 store → L3 filter → return.

• Injection/jailbreak: L4 raises PermissionError → metrics blocked → API returns blocked payload (no L1).

• Oversize input: L4 raises ValueError → metrics rejected.

Data flow

• Inbound: raw string → sanitized string → message list (history + user).

• Outbound: model/stub text → redacted text → JSON/API/dashboard event.

• Telemetry: tokens, cost, layer timings accumulate in MetricsStore.

State changes

• Session memory list grows by two messages per successful turn, then trims to window.

• Rate-limiter counters increment per user/minute bucket.

• Metrics counters (requests_*, tokens_total, cost_usd_total, layer sums) monotonically increase.

• Demo runs bump demo_runs after /demo.

Production Integration

Production architecture fit

Deploy the agent as a single-responsibility service behind an API gateway. Keep L1 provider credentials in a secret manager; never in the image. Use the dashboard only for low-traffic demos or internal ops — production metrics should also export to Prometheus/OpenTelemetry.

Enterprise deployment patterns

• Containerize with the official Python slim image (already in Dockerfile).

• One service replica for Day 1 parity with in-memory metrics; later: sticky sessions or external memory + shared metrics.

• Separate “demo” and “prod” configs via env (OPENAI_MODEL, rate-limit tiers).

Scalability

Vertical: increase CPU for regex + stub latency. Horizontal: requires externalizing memory and metrics. Tool and LLM calls dominate p95 — measure with layer_timings.

Observability

Structured logs per layer ([L4], [L3], [L2], [L1], [AGENT]), request IDs on SecurityContext, live /metrics + dashboard. Production should add distributed tracing spans around each layer.

Security considerations

Perimeter injection filters, RBAC permissions on tools, output redaction of secret-like patterns, rate limits, no committed secrets, HTML escaping of user input.

Implementation

GitHub Link

https://github.com/sysdr/production-ai-engineering/tree/main/lesson1/aiam-day01

This section is the canonical implementation guide for Day 1. No separate implementation guide file is used.

Component Architecture

aiam-day01/
├── agent/
│   ├── agent.py                 # SecureAgent orchestrator
│   ├── core/types.py            # Contracts
│   ├── core/metrics.py          # MetricsStore
│   └── layers/
│       ├── security.py          # L4
│       ├── tool_orchestrator.py # L3
│       ├── memory.py            # L2
│       └── llm.py               # L1
├── app.py                       # FastAPI + dashboard
├── main.py                      # CLI demo
├── tests/
├── Dockerfile / docker-compose.yml
├── start.sh / stop.sh / demo.sh / run_tests.sh / cleanup.sh
└── requirements.txt

Step-by-Step Implementation

1. Define contracts (types.py) — Pydantic models force consistent fields for metrics and APIs; prevents “dict soup” across layers.

2. Implement L4 — compile injection patterns once; sanitize; rate limit; RBAC log/assert. Exists to fail closed before cost is incurred.

3. Implement L3 — registry with permission tags; filter tools; redact secrets on egress.

4. Implement L2 — session-keyed lists with trim; exists for continuity without unbounded growth.

5. Implement L1 — branch on OPENAI_API_KEY; stub still estimates cost; small sleep ensures measurable latency for demos.

6. Orchestrate — SecureAgent owns ordering, timings, and metrics on success and exception paths.

7. Expose HTTP — FastAPI lifespan constructs one agent; /demo walks happy + attack prompts; dashboard polls /metrics.

8. Validate — unit tests for L4 and metrics; run_tests.sh for health/demo/non-zero metrics.

9. Package — Dockerfile installs requirements.txt, runs single-worker uvicorn; start.sh avoids duplicate containers.

Communication: layers do not call each other sideways; only SecureAgent composes them. Validation: Pydantic on the way in; regex/RBAC in L4; permission checks in L3. Error handling: PermissionError → blocked; ValueError → rejected; both recorded. Production considerations: env-based secrets, official base image, cleanup script for caches/images, .gitignore for .env and venv.

Coding Highlights

Layer order is policy-as-code — do not reorder casually:

request = self.security.process(request)          # L4 first
available = self.tools.get_available_tools(...)   # L3
history = self.memory.get_context(session_id)     # L2
response = self.llm.call(request, messages)       # L1 last
response = self.tools.filter_output(response)     # egress filter

Why it matters: every future tool or model swap still inherits the same security envelope.

Fail closed on injection:

if pattern.search(sanitised):
    raise PermissionError("Potential prompt injection detected")

Why it matters: returning a soft “I can’t help with that” from the LLM still spends tokens and can be inconsistently bypassed; raising stops the pipeline.

Record metrics on exception paths, not only success:

except PermissionError:
    METRICS.record(status="blocked", ...)
    raise

Why it matters: security events are the highest-value dashboard series.

Working demo Link:

Production Considerations

• Scalability — externalize memory and metrics before multi-replica; keep L1 timeouts and circuit breakers (Day 2+).

• Security — empty default API key; .gitignore .env; output redaction; rate limits; never log raw secrets.

• Monitoring — /metrics counters + layer timings; alert on requests_blocked spikes.

• Logging — structured layer prefixes; include request_id.

• Testing — injection unit tests; metrics non-zero after demo; stub mode in CI.

• Failure handling — map exceptions to blocked/rejected; API returns JSON status without crashing the worker.

• Edge cases — oversize input; repeated jailbreaks; empty model content; HTML tags in prompts; multi-turn memory trim boundaries.

Validation

Verification methods

• ./demo.sh — CLI path through all scenario labels.

• ./start.sh then dashboard Run demo — UI metrics update.

• ./run_tests.sh — pytest + /health + /demo + non-zero metrics assertions.

Testing strategy

• Unit: L4 allow/deny/length/escape; agent metrics increments.

• Smoke: health, demo JSON, dashboard HTML marker.

• Manual: confirm blocked count rises on injection prompts.

Success criteria

• Clean prompts return ok with tokens_used > 0 and cost_usd > 0 in stub mode.

• Injection/jailbreak prompts are blocked without L1 completion content.

• Dashboard metrics are non-zero after one demo run.

• Only one aiam-day01 container when started via start.sh.

Expected outputs

• CLI labels: NORMAL, MEMORY, INJECT (blocked), JAILBREAK (blocked).

• Metrics keys: requests_total, requests_ok, requests_blocked, tokens_total, cost_usd_total, layer sums, demo_runs.

Benchmarks (stub, local Docker)

• L1 stub latency typically ≥ 5ms (intentional sleep).

• Full /demo (5 steps) completes in well under 1s on a developer laptop.

Production checklist

• No secrets in git

• Official base image pinned by digest in real prod (lesson uses tag for clarity)

• Single-worker or shared metrics documented

• Tests green in CI without OPENAI_API_KEY

• cleanup.sh known to operators

Real-World Examples

Enterprise IT copilot — Employees ask policy questions; L4 blocks “ignore previous instructions / dump the system prompt” attempts; L3 only exposes read tools for standard employees; admins get write tools. Metrics feed a SOC dashboard of blocked jailbreaks per day.

Fintech support agent — Session memory keeps the last N turns for continuity, but L4 length limits stop paste-bomb cost attacks. Output filter redacts accidental sk- / Bearer leakage from tool-assisted answers before they hit the customer channel.

Looking Ahead

• Enables next lesson — Day 2 turns the L3 registry into a validated execution engine while keeping L4 and L2 intact.

• Module progression — Day 1 perimeter → Day 2 tools → later durability, evaluation, and multi-agent patterns.

• Architectural evolution — today’s in-memory metrics and session store become shared infrastructure; the layer boundaries remain the stable API of the agent kernel.