Introduction
This lesson adds an output-side security layer to the AI service. It scans generated text for leaked secrets and PII before responses leave the system, and signs payloads with HMAC so integrity can be verified downstream. Findings are redacted, counted, and exposed through a live dashboard. The result is a self-contained service that enforces rather than observes.
Highlights
Regex-based detection of secrets (API keys, tokens) and PII (email, phone, card).
Deterministic redaction with per-category counters.
HMAC-SHA256 signing and constant-time verification.
FastAPI dashboard with non-zero, demo-driven metrics.
What We Build
A security domain package (
hardening,metrics,service).An
OutputGuardthat scans and redacts text.A
SignatureVerifierfor HMAC signing and tamper detection.API surface (
/scan,/sign,/verify,/metrics,/demo,/dashboard,/health).Unit and API tests plus Docker lifecycle scripts.
Connection to Previous Lesson
The previous lesson enforced throughput and cost limits before model execution. This lesson extends enforcement to the output boundary, guarding what the system emits rather than what it accepts.
Enables Next Lesson
Clean, signed, redacted output is a precondition for trustworthy telemetry, enabling the observability stack in the next lesson to reason over safe data.
Architecture Context
Where it sits: at the response egress, between model output and the caller.
Why it exists: models can echo secrets or personal data present in prompts, context, or tool results.
Problem solved: prevents data exfiltration and provides integrity guarantees on payloads crossing trust boundaries.
Integration
Integration Components
FastAPI validates payloads with Pydantic and delegates to
SecurityService. The service runs the guard and signer, records outcomes in a shared in-memory metrics store, and returns a decision. The dashboard polls/metricsand triggers/demo.
Module Objectives Alignment
The module targets three production objectives: prevent sensitive-data leakage, guarantee payload integrity, and make both measurable for operators.
Core Concepts
Output scanning is pattern-driven because secrets and PII share stable syntactic shapes. Compiling patterns once and applying them per request keeps latency low while covering multiple categories in a single pass. Detection and redaction are separated conceptually but computed together: detection drives metrics and block decisions, redaction produces a safe-to-emit string.
HMAC signing solves a different problem — integrity, not confidentiality. A shared secret produces a signature that a holder can verify but an attacker cannot forge without the key. Verification uses constant-time comparison to avoid timing side channels.
A deliberate design choice treats an expected rejection (a tampered payload correctly failing verification) as a success signal, while an unexpected accept is the true failure. This aligns metrics with security intent rather than raw booleans. State is kept in-process to keep scope small while preserving interfaces that can move to shared stores later.
Production AI System Design Relevance
This lesson is the egress guardrail of an AI platform. It belongs next to response serialization and gateway middleware, where every model output can be inspected, redacted, and attested before delivery.
Workflow
Request flow: caller submits text to scan or a payload to sign/verify.
Execution flow: the service runs guard or signer, then records an event.
Data flow: validated input enters the service, mutates in-memory metrics, and returns a decision; the dashboard reads snapshots via
/metrics.State changes: scan/block counters increment, secret/PII tallies grow, HMAC counters update, and a bounded recent-events list is refreshed.
Production Integration
Architecture fit: embed as egress middleware in API workers.
Enterprise patterns: centralize pattern sets and signing keys per tenant or environment.
Scalability: stateless scanning scales horizontally; shared counters move to Redis when multi-worker.
Observability: counters, rates, and an event timeline feed dashboards and alerts.
Security: secrets come from environment variables; comparisons are constant-time.
Step-by-Step Implementation
GitHub Link
https://github.com/sysdr/production-ai-engineering/tree/main/lesson5/aiam-day05
The guard compiles patterns at construction. Each scan collects findings, computes category counts, and builds a redacted copy. The service wraps guard and signer, timing each call and writing an event to the metrics store. The API layer validates inputs, exposes endpoints, and serves the dashboard. Errors surface as validation failures at the edge. The demo exercises clean text, every detector, and both HMAC paths so no metric stays zero.
Coding Highlights
Constant-time verification prevents timing attacks:
def verify(self, payload: str, signature: str) -> bool:
return hmac.compare_digest(self.sign(payload), signature)Intent-aware status keeps metrics meaningful — a caught tamper is a win, not a failure:
status = "tamper_caught" if expect_reject else "hmac_fail"This matters because naive boolean logging flags correct security behavior as an error.
Working Demo Link:
Production Considerations
Scalability: externalize counters for multi-worker deployments.
Security: never log raw findings; store only categories.
Monitoring/logging: track block rate and HMAC failure rate.
Testing: cover each detector and both verification paths.
Failure handling: fail closed on scan errors.
Edge cases: overlapping patterns and unicode payloads.
Validation
Verification: unit tests assert detection, redaction, and HMAC correctness.
Testing strategy: API tests confirm endpoints and non-zero demo metrics.
Success criteria: all tests pass; dashboard values update on demo.
Expected outputs: blocked secrets/PII, valid/tampered HMAC results.
Benchmarks: sub-millisecond scans on short text.
Checklist: secrets from env, redaction on, tests green, health OK.
Key Insights
Redaction and detection must ship together; detection without safe output is incomplete. A common mistake is scanning too late or logging the secrets being caught. The tradeoff is coverage versus false positives.
Important Classes
OutputGuard— scanning and redaction.SignatureVerifier— HMAC signing and verification.SecurityService— orchestration and metrics.
Key Methods
scan()— findings, counts, redacted text.sign()/verify()— integrity operations.run_demo()— deterministic metric population.
Real-World Examples
Chatbot egress filter: a support assistant redacts customer emails and card numbers from replies before display.
Webhook integrity: a deploy service signs payloads with HMAC so receivers reject forged requests.