Lesson 10 — Prompt Engineering

Introduction

This lesson builds a support-message classifier using prompt templates, few-shot examples, and keyword routing — without calling an LLM. PromptTemplate renders {{placeholders}}; FEW_SHOT_EXAMPLES inject labeled pairs into every prompt; classify() routes messages to billing, technical, general, or urgent. MetricsStore records renders, categories, token estimates, and recent events; a FastAPI dashboard polls /metrics. POST /demo runs five messages so every counter moves from zero after the first click.

The project in aiam-day10/ runs via start.sh, demo.sh, and run_tests.sh.

Highlights

• Mustache-style {{key}} template rendering with unresolved-variable detection.

• Few-shot example formatting for consistent classifier prompts.

• Deterministic keyword routing — fast, testable, no API cost.

• Dashboard auto-refreshes; /demo exercises all four categories.

• Docker lifecycle scripts on port 8089.

What We Build

• prompt/ package: template.py, classify.py, metrics.py, service.py.

• FastAPI (app.py): /classify, /demo, /metrics, /health, /dashboard.

• CLI demo (main.py) printing inputs, categories, and token estimates.

• DEMO_MESSAGES — billing, technical, general, urgent — so every dashboard counter updates.

Connection to Previous Lesson

Day 9 sequences tool calls with ReAct thoughts. Day 10 shapes the prompts that would drive that planner: templates structure instructions, few-shot pairs set behavior, routing picks the right handler path.

Enables Next Lesson

Prompt engineering prepares structured output. Production swaps classify() for an LLM call; the template, few-shot injection, and metrics contract stay the same.

Architecture Context

• Where this component sits: between app.py (API) and prompt/classify.py (routing logic).

• Why it exists: consistent, auditable prompts before any model invocation.

• Problem solved: repeatable prompt assembly with measurable category distribution.

Integration

Integration Components

app.py validates ClassifyBody and calls PromptService.render_and_classify(). The service formats few-shot examples, renders SUPPORT_CLASSIFIER_TEMPLATE, runs classify(), and records metrics. /demo loops DEMO_MESSAGES then marks demo_runs.

Module Objectives Alignment

Prove template correctness, isolate routing from rendering, and expose per-category counters so operators confirm all paths executed.

Core Concepts

Templates separate prompt structure from runtime data. PromptTemplate.render(**kwargs) replaces {{examples}} and {{message}}; leftover placeholders raise ValueError — catching typos before production.

Few-shot examples prime classification behavior. Four labeled pairs (billing, technical, general, urgent) format into the examples block; the model (or router) sees consistent Message: / Category: pairs.

Keyword routing maps substrings to categories: "charge" → billing, "500" → technical, "urgent" → urgent. First match wins; default is general. Deterministic routing enables unit tests without API keys.

Token estimation uses len(prompt) // 4 — a rough budget check before sending to a paid endpoint.

Production AI System Design Relevance

Prompt engineering sits before every LLM call. Day 9’s ReAct planner will consume prompts built this way. Templates version in Git; few-shot sets A/B test in staging; routing pre-filters before expensive inference.

Workflow

• Request flow: POST /classify (one message) or /demo (five messages); Pydantic validates length.

• Execution flow: format examples → render template → classify → record metrics.

• Data flow: each classification becomes a recent_events entry (max 20); aggregates update category counts and timing.

• State changes: prompts_rendered, classifications, per-category counts, demo_runs, token averages advance.

Production Integration

• Architecture fit: stateless API; persist prompt versions externally.

• Enterprise patterns: template registry, few-shot versioning, category allowlists.

• Scalability: single uvicorn worker for in-memory metrics; externalize at scale.

• Observability: export category distribution, render latency, token estimates.

• Security: authenticate /classify; no secrets in templates; 500-char message cap.

Step-by-Step Implementation

GitHub Link

https://github.com/sysdr/production-ai-engineering-p/tree/main/lesson10/aiam-day10

template.py defines PromptTemplate and format_few_shot_examples(). classify.py holds CLASSIFICATION_RULES and classify(). metrics.py locks counters and returns deep-copy snapshots. service.py owns DEMO_MESSAGES and run_demo(). app.py serves the dashboard with auto-refresh.

Coding Highlights

Template rendering with validation:

for key, value in kwargs.items():
    text = text.replace("{{" + key + "}}", str(value))
if "{{" in text:
    raise ValueError(f"unresolved template variables: {missing}")

Few-shot formatting:

return "\n".join(
    f"Message: {message}\nCategory: {category}\n"
    for message, category in examples
)

Unresolved placeholders caught at render time prevent silent prompt corruption in production.

Working Demo Link:

Production Considerations

• Scalability: externalize metrics and prompt versions beyond one replica.

• Security: no API keys in repo; .env gitignored; message length capped.

• Monitoring: alert on skewed category distribution or zero classifications.

• Testing: test_prompt.py for template/routing; test_api.py for HTTP and metrics.

• Failure handling: unresolved template vars raise; unknown keywords default to general.

Validation

• Verification: /health green; /demo completes five classifications; dashboard serves.

• Testing strategy: 9 pytest cases across template, classify, service, and API.

• Success criteria: tests pass; after demo, all category counters and demo_runs are non-zero.

• Expected outputs: charge → billing; API 500 → technical; phone → general; URGENT down → urgent.

• Benchmarks: sub-second demo; ~73 tokens per rendered prompt.

• Production checklist: single replica, auth on /classify, alert on zero category coverage.

Key Insights

Prompt quality depends on template structure and few-shot diversity, not length alone. Demo messages must hit every category — a zero counter signals a broken path. Common mistakes: missing placeholder validation, stale few-shot examples, routing rules that overlap.

Important Classes

• PromptTemplate — renders and estimates tokens.

• MetricsStore — thread-safe counters and recent events.

• PromptService — API runs and demo orchestration.

Key Methods

• PromptTemplate.render() / estimate_tokens() — build and size prompts.

• classify() — keyword routing to category.

• MetricsStore.record_classification() / snapshot() — update and read dashboard state.

Success Criteria

aiam-day10/ runs from start.sh and demo.sh alone. All tests pass. Dashboard metrics non-zero after demo. Cleanup removes containers and artifacts.

Real-World Examples

• A support triage system renders few-shot prompts per queue, routes billing keywords to a payment specialist agent, and logs category distribution for SLA dashboards.

• A content moderation pipeline templates policy instructions, injects labeled violation examples, and pre-routes urgent keywords to human review before LLM scoring.