From prototype to production: Engineering agents with Agent DLC

AWS published a post on its Machine Learning Blog introducing the Agent Development Lifecycle (Agent DLC), a framework for deciding when an LLM agent is ready for production. The post argues that unit tests, CI/CD and passing builds apply to agents but are not sufficient, and it illustrates the gap with a measured reference implementation: an internal HR policy assistant answering questions over a document knowledge base. The accompanying AWS Samples repository includes infrastructure deployment, the agent, custom evaluators and scripts for baseline evaluation and prompt-change re-evaluation (09-run-eval.sh and 10-optimize-prompt.sh), plus optional checks for judge repeatability, model comparison, cost and latency. The scores come from that reference implementation and are explicitly not a customer outcome or a benchmark result, and they can vary across runs and repository versions.
The post names three properties of agents that break conventional practice. Agents are non-deterministic, so a single assertion per test case measures one sample of a distribution. Prompts, tool descriptions, retrieval settings and memory configuration are effectively source code, but behavior cannot be read off the text of a prompt. And model endpoints, knowledge bases and external APIs can drift independently of the repository, so nothing in the repo changes when behavior does. Its proposed answers: run each example k times and judge on pass rate and failure distribution, version every prompt and configuration change with an evaluation attached and keep the lineage, and re-run the standard on a schedule, alerting on deviation from the baseline.
The empirical section is the most useful part for teams designing evaluation suites. Three golden questions were scored on five measures: groundedness (decompose the answer into claims and check each against retrieved sources), chunk-level source precision (is each retrieved chunk as a whole necessary), fact-level source precision (decompose chunks into individual facts and check each), retrieval coverage (does any retrieved chunk answer each sub-question), and response precision (how many claims in the answer are necessary). Chunk-level precision was 1.00 for all three questions. A release gate based on it alone would have accepted every answer. Fact-level precision was 0.59, 0.53 and 0.09 respectively; on the third question, about sick-leave certificates, retrieval coverage was 0.00 and groundedness 0.39. Roughly nine in ten retrieved facts were irrelevant to what was asked.
A prompt-only change (three added constraints: ground strictly in retrieved content, ignore irrelevant retrieved content, keep answers focused) produced opposite results. On the performance-review question groundedness rose from 0.83 to 0.87, fact-level precision from 0.59 to 0.85 and response precision from 0.58 to 0.73. On the sick-leave question groundedness fell from 0.39 to 0.31 and fact-level precision from 0.09 to 0.04. The diagnosis was a knowledge-coverage problem: the corpus had no dedicated sick-leave policy document, only brief references inside a leave-management policy, so instructing the model to ground strictly could not supply missing information. The recommended next step was to fix policy coverage and re-check retrieval rather than keep editing the prompt. This is the kind of finding that argues for fact-level or claim-level retrieval metrics over chunk-level ones in any RAG evaluation harness: chunk-level precision says a passage is on topic, not that it answers the question.
A second result shows why quality scores do not cover responsibility constraints. A benefits-enrollment answer passed with groundedness 0.90 while asserting that the user's preferred contact method, taken from their profile, was social media. No such profile existed; the policy document lists contact preference as an option within a process step, and a noisy public FAQ corpus contained a matching pair. The model merged the two into a false personal claim, which the groundedness score did not catch.
Agent DLC itself is one lifecycle with two loops around a single gold standard, defined as the bundle of verifiable criteria, their thresholds and approved examples, approved by business owners. The outer lifecycle has six activities: Define, Build, Evaluate, Release, Observe and Feed Back. Inside it runs a faster loop: score, inspect the path, change one thing, re-evaluate. The split matters because the inner loop improves an agent against a fixed standard while the outer loop changes the standard; conflating them leads teams to patch a missing criterion with a prompt edit or to widen the standard mid-cycle and lose the ability to attribute score movement. The post credits Xia et al.'s Evaluation-Driven Development of LLM Agents as prior work and cites a Fotor case study on Amazon Bedrock AgentCore runtime reporting workflow overhead falling from minutes to seconds excluding model inference, with thousands of concurrent active sessions supported; quality evaluation there is named as the next step.
Criteria are organized across five dimensions (domain understanding, quality, responsibility, cost, performance) and three tiers that determine whether they block release: hard constraints, expressed as a boolean per run; release gates, expressed as pass-rate thresholds over k runs; and monitoring-only criteria tracked until their evaluator is calibrated. A high score in one area should never offset a hard-constraint violation. Guidance for the inner loop includes matching evaluators to criteria (code for unambiguous rules, telemetry for cost and latency, load tests for capacity, model-based judges calibrated against human-reviewed examples for judgment calls), repeating judge evaluations to measure scoring variability, locating failures at session, trajectory and step level, and making release decisions per intent, with blocked and refused requests included in evaluation.
Getting started, the post suggests a one-week baseline: days 1–2 for the business owner and engineers to agree criteria, tiers and thresholds and prepare 20–30 golden examples; days 3–5 to implement evaluators and publish a baseline including failures; week two to change one thing and re-evaluate, then wire evaluation into release and schedule it for drift. On AWS it points to Amazon Bedrock models, AgentCore Evaluations with built-in and custom evaluators, AgentCore Observability with CloudWatch, and AgentCore runtime, Gateway, Identity and Memory.
Why it matters: teams shipping agents are often gated on aggregate metrics that can pass a broken answer, and this example gives a concrete, reproducible counter-case. Developers building RAG agents should expect to invest in claim- and fact-level evaluators, repeated runs and trace-level diagnosis rather than single-sample assertions, and product owners should own the criteria and tiers that decide release. My inference is that the largest organizational change is treating prompts and retrieval configuration as versioned artifacts with attached evidence, and budgeting evaluator cost and latency as first-class constraints rather than an afterthought.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
Agents stall between demo and release when teams lack a repeatable way to measure how good an agent is and whether the last change helped. This post covers three agent properties that break traditional CI/CD, a measured case where an aggregate metric hides a coverage gap, and Agent DLC: one lifecycle, two loops, and one explicit standard for judging every change.