Building trust in AI agents: A production-ready evaluation framework with Amazon Bedrock AgentCore

AWS's Sales, Marketing, and Global Services (SMGS) Operations team published an account of the evaluation framework it runs behind an internal multi-agent system that serves AWS Field Sales worldwide. The system lets sellers and sales leaders ask business questions in plain English through Amazon Quick, with answers scoped to each user's territory and role-level security. The post's central claim is that the hardest failures in agent systems are not crashes but fluent, confident answers that are quietly wrong, and that catching them requires evaluation running continuously across the whole development lifecycle rather than as a final QA step.
Behind the conversational front end sit several specialized sub-agents. The Performance Insights Agent turns complex sales data into natural-language business intelligence narratives. The Customer Insights Agent synthesizes product usage, support history, contract status and engagement data into a 360-degree customer view. The Data Insights Agent is a text-to-SQL agent that lets account teams query structured analytics data without writing SQL. Direct dashboard access rounds out the experience. The same system serves both front-line sellers and leaders preparing Weekly Business Reviews.
The motivating examples come from pre-launch testing of the Customer Insights Agent. It reported $410.08 of a customer's three-month revenue on a service as "$410.08 Million." It labeled figures as Sales Revenue excluding Adjustments when they were plain Sales Revenue. It produced an Annual Recurring Revenue estimate that appeared nowhere in the data, and presented its own calculations as sourced facts. Monitoring stayed green throughout, because it confirmed the system had responded, not that the answer was right.
The team also measured the cost of mismatched releases. When a data change, a tool change and a paired prompt update are meant to ship together but land apart, the system runs in an inconsistent state until everything catches up. In one pre-production test, that mismatch window alone dropped the pass rate from 78 percent to 38 percent. Weekly upstream data refreshes mean risk grows even without any release.
The post frames agents as a third layer of operational complexity after deterministic software and probabilistic ML models. Agents are non-deterministic, follow different reasoning paths for identical inputs, and cascade decisions across routing, tool selection and synthesis, so one bad step can corrupt everything downstream while latency and error rates stay flat. Evaluation is split into five dimensions: safety, quality, performance, cost, and operational control. Safety is described as the first priority, with no agent shipping or continuing to run if it cannot meet the safety bar.
AgentCore Evaluations, a capability of Amazon Bedrock AgentCore, is the connective tissue. It sits alongside Amazon Bedrock Guardrails for safety, Amazon CloudWatch for metrics, alarms and tracing, Amazon S3 and Amazon Redshift for evaluation persistence, Amazon Quick for dashboards, and AWS Lambda for custom evaluators.
The framework has nine stages, each tied to a question teams face as agents mature: defining what good looks like (Foundation), comparing versions (Experiment), checking for breakage at merge in CI, gating for production readiness in CD, canary scoring, adversarial testing, drift detection in live traffic, real-time health monitoring, and an automated feedback loop. The same nine stages apply simultaneously at three levels: each individual tool, each sub-agent end to end, and the multi-agent orchestration paths, with each level carrying its own golden set, evaluators and promotion criteria. This layered structure is the part other teams are most likely to under-copy. Most evaluation efforts stop at the agent level, which cannot see a weak tool underneath or a routing error that only appears when agents interact.
Stage 1 translates business outcomes into tracked metrics with explicit thresholds and alarm conditions. For sellers, success means accurate, current intelligence at the right time; for leadership, reliable review inputs without manual preparation. These map to entity resolution accuracy, grounding rate, faithfulness, hallucination rate, helpfulness, scope adherence, accuracy pass rate and response coverage, plus safety policies, p50, p90 and p99 latency targets per tool and per agent, cost-per-query budgets and SLAs tied to runbooks.
The team defines six failure types drawn from real production errors, any one of which marks an answer inaccurate: metric swap (right customer and period, wrong metric), wrong period (for example a year-over-year figure presented as month-over-month, or the reverse), wrong entity (conflated accounts or subsidiaries), pure hallucination (ungrounded figures with no traceable source), wrong symbols (correct metric, wrong currency or unit), and unauthorized calculation (derived figures not present in the source data). Accuracy pass rate is the headline number; per-dimension tracking is what makes it actionable. Response relevance, faithfulness, helpfulness, instruction following and tool selection accuracy are also tracked on the reasoning that a correct answer buried in text or arriving too late has also failed the seller.
The evaluator stack combines four kinds. AgentCore built-in evaluators cover relevance, faithfulness and helpfulness as a baseline. Six customized evaluators, one per failure type in the taxonomy, handle signals that native templates do not cover. Third-party managed evaluators from the open-source DeepEval and AutoEval libraries are also deployed. Deterministic evaluators check ground truth where the answer is known exactly, such as revenue, renewal date and contract status. Where no single correct answer exists, LLM-as-judge rubrics score reasoning quality. AgentCore Evaluations supports both ground truth and LLM-as-judge natively, which the team says let it stand up both paths without building a scoring engine.
The golden set is a version-controlled, continuously extended dataset with entries at all three levels. At the tool level, each critical tool has deterministic ground truth pairs; customer identification must resolve to the exact canonical entity or fail. At the agent level, real user queries are analyzed into four question types (comprehensive, reporting, reasoning and ambiguous), each requiring a different evaluation strategy and revisited monthly. At the multi-agent level, orchestration scenarios catch errors that only surface when agents interact. The set also carries safety, latency and cost reference cases so later stages can check all five dimensions against one dataset. It draws from production samples, every regression caught in staging or production, and deliberately constructed synthetic edge cases such as parent-subsidiary and ticker-symbol disambiguation.
A likely trade-off, as this suggests, is cost and maintenance: custom evaluators, LLM-as-judge scoring and monthly golden-set refreshes consume engineering time and inference budget of their own, and a six-way failure taxonomy is only as good as the labeling discipline behind it. A second, inferred risk is that LLM-as-judge metrics can drift alongside the systems they score, so the deterministic ground-truth checks carry much of the real assurance burden.
Why it matters: Teams moving agents from demo to production need evaluation that runs in CI, at release gates and on live traffic, because agent failures surface as plausible wrong answers rather than outages that latency and error-rate dashboards can catch. The example showed monitoring can appear healthy while accuracy is broken, and that mismatched data, tool and prompt releases can cut pass rates sharply, so coordinating releases and scoring canaries becomes part of the deployment job. For developers, the practical lesson is to instrument at tool, agent and orchestration levels with a failure taxonomy specific to the domain, rather than relying on generic relevance scores alone. The broader customer-facing product described here is an internal AWS system, not a published benchmark or benchmark of AgentCore against alternatives.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
Autonomous AI agents fail softly: latency stays flat while answer quality quietly erodes. This post shares a nine-stage evaluation framework, built on Amazon Bedrock AgentCore, that catches silent quality regressions across the full agent lifecycle, from first commit to live production.