The agent evaluation playbook: Built-in, DeepEval, and AutoEval on Amazon Bedrock AgentCore Evaluations

Amazon Bedrock AgentCore Evaluations is being positioned as a single place to measure whether a deployed agent actually accomplishes what the business needs, rather than merely running without errors. The AWS Machine Learning Blog post frames the problem around visibility: teams typically spot-check outputs, watch latency and gather user feedback, but none of that reveals whether an agent finished the user's task, held context across a long session, exposed personally identifiable information or skipped required steps. AgentCore answers this with a catalog of evaluators drawn from three sources, three ways to run them, and a custom path for behaviors nothing in the catalog covers.
The catalog splits into built-in, third-party managed and custom tiers. Built-in evaluators are authored and quality-tested by AWS and referenced by string IDs such as Builtin.GoalSuccessRate and Builtin.Helpfulness, alongside 13 others. Because AWS owns the judge prompt and the model choice, they are the low-effort starting point for common dimensions. Third-party managed evaluators come from DeepEval and AutoEval, two existing open-source evaluation libraries. In managed mode AgentCore hosts the library code, selects the judge model and runs inference on its own capacity, so a team already using those libraries in development can carry the same metrics into production without standing up a separate evaluation service. Custom evaluators take a built-in or third-party evaluator as a base and let you substitute your own model and edit the prompt or rubric; once created, a derived evaluator behaves like any other.
The built-in set is organized into five groups. Response quality holds Correctness, Faithfulness, Helpfulness, ResponseRelevance, Conciseness, Coherence, InstructionFollowing and Refusal. Task completion is covered by GoalSuccessRate, which reviews an entire session for whether the user's goal was met. Tool use has ToolSelectionAccuracy and ToolParameterAccuracy, checking that the agent chose the right tool and passed correct arguments. A skills group offers SkillSelectionAccuracy and SkillInstructionFollowing for agents that load reusable skills. Safety includes Harmfulness and Stereotyping. Beyond those sit three ground-truth trajectory checks, TrajectoryExactOrderMatch, TrajectoryInOrderMatch and TrajectoryAnyOrderMatch, which compare the agent's actual tool sequence to an expected one, for cases where the route taken matters as much as the output.
DeepEval contributes metrics across four groups: TaskCompletion, GoalAccuracy and ConversationCompleteness for task completion; TurnRelevancy, KnowledgeRetention and Summarization for conversational quality; ToolUse; and Bias, Toxicity and PIILeakage for safety. These lean toward multi-turn agents. ConversationCompleteness is meant to catch an agent that answers the latest message while dropping earlier requests, and KnowledgeRetention catches one that forgets something said several turns back. AutoEval adds narrower signals: Security, which flags malicious output such as instruction injection attempts or system-prompt leakage; Possible, which catches agents that wrongly declare a valid task impossible; and Humor, for consumer-facing agents where personality is part of the product.
The post recommends working backward from business outcomes rather than picking metrics by name. In its framing, beginning with something like Helpfulness produces numbers with little insight because the label, not the actual success criterion, drove the choice. The suggested method is to define success in business terms, decompose it into observable behaviors, then map each behavior to an evaluator at trace or session level. Trace-level metrics fit per-turn or per-tool-call checks; session-level metrics fit outcomes that only make sense over a whole conversation. This is a sensible discipline for anyone who has watched an evaluation dashboard produce green scores that say nothing, since it forces each metric to justify itself against a stated failure mode.
Multi-dimension evaluators address a gap in single-score judging. A support reply might need to resolve the issue, follow refund and data policy, and stay professional; one judge weighing all three collapses them into a number that hides which criterion drove the result. A multi-dimension evaluator is a custom LLM-as-a-judge given a list of dimensions, each with its own name, instruction and numeric or categorical scale. The judge reads the trace once and returns a separate result per dimension, with no composite score. That lets teams gate releases on individual criteria, so an improvement in tone that regresses policy compliance still fails. The post recommends keeping criteria in separate evaluators when a built-in or third-party metric already covers one, when criteria need different levels or judge models (dimensions within one evaluator share both), or when only some criteria depend on ground truth such as an expected response or assertions, since reference-free criteria can then keep running on live traffic.
Three modes cover the lifecycle. On-demand evaluation is the tightest loop: a session is passed to client.evaluate() with an evaluator ID and session spans, and the response returns a value and explanation per result. Every evaluator, including third-party ones, works here. Online evaluation attaches to a live agent runtime and samples real traffic at a configured rate, with built-in and third-party evaluators mixed in one configuration. Batch evaluation runs a fixed evaluator stack against a curated set of sessions, functioning as a regression suite before a prompt change or model upgrade ships. All three share the same evaluator IDs, so moving from development testing to production sampling is a configuration change rather than a rewrite. This suggests the practical benefit is less about the metrics themselves than about avoiding a parallel evaluation pipeline that drifts from what developers test against.
Ground-truth-dependent evaluators run in batch only, because live traffic has no reference answer or expected path. The post sketches starting stacks per agent type. Customer support begins with GoalSuccessRate, ConversationCompleteness and PIILeakage. Multi-step research starts with TrajectoryAnyOrderMatch, GoalAccuracy and KnowledgeRetention, adding Correctness or PIILeakage as needed. Regulated domain content, such as medical, legal or financial, starts with Correctness, PIILeakage and AutoEval Security. The full catalog plus any custom evaluators in an account comes back from the ListEvaluators API, and exact judge prompts, scoring scales, trace or session levels and input requirements are documented in a prompt template library.
Why it matters: teams shipping agents on Bedrock get one evaluation surface that spans AWS-authored, open-source-derived and custom judges, with the same IDs from local debugging through production sampling. For developers, the practical shift is that evaluation becomes an API call and a configuration rather than a separate toolchain, and per-dimension scoring makes it possible to fail a release on a single criterion. The open question is cost and latency at scale: online sampling means continuous judge inference on real traffic, and a likely trade-off is that richer evaluator coverage raises that bill, so sampling rates and stack size will need deliberate tuning.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
When you deploy an agent, whether it is working is really several questions: did it finish the task, stay on topic, leak data, or skip steps? This post shows how to evaluate agents on Amazon Bedrock AgentCore with built-in, DeepEval, and AutoEval evaluators, and how to match each behavior that matters to the right evaluator at the right level.