AivexaNewsSearch
AI news for builders and product teamsChecked every hour

UAEF: A framework-agnostic system for closed-loop evaluation of AI agents

Collected Nov 11, 2026

AWS published the Universal Agent Evaluation Framework (UAEF) on the AWS Machine Learning Blog, an Apache 2.0 open-source project in the awslabs/uaef GitHub repository. It targets a concrete friction point: agents built on LangGraph, LangChain, Strands Agents or Amazon Bedrock AgentCore each record messages, tool calls and handoffs in their own format, so evaluation logic written for one framework has to be rebuilt when teams switch or run several at once. AWS says customers across 12 organizations in telecom, healthcare and financial services used UAEF to score their agents on a shared metric set and compare versions before release.

The architecture rests on a normalise-first, score-second split. An adapter reads an agent's native execution record and converts it into a single canonical representation called the AgentTrace, which is the only thing downstream scoring touches. Adapters exist for LangGraph, LangChain, Strands Agents, Bedrock AgentCore, Langfuse, and other frameworks through a generic JSON adapter. Agents can run locally, behind an HTTP endpoint, or on a managed runtime. Traces arrive either by handing over an existing record from a log or observability store, or by letting UAEF invoke the agent itself on each dataset input using a runtime ARN or HTTP endpoint. Because every metric and report operates on the AgentTrace, an agent built on one framework can in principle be compared with one built on another on a single scale.

The evaluation engine scores across 7 dimensions: tool calling (correct tools, sensible order, valid arguments), response quality (relevance, completeness, accuracy, grounding), responsible AI (safety, bias, toxicity, prompt-injection resistance), performance (latency, token use, throughput), multi-turn (context retention, coherence, goal completion), multi-agent (delegation and division of work), and reasoning (consistency and fallacies in the chain of thought). Each dimension draws on one or more of 36 built-in metrics, and every metric returns a normalised score between 0 and 1, so dimensions and the overall result share one scale.

Metrics fall into two families. Deterministic metrics compute a score directly from the trace and ground truth and reproduce exactly on repeated runs. LLM-as-a-judge metrics handle qualities without a closed-form definition, such as relevance or safety, by prompting a judge model on Amazon Bedrock. Custom metrics and RAGAS and DeepEval integrations are also supported. Judge metrics feed agent output back into a prompt, so UAEF treats that content as untrusted: the rubric travels separately, content is escaped so it cannot break out of its delimiter and be read as instructions, and each judge response must cite a verbatim span of the scored content. A score whose evidence cannot be found is rejected rather than recorded. AWS explicitly frames judge scores as evaluation signal rather than a safety decision, recommending Amazon Bedrock Guardrails for runtime enforcement on the agent's own invocation path.

Ground truth is optional and only used by metrics that compare against a reference. Evaluating tool calling properly requires the dataset to include expected tool calls: which tools, in what order, with what arguments. Before storage, UAEF screens traces for personally identifiable information such as email addresses, phone numbers and credit card numbers, and PII redaction can be turned on for saved copies. UAEF also automatically skips metrics that need a reference answer when no ground truth is present, so partially labeled datasets and unlabeled production traffic remain evaluable.

Persisted evaluations become experiments. Calling evaluate() or batch_evaluate() with persist=True creates a new experiment with an optional name and objective. Amazon DynamoDB holds one row per experiment with listing and comparison metadata: evaluation count, average score per metric, and an overall average. Amazon S3 holds a JSON file with full results, including per-metric scores and the judge's reasoning. An experiment contains runs, and each run records the agent design under test: framework, model, tools, system prompt and parameters, optionally a configuration snapshot and a Git commit. A run can be marked as the experiment's baseline, tying each score to the change that produced it.

Comparison works on stored per-metric averages. For metrics two runs share, the engine calculates absolute and percentage change and classifies each as regression, improvement or unchanged using a configurable threshold defaulting to 5 percent. Regressions get severity bands: minor (5–10 percent), moderate (10–20 percent), severe (over 20 percent). Grouping by dimension is the useful part in practice, since it reveals whether a prompt change, say, moved tool calling but left response quality flat. Several runs can be compared against one baseline in a single call, and the web UI's Compare tab calls the same library engine, so interface and API results agree.

Reporting comes at three levels. The results view shows overall score, test-case count, framework, queries that failed to run, per-metric averages with radar and bar charts, and a per-test-case table that expands to show full responses with dimension and metric scores, including per-turn breakdowns for multi-turn conversations. A detailed statistical report is generated on demand from stored S3 results and includes confidence intervals, outliers, score distributions, metric correlations, a responsible AI summary, failing test cases, and recommendations pointing at the lowest-scoring dimensions and most variable metrics. It is computed from scores alone without an LLM, so it is deterministic. Separately, the Python library can pass a report to a Bedrock model for insights covering strengths, critical weaknesses, root-cause hypotheses and prioritised recommendations, or for multi-run comparisons the overall trend and likely causes of regressions.

Two interfaces share one engine: a Python library for notebooks, scripts and CI/CD pipelines, and a deployable AWS service with a low-code web UI. Datasets are CSV or XLSX, one row per test case, with only Question required; Answer is optional and accepts aliases such as Query and Expected_Output, matched case-insensitively. Rows sharing a lowercase session_id are sent in order to a single agent session, and the endpoint must maintain its own conversation state because UAEF sends each turn with its session ID rather than replaying history.

Running the library alone creates nothing, but persisting writes to DynamoDB and S3, defaulting to uaef-experiments and uaef-results overridable via the UAEF_DYNAMODB_TABLE and UAEF_S3_BUCKET environment variables. Service teardown needs care: the DynamoDB tables, S3 buckets and Cognito user pool are created with a retain policy, so deleting the stack leaves them and their contents, and their storage cost, in place.

A likely trade-off is cost and latency from judge-based metrics, since every LLM-scored test case means a Bedrock invocation, which is why AWS's guidance is to start from the metrics that match the agent's scenario and add more as questions arise. The evidence-citation rule on judges is a sensible hedge against an agent influencing its own score, but it is a mitigation, not a guarantee.

Why it matters: teams running agents in production have struggled to answer whether a prompt, model or tool change actually helped, and cross-framework shops have had to rebuild scoring per framework. UAEF's adapter-plus-canonical-trace design, its baseline and regression classification, and its Apache 2.0 release mean evaluation can plausibly become a shared CI step rather than framework-specific glue. The open question, my inference, is how faithfully the generic adapters capture framework-specific semantics such as complex handoff graphs, which would matter if comparisons across frameworks are to be trusted.

Read at AWS Machine Learning Blog

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

The Universal Agent Evaluation Framework (UAEF) is a framework-agnostic system for evaluating AI agents. UAEF converts each framework's traces into one canonical format, scores them with 36 built-in metrics across 7 dimensions, and stores results so teams can compare agent versions and catch regressions.