AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Datadog Brings Production Experimentation to AI Agent Observability

Collected Nov 17, 2026

Datadog has extended its Agent Observability platform with a capability called Online Experiments, which lets engineering teams compare different versions of an AI agent against live production traffic rather than relying only on predefined evaluation datasets. The move, reported by Craig Risi for InfoQ, targets a familiar problem in agent development: a change that scores well in a controlled test can behave very differently once it meets unpredictable user requests.

The core idea is to bring controlled experimentation into the same place teams already watch their agents. Online Experiments combines agent evaluation scores with operational measurements — latency, token consumption, execution cost and errors — so teams can judge whether a change to a model, a prompt, a retrieval strategy or a tool actually improves the agent rather than just shifting its behaviour.

Offline evaluation is not going away. Teams can still run versions against predefined datasets, inspect responses and catch known regressions before anything ships. The limitation is that such evaluations are only as good as the datasets and scoring criteria behind them. New user workflows, unexpected requests and changing tool responses can surface weaknesses that existing test cases never anticipated.

Online experiments address that gap by testing against real interactions. A control version and a modified variant are compared by randomly distributing production traffic between them, which reduces the influence of differences in the kinds of requests each version happens to receive. In practice that means the comparison is less likely to be skewed by one variant drawing an easier batch of users. It also helps teams discover new scenarios worth promoting into their offline regression datasets.

The mechanics rely on two existing pieces of Datadog machinery. Agent Observability evaluators are wired into the Experiments capability, so quality scores sit next to conventional operational metrics. Teams can measure response quality, accuracy and completeness while tracking execution duration, token consumption, estimated cost and workflow errors. Feature Flags handle traffic allocation, letting engineers expose a change to a small percentage of users before widening the rollout. This suggests the workflow is meant to feel like a standard progressive rollout, with evaluation data attached to each step.

Connecting results to individual traces is arguably the most useful part for debugging. Engineers can inspect tool calls, the arguments supplied and the results returned, which is where the reason for a behavioural difference usually lives. A headline score can hide a lot; a trace usually cannot.

Datadog illustrated the approach with an experiment comparing GPT-5.6 Luna against GPT-6 Luna, splitting traffic evenly between the two. An LLM-based evaluator scored investigation quality from one to five, weighing accuracy, completeness and whether conclusions were supported by the available evidence. The reported results showed a 44.3% reduction in estimated cost for the newer model, with quality scores close together at 3.84 and 3.86 respectively.

The operational picture was less clean. The newer model showed estimated increases in execution duration and workflow errors, though the confidence intervals were too broad to establish definitive regressions. Trace inspection revealed a more concrete issue: the newer model supplied invalid tool parameters more often, usually recovering after retries. That behaviour was not sufficiently visible in the overall quality score. Datadog used the finding to improve tool handling before proceeding with the model change, and the failed calls became candidates for additional offline regression tests.

That example is a useful illustration of why evaluation scores alone can mislead. A quality score measures whether the final answer looks good, not how much machinery was needed to get there. Extra retries inflate token spend and latency, and they add failure paths where a user could see an error if the recovery does not succeed. A likely trade-off for teams adopting this is that the richer operational view demands more careful statistical reading — as Datadog's own broad confidence intervals show, distinguishing a real regression from noise requires enough traffic and a disciplined experiment design.

Datadog is not alone here. LangSmith and Langfuse offer agent tracing and evaluation, while experimentation platforms such as LaunchDarkly support controlled feature rollouts. Datadog's pitch is that it ties these concerns together inside its existing observability ecosystem, linking evaluation results directly to production experiments and the execution evidence behind them, rather than asking teams to stitch separate tools together.

The broader proposition is a feedback loop between production and development testing. Unexpected failures in live traffic become new regression cases, and later experiments show whether the fix actually works. For agent engineering, that reframes the central question from whether an agent passes a predefined evaluation to whether it performs better under real conditions.

Why it matters: Teams building agents now have a way to run A/B-style comparisons on live traffic with quality and cost metrics attached to the same experiment, which should make model and prompt changes easier to justify internally. The Datadog example suggests the biggest wins may come from trace-level findings — like malformed tool arguments — that aggregate scores miss entirely. Expect production experimentation to become a routine part of agent release processes, though teams will need enough traffic volume and statistical rigor for the results to mean anything; that last point is my inference, not a claim from the report.

Read at InfoQ · AI, ML & Data Engineering

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

Datadog is extending its Agent Observability platform with Online Experiments, allowing engineering teams to compare AI agent versions against real production traffic rather than relying exclusively on predefined evaluation datasets. By Craig Risi