Grafana Applies SLOs and Error Budgets to AI Agent Behaviour

Grafana Labs has published an article describing how evaluations, service-level objectives (SLOs) and error budgets can be applied to the behaviour of AI agents rather than only to conventional infrastructure, with Craig Risi covering the proposal for InfoQ.
The core argument is that ordinary application observability cannot answer the question that matters most for an agent. Traditional metrics can show that an agent responded quickly, produced no technical errors and consumed a certain number of tokens, but Grafana notes that none of these necessarily indicates whether the answer was correct, grounded in the supplied information, or useful to the user. Its alternative is to evaluate the behaviour itself and convert the result into a service-level indicator (SLI) that can be measured against an SLO.
Evaluation comes first. An evaluation inspects an agent interaction and decides whether it met a specific behavioural criterion. Grafana's examples include whether a response was grounded in its supplied context, whether it fulfilled the user's request, whether it contained toxic content, whether it exposed personal information, and whether it succumbed to prompt injection. Evaluations can be performed by another language model acting as a judge, but deterministic checks such as regular expressions can also be used where a behaviour can be identified without an LLM.
Grafana argues against compressing this into a single agent quality score. Quality is multidimensional: an agent might be well grounded but expensive, helpful but occasionally toxic, or accurate but slow. Keeping the measurements separate lets a team set different objectives for the behaviours that matter to a particular application, instead of letting important failures disappear inside an aggregate number.
Once a behaviour is measurable, standard site reliability engineering (SRE) concepts apply. Grafana's worked example is an SLO requiring that 95% of conversations be judged as fulfilled over a 30-day window. The remaining 5% is the error budget, the amount of unacceptable behaviour the team has explicitly decided it can tolerate in that period. Grafana's existing SLO tooling defines an objective through an SLI, a target and a time window, with the error budget derived from the gap between the target and 100%.
The value lies less in the percentage than in the decision model it creates. While error budget remains, a team can experiment with prompts, models or agent behaviour. As the budget is consumed, priority can shift from introducing changes to improving quality. This is the same reliability-versus-velocity trade-off SRE teams have used for years, now applied to behaviour rather than uptime.
The approach is aimed at gradual degradation rather than dramatic failure. A conventional alert can catch a sudden spike in failed requests, but an agent can keep answering successfully while its answers become quietly less useful. A model change, prompt edit, retrieval change or shift in user behaviour might reduce quality by a fraction of a percentage point per day. Monitoring error-budget burn is meant to make that drift visible: a single poor interaction is insignificant against a 30-day objective, while a persistent decline steadily consumes the budget. The signal is not whether one execution failed but whether aggregate behaviour is moving outside an agreed reliability envelope.
For developers, the practical difficulty is less in constructing the SLO than in deciding what acceptable behaviour means. Availability targets benefit from decades of operational experience; there is far less consensus on whether an agent should reach 85%, 95% or 99% fulfilment or groundedness. Grafana's suggestion is to establish a baseline through evaluations and use historical results to recommend a target, then tighten it as the agent matures.
The proposal sits inside a wider change in observability's scope. Grafana's Agent Observability work grew out of monitoring its own Grafana Assistant, where changes to prompts, models and tools made traditional monitoring insufficient. In community discussions, Grafana engineers have drawn a distinction between offline evaluations, which validate changes against known scenarios, and online evaluations, which assess behaviour against real production interactions. This suggests a shift from asking whether a system is available and within technical limits to asking whether its outputs stay within acceptable behavioural boundaries, creating a feedback loop between evaluation, observability and engineering change.
A clear limitation is that an SLO is only as good as the measurement beneath it. If an LLM judge consistently misclassifies responses, the error budget can convey false precision. A poorly designed evaluation can also reward an agent for technically satisfying a criterion while missing the user's actual intent. That leaves organisations needing representative evaluation datasets, clear definitions of acceptable behaviour, deterministic checks where possible, human review for ambiguous cases and continuing validation of the evaluators themselves. As Grafana frames it, the SLO supplies the governance mechanism; it does not make the underlying quality measurement correct.
The work is part of a broader industry effort to make agent behaviour measurable. Anthropic has developed formal evaluation practices using defined tasks, repeated trials and multiple graders, and its distinction between pass@k, succeeding at least once across several attempts, and pass^k, succeeding consistently across all of them, is directly relevant to production agents. OpenTelemetry is working a layer below, on standardised semantic conventions for agent telemetry so traces, metrics and logs give a consistent basis across frameworks and vendors. Read together, evaluations decide whether an agent behaved correctly, observability supplies the evidence, and SLOs set the operational threshold.
Why it matters: teams putting AI agents into production need a way to detect degradation in behaviour, not just availability, and error budgets give them a familiar governance lever for deciding when to ship changes and when to stop and fix quality. The main caveat for those teams is that a target is only as trustworthy as the judges and checks behind it, so investment in evaluation design and human review is not optional. Expect this pattern to spread as agent observability tooling matures, with standardised telemetry conventions likely making such objectives easier to compare across frameworks and vendors.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
Grafana Labs is proposing a familiar reliability engineering approach for one of the less familiar problems in modern software: how to measure whether an AI agent is actually behaving well. By Craig Risi