AivexaNewsSearch
AI news for builders and product teamsChecked every hour

How Circle K accelerated DevOps incident triage to under 15 minutes with Amazon Bedrock

Collected Nov 2, 2026

Circle K's DevOps team reduced mean incident triage time from 45–60 minutes to under 15 minutes with an automated triage system built on Amazon Bedrock. The company, which operates more than 17,200 convenience and fuel retail stores across 27 countries and runs point-of-sale, loyalty and EV charging applications on AWS, deployed the AgentCore version of the system in April 2026. Cloud DevOps Engineer Marek Klatowicz co-wrote the account with AWS. The stated 67–75 percent reduction is based on the team's internal incident-response metrics.

The problem it addresses is familiar to on-call teams. Grafana alerts across several AWS accounts flag container restarts, elevated error rates, latency spikes and failed deployments. Investigating each one meant pulling CloudWatch logs, reading traces in Amazon OpenSearch Service, checking ECS task status, and cross-referencing symptoms against runbooks in GitHub — all before remediation could start. Two factors drove the R&D effort: the same queries repeat for every incident, and triage quality varied with how well the on-call engineer knew the affected service.

Circle K evaluated self-managed open source agent frameworks, which would have meant owning model hosting, prompt versioning, tool orchestration and security controls. Bedrock gave it model access through one API with swappable models, and fit existing IAM policies, VPC configuration and logging pipelines. Bedrock Knowledge Bases, the managed retrieval-augmented generation service, ingests and retrieves internal product cards describing services and their dependencies, so no separate vector database is needed. AWS states that Bedrock does not share customer data with model providers or train foundation models on it, which the team says met its requirements for feeding in production logs.

The first architecture used Amazon Bedrock Flows as the orchestrator over three specialized agents. A Grafana alert publishes to Amazon SNS, which forwards to an Amazon SQS queue; a Lambda function polls the queue and invokes the Flow. Two agents then run in parallel: one queries ECS task status, Application Load Balancer target health, and CloudWatch metrics and logs across workload accounts managed in AWS Organizations; the other queries OpenSearch for application logs and distributed traces and has exclusive access to the knowledge base. Keeping OpenSearch isolated keeps each context window manageable, since it returns large log volumes. A third agent, for GitHub, runs only after those results return, so it knows which error or component to search for across code, infrastructure definitions and recent commits. A final model call with no tools attached produces a structured summary: probable root cause, evidence, affected service and owner, and suggested fixes, formatted to be scanned in under two minutes and posted to Slack. Alert to Slack took 5–15 minutes.

A worked example shows the output. Faced with HTTP 5xx errors above 5 percent for three consecutive minutes, the system found an exhausted database connection pool, logs showing connection refused from the service to its PostgreSQL database, and code that opened a new connection per request rather than reusing them. The summary correlated the spike with a marketing campaign that began ten minutes before the alert and recommended Amazon RDS Proxy or connection reuse plus an Auto Scaling rule.

As incident data and context grew, the Flows design strained. Aggressive log filtering, shorter context windows and splitting services into separate Flows helped but did not keep pace, and some investigations needed longer than a Flow node allowed. The team containerized its agent logic and moved to Amazon Bedrock AgentCore Runtime, consolidating the three agents into one agent in a single session of up to 8 hours; results return to Lambda within its 15-minute timeout. AgentCore Gateway exposes OpenSearch, Knowledge Bases and GitHub as tools the agent invokes, centralizing access control. Session persistence lets the agent run a second, targeted log pass without losing prior context; OpenTelemetry traces surfaced in CloudWatch show per-step timing and knowledge base queries for prompt tuning; and containerized agents can be built with a range of frameworks.

Results include summaries in 5–8 minutes for straightforward incidents and under 15 minutes for complex multi-service ones that previously took over an hour. Every engineer gets the same summary format, escalations carry context, and false alarms can be dismissed once analysis shows healthy services. Four lessons stand out: pre-query logs in Lambda and pass actual error lines rather than letting the agent query CloudWatch itself; default Knowledge Bases configuration performed best, with a well-designed system prompt and tuned temperature for generation; run the code search after log and resource analysis to narrow it; and keep prompts, tool definitions and analysis steps independent of the orchestration layer, which is what let the move to AgentCore happen without rewriting triage logic.

Next, Circle K plans to integrate AWS DevOps Agent so the system can perform simple remediations — restarting a task, scaling a service, rolling back a deployment — with engineer approval in Slack, and to add Jira and Confluence context plus runbooks in the knowledge base.

Why it matters: Teams whose incident response is dominated by context gathering rather than fixing have a concrete reference architecture here, and the migration path carries a lesson of its own: separating analysis logic from orchestration made a platform swap comparatively cheap. The 8-hour session limit and the Lambda 15-minute ceiling are the practical constraints to design around, and a Lambda invoking a long-running agent session is a pattern worth stress-testing before adopting. Human approval before automated remediation, as Circle K plans it, is likely to remain the norm for production changes.

Read at AWS Machine Learning Blog

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

Circle K's DevOps team cut incident triage from 45-60 minutes to under 15 by building an agentic triage system on Amazon Bedrock. This post walks through the architecture, the initial Amazon Bedrock Flows implementation, the migration to Amazon Bedrock AgentCore, and the lessons learned from scaling it.