AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Cutting our inference bill by 68% by being obsessive about optimization

Collected Oct 19, 2026

Cloudflare's AI team reported that the average cost of an AI code review on its internal reviewer fell from $2.43, averaged over its two most expensive weeks, to $0.77 — a 68% reduction — with the median review completing in 1m 32s instead of 4m 09s. The optimization push began around mid-September, after the cost per review had climbed from $1.19 in April to about $2.46 by late August despite no significant architectural change. About 40% of the savings came from changes to the reviewer's code; the rest came from testing which models to use and how much effort to let them spend.

The cost growth tracked model changes more than architecture. Weekday cost per review, holding the tier mix fixed, rose from $1.22 in March to $1.44 in June, $1.74 in July and $2.32 in August. A small part was mix: the most expensive tier accounted for 59% of reviews in March and 65% in August. The bigger factor was moving every reviewer onto Anthropic models at the start of August, which lifted cost per job from $1.38 to $2.03. The internal standards reviewer, Codex, landed on Claude Opus 4.7 for performance and reliability, going from about $0.25 to $0.88 per review — more than the coordinator agent itself. Tokens per review more than tripled, from 0.91 million to 2.99 million, even as the cache hit rate rose from 85.7% in April to 97% by September.

Before optimizing, spend split roughly 23% output tokens, 43% cache writes and 34% cache reads — most of the money went to context the model had seen or would see again. The code quality reviewer averaged 16.6 LLM calls per review, writing 72k tokens to cache and reading back 1.08m, because each call resends the whole conversation. With Anthropic's prompt cache, a stored prefix costs 1.25 times the input price to write and 0.1 times to read, and expires after five minutes unless read.

Instrumentation from September 14 showed a session's first request found a cached prefix only 14% to 26% of the time, even though 11k to 21k tokens of tool definitions and agent prompt are identical across repositories. The cause was OpenCode v1's system prompt layout: one block containing the agent prompt plus an environment section with working directory and date, with the only cache breakpoint at the end, so the shared portion was never stored alone. On September 16 Cloudflare added a marker at the end of each agent prompt and a plugin hook that split the block there, disabling the skills list and freezing tool descriptions so the first half stays byte-identical across jobs. That same day the Codex reviewer's reference index moved into its prompt, cutting its calls per run from 15.9 to 9.6 and cost from $1.12 to $0.72. Together these took cost per review from $2.47 to $2.02.

Changes on September 21 and 23 removed about $0.20 per review: disabling tools sub-reviewers could not use (they tried the posting tool in 57% to 76% of sessions, were rejected and spent a call recovering, plus 2.9k fewer tokens of definitions); reading files in parallel, with reviewers pulling shared context and all patches in their first response up to 150 KB; skipping reviewers on re-reviews (AGENTS.md 56% of the time, performance 23%, documentation 13%); stopping after the review is posted rather than regenerating an unseen summary, saving roughly $0.07 per job; treating a timed-out reviewer as slow rather than a provider failure, alongside longer time limits and lower effort, taking security timeouts from 1.8% of runs to nearly none; and having jobs poll their own status every 15 seconds, since 358 of 446 sampled cancelled jobs had kept running under GitLab's Kubernetes executor.

The most involved change shares one prefix across the code quality, security and performance reviewers: identical first requests (tools, shared rules file, merge request with all patches), then each reviewer's own instructions, with one writing the cache and the other two reading. Because a cache entry isn't readable until the writing response is streaming, the second and third wait up to 20 seconds for the first to start, saving about nine cents per reviewer at a small latency cost.

Model and effort changes yielded $1.06 of the $1.67 drop in that breakdown, against $0.65 from code; Cloudflare notes the before-and-after comparison is not a controlled experiment. The coordinator went to Opus 5.5 at medium effort after a randomized split showed no measurable change in outcomes versus Opus 5, while high effort cost more without a meaningful gain. A 50/50 split of Sonnet 5 and Sonnet 5.5 from October 3 found 5.5 cheaper per run — 62% for code quality, 58% for performance, 47% for security — at the same per-token price, so reviewers moved to 5.5 on October 5. On 5.5 at medium, the code quality reviewer makes 1.7 calls per run instead of 6.2, with over half finishing in a single call. High effort costs 61% more per run but posts 58% more critical and warning findings — about two cents per review — so code quality moves to high.

Latency improved as a side effect: 37.9% of reviews took over five minutes in the four weeks before September 14, versus 0.7% now. Median full-tier reviews fell from 5m 16s to 1m 54s, lite from 2m 58s to 1m 14s, and trivial from 77s to 30s. Part of that recovered ground lost since April.

On quality, the code quality reviewer posts 40% fewer critical and warning findings than in late August; security and performance, at medium effort, post 56% and 71% fewer. Cloudflare checked whether authors acted on findings: the share acted on rose from 34% to 59%, with acted-on findings per 1,000 reviews roughly flat (132 vs 145). Blinded judging by models from two providers on 657 findings found important findings per review rose from 0.28 to 0.30, while inflated severities fell from 0.46 to 0.14 per review. Medium effort for code quality lost real findings, which is why it moves to high; performance lost about 64% of important issues, roughly one every 24 full-tier reviews, and security fell about a quarter without statistical significance. Reviews ending without a verdict dropped from 0.79% to 0.27% and security retries from 3.5% to 0.04%. The reviewer also moved to OpenCode v2 in early October after a shadow run agreed on 98% of merge request decisions, with full-tier reviews about 13% faster.

Why it matters: Teams running LLM agents at volume face the same compounding cost curve — every extra call resends context and every new model generation can quietly raise the bill. Cloudflare's results suggest the first place to look is whether a session's first request actually reads from a shared cache prefix rather than writing a new one, and that trimming wasted calls and verifying quality with acted-on-finding metrics pays off in both cost and latency, though a likely inference is that cheaper models and lower effort settings need per-workload quality checks before adoption.

Read at Cloudflare AI

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

When our AI code reviewer's cost per run doubled, we dug into the token usage to find out why. Learn how we used prompt caching and intelligent model routing to slash our inference bill by 68% while speeding up reviews.