AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Artificial Analysis v5 and the benchmarker's dilemma

Collected Oct 28, 2026

Artificial Analysis has updated its Intelligence Index to v5, the latest version of what has become the most prominent independent intelligence metric for frontier language models. The update was framed by the organization as reflecting real-world usage today, and the accompanying ranking places GPT 6.1 Terra above Claude Fable 5.1 at the top. The change prompted a review of the benchmarking industry, its structural problems and what is missing from current evaluation practice.

The index is built from four equally weighted pillars, at 25% each. Knowledge Work combines AA-Briefcase at 15%, AutomationBench-AA at 5% and GDP.pdf at 5%. Coding combines FrontierCode 2.0 at 15% and Terminal-Bench 4.0 at 10%. Science combines Terminal-Bench-Science at 15% and the hard subset of CritPt v2 at 10%. General combines AA-Omniscience v1.1 at 15% — split into 10% accuracy and 5% non-hallucination — with HLE-Diamond at 10%. Each pillar carries the same weight as the others, so a model's standing depends as much on which sub-benchmarks sit inside a pillar as on its raw capability.

The stated goal is a single number that is representative of current models, matches vibe checks, community sentiment and scientific analysis, and stays forward-looking enough not to be outdated within a few months. That ambition runs into a paradox every benchmark builder faces: the score has to collapse into one number so it is easy to discuss, report and compare, but every aggregation method carries meaningful trade-offs. Choosing weights is a judgment call about what counts as a good model, not a neutral measurement.

On the new index, the model ranking looks broadly plausible, but it shows strain. GPT 6.1 Terra beating Claude Fable 5.1 on AAv5 is the kind of result that invites scrutiny of what is being measured. Some evaluations target the absolute frontier of capability, in open-ended tasks run through a user's preferred harness, which typically rewards a large model. Others measure narrow skills needed for repetitive agentic work. The sense is that AAv5 leans toward the former — a general agentic collaborator — though these behaviors overlap and do not separate cleanly.

A widely circulated plot from the update shows the balance of open and closed models on the new benchmark: eight closed models outperform the best open-weight model, Kimi K3.1 Pro. That reflects a deliberate choice to measure where frontier models are heading, which widens the gap to open models. The decision is defensible, but if existing open-model workflows become a significant part of the economy, the index could end up missing where the core economic engine of the industry sits.

The open-model picture is more balanced than that plot alone suggests. The previous index flattered open-weight models too strongly. On the Kimi K3.1 Pro launch data, open models are strong and useful in some cases, but the best OpenAI and Anthropic models have been a major step ahead in flexibility and reliability. The evaluations where open models lag most in AAv5 are science tasks such as Terminal-Bench-Science and CritPt v2. This is consistent with the argument that open models were never primarily trying to compete at the true frontier. The value proposition is twofold: reducing domains with high inference demand to a commodity price of hardware, and allowing organizations to own intelligence, specialize it on internal data and control internal agents. Both take a long time to materialize and are only partly represented in any benchmark. Compute supply is currently so low that open-model providers are earning solid margins; when those margins shrink, the system will be closer to working as intended.

There are more nuanced explanations for the performance gap. Many recent training gains come from reinforcement learning environments whose distributions sit close to the evaluations. Keeping pace across many domains requires a sophisticated RL stack that not all open labs may have, and some may depend on buying data with moderate lead times. The expectation is that at the next major model cycle, roughly two to three months out with releases such as GLM 6 or Kimi 3.5, the leading open labs will make substantial steps on the index — a cycle that has repeated for years.

The v5 update also shifts toward private benchmarks, where prompts and test environments are not public, framed partly as a response to open labs tuning to known evaluations. That direction is defensible, but not unanimous. Private benchmarks place complete trust in the builder, because the community can no longer find issues and push the benchmark to improve — something that happens often. With this much money on the line, transparency and many eyes on the tests have real value. One can still train toward a hidden evaluation's shape, though it is harder.

A further dynamic flatters the closed labs. OpenAI and Anthropic moved on from caring much about these indices months or years ago; their attention is largely on internal uses such as AI R&D automation and on customers spending billions. Public benchmarks drive only a minority of their capability work. For small and medium labs, appearing solid on these indices is closer to existential, because their branding is built around catching up to the frontier. That produces a lasting cat-and-mouse game.

The structural answer is more indices with different biases, for example one dedicated to coding alone, alongside newer entrants such as Vals AI and intelligence.ai. Mindshare, however, remains lopsided toward Artificial Analysis. The business model matters here: Artificial Analysis earns money from labs paying to verify results, and other benchmarking companies run similar arrangements, such as pre-release testing. The entities regarded as most fair tend to be non-profits, academics and government institutions such as CAISI, while the startups are trusted for accuracy on frontier evaluation. No number is perfect, and there is wide wiggle room in how the figures can be read.

Why it matters: developers and product teams increasingly use a single index score as a shortcut when picking models, and v5 changes which capabilities that shortcut rewards. Because each pillar carries equal weight, teams optimizing for coding or knowledge work should look at the sub-benchmark breakdown rather than the headline number, which is my inference from how the weights are structured. The shift toward private benchmarks also means less external scrutiny of the tests themselves, so treating any composite as one input among several remains the safer practice.

Read at Interconnects

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

A sort of review of the benchmarking industry and what's missing.