New Artificial Analysis index widens the gap between open and closed models

Artificial Analysis has overhauled its Intelligence Index, and the change reordered its leaderboard. Kimi K3.1 Pro, which had launched in fourth place, fell to ninth in version 5, now sitting behind eight closed models. The organization also reports that the gap between open models and the top of the rankings has grown from 3 points to 12 points under the new methodology.
The central change is the data behind the score. For the first time, 60 percent of the index relies on private test sets. The stated goal is to curb "benchmaxxing" — training models specifically on questions that are known to appear in public evaluations. When a benchmark's items are widely available, a lab can tune a model toward those exact items, and the resulting score reflects familiarity with the test as much as general capability. Holding most of the weighting in sets the labs cannot see makes that kind of targeting harder.
Kimi K3.1 Pro's five-place slide is the clearest illustration of the effect. Artificial Analysis frames it as evidence that the previous index made some models look stronger than they actually were. Grok 4.7 also lost 12 points under the revision. In practice, a drop of that size points to a scoring shift rather than any change to the models themselves, since the underlying weights did not move — only the yardstick did.
AI researcher Nathan Lambert argues that benchmaxxing explains only part of the pattern. A large share of current training gains, in his view, comes from reinforcement learning environments whose tasks closely resemble the benchmarks. Competing across many domains at once requires sophisticated RL infrastructure, and not every lab has it. This reading suggests the private-test change may be measuring something real: the capacity to build broad, general training pipelines rather than to memorize particular questions.
Private test sets carry a cost of their own. When the prompts are hidden, outside researchers can no longer inspect them and flag mistakes. The AA-Omniscience knowledge test demonstrates that errors do occur — Artificial Analysis had to revise ambiguous and flawed questions in that benchmark to build the new index. A closed evaluation reduces the risk of gaming but weakens the community's ability to audit the questions themselves, a trade-off worth watching as more of the index moves behind closed doors.
Why it matters: Teams choosing between open-weight and closed models often lean on aggregate leaderboards, and a methodology change this large means scores from v5 are not directly comparable with earlier ones. Anyone tracking Kimi K3.1 Pro or Grok 4.7 should re-baseline against the new index and treat older rankings with caution. For model builders, the shift toward hidden test sets likely raises the value of broad reinforcement learning capability — an inference from the reported gap, not a confirmed requirement.
Based on reporting from the original publisher. Visit the source for full context and later updates.
Publisher excerpt
Artificial Analysis overhauled its Intelligence Index, causing Kimi K3.1 Pro to drop five spots in the rankings. The article New Artificial Analysis index widens the gap between open and closed models appeared first on The Decoder .