Can we trust LLMs to judge whether a research idea is novel?

Ai2 (the Allen Institute for AI) has published "Old Ideas, Novel Problems," a study of how reliably large language models can judge the novelty of research ideas, together with a benchmark, data, code and an automated pipeline for building new evaluation sets. The release builds on Ai2's broader line of scientific-discovery tools — DataVoyager for structured data exploration, AutoDiscovery for hypothesis generation and testing, Theorizer for synthesizing evidence into candidate theories, and Asta — and addresses a specific gap: systems that propose ideas need a way to tell a genuinely new direction from a repackaged one. That job is typically delegated to a "novelty judge," another LLM asked to compare an idea against prior work, and the study finds the practice far less dependable than it looks.
Across six judges — GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6 — modest changes to the evaluation setup produced markedly different verdicts on identical ideas. The researchers varied one factor at a time: the prompt, reasoning budget, whether the judge could see related papers, how ideas were formatted, how many judgments were aggregated into a final answer, and, for AI-generated ideas, which model produced them.
To build ground truth, the team mined ICLR 2026 peer reviews rather than using acceptance as a proxy for originality. A paper can be accepted while being only moderately novel, and a rejected paper can contain an original contribution. From the reviews they extracted passages where reviewers explicitly discussed originality, assembling a high-novelty pool of 154 human-authored ideas: accepted papers in the top 10% of their ICLR research area whose reviewers positively identified novelty with no reviewer disputing it. The low-novelty pool holds 145 ideas from rejected papers in the bottom 10% where reviewers raised originality concerns and none endorsed novelty. In a check of the extraction process, an annotator agreed with the polarity of the mined signal in 47 of 50 cases, and no disagreement would have changed whether a reviewer counted as praising or disputing novelty. These pools form the Human-Only setting.
A second setting, Human+Generated, pairs the same 154 human ideas with 154 ideas from a simple LLM generator that received only a research area and an instruction to propose something novel — no literature search, tools, iteration or feedback. The generated ideas are assumed weaker than their human counterparts, which the authors describe as weak labels rather than verified per-idea judgments. As checks, an expert reviewed 30 pairs weighted toward cases the strongest judge got wrong, preferring the human idea in all 30 and judging none of the generated ideas novel; the team also rewrote both sets into a common structure to test whether judges were reacting to writing style. Notably, many core findings do not depend on the labels: when two reasonable setups disagree on the same pair, that instability is measurable on its own.
The headline result concerns prompt sensitivity. Shown the same Human+Generated pairs, GPT-5.4 reached 93.5% accuracy when told that reviewers had found exactly one idea in each pair novel and asked which; with a simpler prompt asking only which idea was novel, accuracy fell to 40.9% — below chance — and the verdict flipped on more than half the pairs. The same change moved accuracy by roughly seven points on Human-Only, suggesting judges can look stable on human research while behaving very differently on generated ideas. Human ideas were not immune: the change shifted Claude Opus 4.6 by 21 points on Human-Only. No prompt was universally best; removing an instruction warning judges not to treat incremental work as novel helped some models and hurt others.
In a common use case — comparing ideas from two generators (Claude Opus 4.5 versus Sonnet 4.5, and GPT-5.4 versus GPT-5.1) — roughly a quarter to a third of verdicts changed on average across the six judges under the tested prompt variations. More compute did not rescue things: GPT-5.6 Sol with an extra-high reasoning budget plus live, iterative arXiv search scored 0.78 macro-F1, 11 points behind Claude Opus 4.6 at 0.89 and in range with simpler setups, at roughly 10× the cost per decision at current API prices. Macro-F1 here measures how well a judge separates novel from non-novel ideas.
Purpose-built novelty evaluators also trailed a plain baseline on Human+Generated. A simple Claude Opus 4.6 prompt with low reasoning and majority voting over three calls reached 0.854 macro-F1 for $2.59 total. Idea Novelty Checker, a literature-grounded retrieval and reranking pipeline, scored 0.642 at $30.66; Sakana AI's AI Scientist, whose evaluator iteratively searches Semantic Scholar, scored 0.772 at $82.05. Both dedicated systems used Claude Opus 4.6 with one call per decision.
My reading: this is a caution about treating LLM verdicts as measurements. Prompt phrasing, aggregation and model choice are all part of the instrument, and a judge tuned on human papers may not transfer to the machine-generated ideas it is deployed against — a construct-validity problem familiar from other LLM-as-judge work. The cost figures also undercut the intuition that retrieval pipelines buy reliability. The authors frame the path forward as testing judges on the ideas they will actually face, measuring sensitivity across prompts and models, improving literature grounding, and building benchmarks that evolve with the literature; the released pipeline is meant to be reused on new peer reviews, other communities and future judges.
Why it matters: anyone using an LLM to rank research proposals, filter ideation output or compare two idea-generation systems is relying on a tool whose verdicts can swing by tens of percentage points with a reworded prompt, so point estimates of "novelty" deserve skepticism and sensitivity checks. Teams building evaluation harnesses should expect to report prompt and model variance alongside scores, since a single configuration can overstate a system's advantage. The inexpensive baseline beating costlier pipelines is, in my inference, a reason to benchmark simple setups before investing in retrieval infrastructure — and a reminder that automated scientific discovery needs a credible way to recognize new ideas, not just generate them.
Based on reporting from the original publisher. Visit the source for full context and later updates.