AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Toward a Theory of Evaluability

Collected Nov 16, 2026

Kunal Talwar, an author publishing through Apple Machine Learning Research, has released a paper titled "Toward a Theory of Evaluability" that reframes how models can be assessed when the usual assumption of a clean holdout set no longer holds. The work targets large foundation models, which are now judged along several dimensions at once: task-specific accuracy alongside safety, calibration and fairness measures. The classical evaluation framework presumes the test set was withheld from training, and the paper argues that presumption is increasingly difficult to defend at foundation-model scale.

Rather than treating that contamination problem as a caveat, the paper formalizes an idea of evaluability that does not depend on it, then asks whether accurate evaluation is feasible at all when test samples may have been seen during training. The analysis proceeds by the level of access an evaluator has to the model.

The first setting is the most natural one: the evaluator observes only the model's outputs on the test set, with no other interaction. For this purely observational case, the paper fully characterizes the class of distributions under which accurate evaluation is possible and gives matching upper and lower bounds on the sample complexity of evaluation. The headline conclusion is stark: black-box evaluation in this setup is feasible if and only if the distribution is close to having small support.

That condition deserves unpacking for readers outside learning theory. "Small support" means the data effectively concentrates on a limited set of cases rather than spreading mass across a large, diverse space. Intuitively, a leaked test set is harmful because a model can memorize individual points; when the underlying distribution is concentrated, there are few enough distinct points that observed performance behaves predictably, so an evaluator can draw conclusions even without a guaranteed holdout. When the distribution is broad and rich, memorization becomes indistinguishable from genuine generalization at the level of outputs alone. The tight upper and lower bounds matter because they show the restriction is not an artifact of a weak method, but a limit of the observational setting itself.

The paper then moves to evaluators that can query the model on additional inputs beyond the test set. Here the work draws a connection to self-correctors, a notion from program testing in which a system uses its own outputs to check or improve itself. The results indicate that a small amount of query access can increase an evaluator's power for natural distributions and concept classes. In other words, the ability to probe the model, not just read its answers on fixed samples, loosens the severe constraint of the observational case.

For developers and product teams, this is theory rather than a downloadable tool or benchmark, and the paper abstracts away the particular metrics being estimated. The practical reading, my own inference rather than a claim in the paper, is that it gives language and limits to a familiar worry: if you cannot certify a private holdout, your accuracy numbers on public or scraped data carry less weight than they appear to. The characterization suggests such numbers are most trustworthy when the evaluation concentrates on a narrow set of cases, and least trustworthy when it spans diverse, open-ended inputs.

The second result points, tentatively, toward evaluation designs built around active probing rather than static test sets. A likely trade-off is cost and engineering complexity: adaptive querying requires running the model more times and constructing inputs deliberately, which is harder to standardize and reproduce than scoring a fixed benchmark. The paper's framing also raises questions about how such probing strategies would be audited or shared across teams, and how well the theoretical gains translate to the messy distributions real products encounter, which the source does not address.

Why it matters: Anyone reporting model quality on data that may be in the training corpus, which is most public benchmarks for foundation models, is affected by how credible those numbers are. This work suggests output-only checks have provable limits unless the evaluation is narrow, and that interactive probing may restore some of the lost signal. For product teams, that is an argument for treating contaminated benchmarks skeptically and, as an inference, for investing in evaluation methods that engage the model rather than just score it.

Read at Apple Machine Learning Research

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

Machine learning models, particularly large foundation models, are increasingly evaluated along multiple axes that are relevant for their usage. In addition to accuracy measures appropriate for the specific task of interest, measures capturing safety, calibration and fairness properties are often of interest. The classical approach to these problems assumes that the test set has been held out from the training process, an assumption that is increasingly harder to justify for large foundation models. In this work, w