Measuring benchmark optimization in speech recognition

Hugging Face researchers have published work introducing three tests to quantify benchmark optimization in automatic speech recognition, sometimes called "benchmaxxing." They evaluated 11 widely used open-source ASR models and paired the tests with held-out sets in Real World VoiceEQ, the Open-ASR Leaderboard, and the Far-field ASR Leaderboard.

The first test, a consensus disagreement probe, compares models against an ensemble selected for low phoneme error rate, which flags cases where models unanimously disagree with a benchmark's reference transcript. The researchers report that in one VoxPopuli clip where the audio includes "Thank you, Mr. President" but the reference omits "Thank you," six of 11 models reproduced the erroneous transcript. Models that omitted the phrase also matched the benchmark's punctuation style, writing "Mr" without a period. The behavior often weakened or disappeared when the same content was presented in newly collected voices, including a generic text-to-speech voice.

The second test masks numbers in audio. Models sometimes supplied the silenced number, with some reproducing masked numbers in roughly 30-40% of LibriSpeech examples.

The third test examines orthographic switching, such as "any one" versus "anyone" and "Mr." versus "Mister." Multiple models exceeded the 50% random-choice baseline, with some reaching roughly 90% switch accuracy, suggesting they can identify an audio sample's dataset and select its spelling convention.

The researchers say the methodology flagged potential reference errors in 40% of VoxPopuli test clips analyzed, affecting roughly 3% of all reference words, and that models exhibiting benchmark-optimized behavior reproduced erroneous reference transcripts 18-30% of the time. Models with the lowest word error rate were most likely to reproduce these errors.

They recommend fully held-out evaluation sets and avoiding simple independent and identically distributed test splits in favor of temporal, speaker, or other metadata-based separation. A "Benchmark fitting" tab has been added to the Open ASR Leaderboard, with scripts and un-normalized model outputs open-sourced on GitHub.