AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged

Collected Oct 12, 2026

Apple Machine Learning Research published a paper titled "Large Language Models for Machine Translation Quality Annotation: Humans and Models Are Both Challenged," authored by Hala Almaghout, Christian Federmann and Qin Gao. The work tests whether large language models can stand in for human judgment when scoring machine translation output, and it reports that the answer is a qualified no for most conditions studied.

The premise the authors work from is that LLMs are widely pitched as a cheaper and faster substitute for human evaluation of machine translation. Because translation quality assessment spans many language pairs, domains and levels of annotation granularity, the paper argues these models need testing along all of those dimensions before they can be trusted as replacements. The authors focus on two established evaluation schemes. Multidimensional Quality Metrics (MQM) is a framework in which annotators mark fine-grained error categories, such as accuracy or fluency problems, at specific points in a translation and assign severity levels. Error Span Annotation (ESA) is a lighter-weight protocol in which annotators highlight a span of text containing an error and label how severe it is, trading granularity for speed and consistency. The paper compares how well LLMs agree with human annotators under each scheme.

The test material spans a long-context set covering 70 language pairs plus the publicly available WMT23 and WMT25 datasets. The authors examine agreement on both overall scores and error span annotations across a range of language pairs and domains. This is a broad sweep by the standards of translation evaluation work, which often concentrates on a handful of high-resource pairs.

Two headline findings emerge. First, task-dependent: for some evaluation tasks the agreement between an LLM and human annotators is actually higher than the agreement between human annotators themselves. That is a notable result, since inter-annotator agreement is the usual yardstick for how reliable a human evaluation is. Second, and more sobering, agreement varies substantially across annotation schemes, language pairs and domains, and the authors conclude that LLM annotation remains unreliable for most settings. In other words, the encouraging cases are real but not generalizable on current evidence.

The paper also profiles where each kind of annotator breaks down. Human annotators are described as particularly challenged by fine-grained MQM annotations and by low-resource language pairs. LLMs, meanwhile, struggle with minor errors, wrong language variants and error span annotation. That asymmetry is worth pausing on. Minor errors and wrong language variants are exactly the failure modes that matter in production localization, where a translation can read fluently yet use the wrong regional spelling or terminology. This suggests a realistic division of labour: models may be competitive on coarse judgments while humans retain an edge on precision work, and vice versa in low-resource languages where human expertise is scarce.

The authors frame the outcome as highlighting room for improvement on both sides, and they point toward human-LLM collaborative annotation pipelines as a possible way to address the reliability problems identified. That is a direction rather than a shipped system; the paper reports measurements, not a new annotation tool.

The broader context is that machine translation evaluation has been shifting toward neural and model-based metrics for years, because human evaluation is expensive and hard to reproduce. MQM in particular has become the reference standard in shared translation tasks, but it demands trained annotators and careful adjudication, which is why cheaper automated alternatives are attractive. This work is a reminder that replacing that standard is not a drop-in substitution: the reliability of the substitute depends on the annotation scheme, the language pair and the domain, and those dependencies are not stable enough to ignore.

Why it matters: teams that score translation output with model-based annotators should treat this as a caution against blanket substitution and consider validating agreement per language pair and per scheme before trusting automated scores in release gates. The finding that LLMs sometimes beat human-to-human agreement on some tasks, while failing on minor errors and language variants, is inference-friendly in one direction: hybrid pipelines with human adjudication on fine-grained and low-resource work look like the pragmatic near-term path, though the paper itself presents this as a potential rather than a proven design.

Read at Apple Machine Learning Research

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

Large Language Models (LLMs) are considered to be a more efficient and cost-effective alternative to human judgment for Machine Translation (MT) evaluation. With MT evaluation spanning a large number of language pairs, domains and levels of annotation granularity, LLMs must be thoroughly evaluated across these dimensions before being reliably used as alternatives to human evaluation. In this paper, we evaluate the performance of LLMs for two prominent MT quality evaluation schemes: Multidimensional Quality Metrics