AivexaNewsSearch
AI news for builders and product teamsChecked every hour

Theoretical Limits of Language Model Alignment

Collected Nov 12, 2026

Apple machine learning researchers Lucas Monteiro Paes, Natalie Mackraz, Barry-John Theobald and Federico Danieli (the first two listed as equal contributors) published a theoretical study, "Theoretical Limits of Language Model Alignment," characterizing how much reward improvement is achievable when a language model is aligned subject to a fixed KL-divergence budget. The work was accepted at the Fine-Tuning in Modern Machine Learning (FITML) Workshop at NeurIPS 2024.

The setup is standard for preference alignment: two dominant families are compared. Reinforcement learning maximizes expected reward under a KL-divergence constraint against a reference policy, while best-of-N alignment draws N independent samples and keeps the one with the highest reward. Despite both being widely deployed, the paper argues the fundamental ceiling on reward improvement under a KL budget has been poorly understood. KL divergence here measures how far the aligned policy has drifted from the base model; constraining it is what keeps the model from drifting into degenerate, high-reward-but-unusable outputs.

The paper's headline theoretical result is a closed-form expression for the optimal achievable reward improvement at a given KL budget. Notably, that bound is governed by a Jeffreys divergence term rather than the square-root KL scaling assumed in earlier analyses. This matters in practice: if the true growth curve differs from the assumed one, engineering decisions about how hard to push alignment may be mis-calibrated.

The authors then recast the expression as a covariance under the base model, which yields a practical estimator. In plain terms, that means the achievable alignment gain can be predicted from base-model samples alone, without first running an expensive alignment procedure. For teams choosing between methods, a cheap estimate of the ceiling before committing compute is a useful planning tool, though its reliability on a given task or reward model is something practitioners would need to check.

The analysis also covers proxy rewards, the setting where the reward model used for optimization is only an approximation of true human preference. Here the authors show that the gap between ideal alignment and proxy-reward alignment — reward hacking — grows with the magnitude of reward error and shrinks as the KL penalty factor decreases. They then prove that reward ensembling mitigates reward hacking, giving theoretical backing to a technique already in common use. A likely reading is that averaging several reward models reduces the effective error that optimization can exploit, which is consistent with why ensembles are popular in practice.

On the empirical side, the team computed the KL-reward Pareto frontier for two language model tasks, safety and summarization. Best-of-N comes close to the theoretical limit, while PPO and GRPO — two widely used reinforcement-learning alignment algorithms — remain substantially suboptimal. GRPO, popularized for reasoning-oriented training, is particularly notable here: the results suggest that common preference-optimization pipelines leave meaningful reward on the table at a given KL budget.

That finding carries a real-world trade-off. Best-of-N reaches closer to the optimum but requires N samples at inference time, so its cost scales with N rather than being paid once at training. The authors conclude that algorithmic improvements are needed to achieve optimal alignment without high inference costs — in other words, to close the gap between the RL methods and the theoretical ceiling so that inference does not have to carry the burden. The paper also says its results illuminate several empirically observed phenomena in the alignment literature, though the abstract does not enumerate them.

This is theory and benchmarking work rather than a shipped model or tool; no code release, model, dataset or pricing is announced. For context, KL-regularized RLHF with PPO has been the standard recipe since the InstructGPT era, and best-of-N sampling is a simpler, inference-time alternative that many teams already use as a baseline. A study that quantifies the ceiling for both, and places PPO and GRPO relative to it, reframes the choice between training-time and inference-time alignment as a question of how close each gets to an information-theoretic bound.

Why it matters: teams training preference-aligned models, and product teams weighing inference-time sampling against RL fine-tuning, get a quantitative ceiling to plan against and evidence that reward ensembling is not just folklore. The finding that best-of-N approaches the theoretical limit while PPO and GRPO fall short suggests, as inference, that pushing more compute into RL alone may hit diminishing returns until better algorithms arrive, and that sampling budgets deserve a place in alignment planning. Because the estimator works from base-model samples, it may also make early feasibility checks cheaper for teams deciding how much alignment effort a task warrants.

Read at Apple Machine Learning Research

Based on reporting from the original publisher. Visit the source for full context and later updates.

Publisher excerpt

Language model (LM) alignment improves model outputs to reflect human preferences while preserving the capabilities of the base model. The most common alignment approaches are (i) reinforcement learning, which maximizes the expected reward under a KL-divergence constraint, and (ii) best-of-NNN alignment, which selects the highest-reward output among NNN independent samples. Despite their widespread use, the fundamental limits of reward improvement under a KL budget remain poorly understood. We characterize the info