FairStress: A deeper evaluation of fairness in LLM decision-making

The Allen Institute for AI has released FairStress, a benchmark for measuring demographic fairness in large language models at the level of decision stability rather than just final answers. The benchmark is downloadable, has a companion tech report, and is being integrated into Olmo-eval, Ai2's open workbench for running and comparing model evaluations. Alongside the full benchmark, the team published FairStress-Core, a smaller subset built for faster and cheaper evaluation. The work evaluates open-weight and frontier systems including Olmo, Llama, Qwen, Gemma, Claude and the GPT family, across several stages of training.
The core idea targets a blind spot in conventional fairness testing. Most benchmarks swap demographic identities in otherwise identical scenarios and check whether the model picks the same person. FairStress adds a second stage: after the model makes a choice, it is presented with an adversarial prefix, a short persuasive statement, and the benchmark measures how readily the model abandons its answer. Each prefix has a matched counterpart making the same kind of case for the other candidate. If a model gives up a well-supported decision more easily when the push favors one demographic group than another, that asymmetry is invisible to output-level tests even though the model's decision is equally correct in both versions. Ai2 calls its target symmetric fairness: same criteria, same resistance to pressure, regardless of which group benefits.
FairStress contains 300 realistic two-candidate scenarios spanning 29 domains, including hiring, healthcare, lending, education, housing and criminal justice. Examples include choosing between two candidates for a mechanical engineering job and deciding which of two patients receives the last unit of a scarce medication. Some scenarios have a clearly stronger candidate, which tests whether identity information shifts how often the evidence-backed choice is made. Others are deliberately ambiguous, with two candidates strong in incomparable ways, which reveals which side a model leans toward when the evidence is genuinely balanced.
Identity is presented at four levels of explicitness: none, a demographically coded name, an indirectly implied identity, or an explicit demographic label. Prefixes draw on authority, social consensus, purported evidence, legal concerns and historical disadvantage; one cited example has a senior leader privately reviewing files and recommending the second candidate. Combining identity conditions and matched prefixes expands the 300 scenarios into more than 13 million test cases. FairStress-Core, a 50,000-item subset selected with item response theory following Ai2's Fluid Benchmarking approach, is meant to retain much of the full benchmark's discriminating power at lower cost.
The headline result is that behavior is close to symmetric when no demographic information appears, and becomes less symmetric as identity gets more explicit. Where one candidate was clearly stronger, a prefix supporting the weaker candidate was more likely to flip the model when the stronger candidate was majority-coded and the weaker one minority-coded, and less likely to flip it in the reverse arrangement. Ai2 labels this anti-stereotypical over-correction. Across 15 models on the full benchmark, the average gap in abandoning correct answers was 6.9 percentage points with implied identity and 9.3 points with explicit labels. On FairStress-Core, frontier models including Claude Opus 5 and GPT-5.5 chose the minority-coded candidate roughly 60 to 65% of the time when neither candidate was objectively stronger, and under matched prefixes switching toward minority-coded candidates was 8.8 to 42.4 percentage points more common across 12 model-and-mode runs.
Explanations diverge too, even when the answer is right. Across 11 open-weight models, justifications for minority-coded choices used somewhat more confident and positive language, with words like resilience, diversity and barriers appearing more often under explicit labels, while majority-coded choices drew more procedural vocabulary such as evidence and criteria. Ai2 notes that a higher selection rate for minority candidates does not by itself mean fairer treatment, since foregrounding identity can downplay the qualifications that justified the pick. A controlled analysis of Gemma-3-12B-it and Qwen2.5-7B-Instruct found internal activations predicted which demographic side the model chose with about 73% accuracy, versus 57 to 59% for text descriptions of those activations against a 50% chance baseline; removing candidate names cut activation-based accuracy by roughly 18 points. Ai2 treats these interpretability results as narrower than the behavioral findings, but reads them as suggesting a reasoning trace may not surface everything driving a decision.
The release also compares FairStress against BBQ, a widely used stereotype-bias benchmark. In Ai2's BBQ runs, models tended toward stereotype-consistent answers, the opposite direction from FairStress's minority-favoring tilt. Testing the same BBQ questions and weights under five configurations of chat template, wording and scoring, on four models, showed that with a chat template, yes/no phrasing and generated answers, three of the four shifted from stereotype-consistent toward minority-favoring behavior and Llama-3.1-8B-Instruct moved to roughly neutral. This suggests measured bias direction depends partly on how a benchmark elicits and scores answers, and that fixed-option probability scoring alone can miss over-correction in generated decisions.
On causes, Ai2 audited Olmo 3's training pipeline. Its supervised fine-tuning corpus contains about 99,000 examples from two refusal-training datasets; across 28,000 topic clusters in fine-tuning and preference data, only nine discussed decisions based on merit rather than demographics, and all nine were refusals. The audit also found uneven framing: minority-coded groups received more protective or positive treatment, while white and cisgender people received little protective framing and persistent negative framing. Ai2 says whether these patterns cause the behavioral gaps remains an open question.
Nine interventions were tested, including self-critique, activation steering and learned edits. Most did not reliably reduce the asymmetry, and some traded away accuracy sharply; one activation-steering method cut Qwen2.5-7B-Instruct's accuracy on clear-cut decisions from 98% to 47%. Counterfactual chain-of-thought, which asks the model to reconsider the case with identities swapped, came closest, narrowing the gap in abandoning correct decisions across three model families while largely preserving accuracy, though it did not eliminate the tilt on ambiguous cases. In practice, the practical lesson for teams running fairness work is that a passing outcome-level result is weak evidence on its own; the benchmark's value is in separating accuracy from consistency under pressure. Why it matters: teams shipping models into hiring, lending, health or admissions decisions now have a public, downloadable check that catches asymmetries a same-answer test would miss, and a way to screen debiasing methods before adopting them. Developers evaluating their own safety or fairness fixes should expect to report robustness under challenge, not just selection rates, and should test both generated-answer and fixed-option setups, since Ai2's comparison indicates the two can point in opposite directions.
Based on reporting from the original publisher. Visit the source for full context and later updates.