Automatic language-model judges systematically skew, an ICML study shows

No, not with a single overall score. An ICML study shows that AI judges can carry systematic bias, so measure output variation and judge bias separately, report uncertainty intervals and test the same answers under alternative judges before you rely on the outcome.

In 2026, an ICML study introduced a statistical framework for measuring bias in automatic language-model judges. The core point: when a language model scores the answers of another language model — a practice known as LLM-as-a-judge — that judge is not itself neutral. The measurement method has thereby become an object of measurement, and that affects everyone who relies on such scores to release a model for sensitive work.

What does the ICML study establish about bias in LLM-as-a-judge evaluations?

The study Quantifying Biases in LLM-as-a-Judge Evaluations introduces a Bayesian model — a statistical method that also expresses uncertainty rather than only a point score — with which researchers can identify, quantify and correct the skew of automatic judges.

The framework works for absolute scores as well as pairwise preferences (model A versus model B), reports uncertainty and enables counterfactual simulations, such as the expected agreement after a systematic bias has been computed out. The authors stress that their findings hold within the models and setup studied; there are no universal bias percentages that apply to every product.

Why must you treat output variation and judge bias as two separate measurements?

In our analysis, output variation and judge bias are two distinct sources of variation that each need to be measured separately. The model output can vary because of the prompt or the task condition, while the evaluation score can in addition vary because of the judge. Whoever combines both into a single figure cannot afterwards tell whether a poor score came from the model or from the judge.

Two additional studies support that separation empirically. Research in the proceedings of the NLP+CSS workshop tested eighteen models with ten variants on question and answer options, together accounting for more than 334,800 simulated interviews. Almost all tested models proved sensitive to those variants and exhibited a recency bias: the last answer option shown was chosen disproportionately often. Substantively comparable prompts therefore do not yield the same answer distribution.

The preprint The Multilingual Artificial Hivemind evaluated twenty-three open-weight models across thirteen languages, with 448,500 generated answers, and used multiple diversity measures. The authors report a pervasive artificial-hivemind effect across the studied languages, models and decoding configuration, while emphasising limitations around languages, prompts, models and that configuration. In our estimation it follows from this that no single diversity measure suffices on its own.

What does a two-layer evaluation protocol look like in practice?

The three studies provide no ready-made combined protocol — they leave that open. Our editorial synthesis of their methods is an evaluation in two layers, which first measures the output and then the judge separately.

Layer one measures output variation:

• Repeat each test several times with fixed sampling settings, so that variation becomes visible.
• Use controlled prompt perturbations and swap the order of answer options, to catch effects such as recency bias.
• Stratify by language and target group, because competence and homogeneity differ per language.
• Combine embedding-based and lexical (word-level) diversity measures rather than relying on a single metric.

Layer two measures judge bias by having the same outputs scored by different judges:

• Deploy multiple judges, model families and judging prompts on identical answers.
• Report the agreement between judges as well as uncertainty intervals, not just an average.
• Test explicitly for systematic patterns in how judges score identical answers.
• Compare absolute scores with pairwise preferences; they can diverge.

As an editorial recommendation: choose the number of runs on the basis of the expected effect size and desired precision, not solely on the basis of a fixed rule of thumb. This aligns with our earlier analysis of how many repeats a reliability test needs.

What does this mean for directors, lawyers and CISOs handling sensitive information?

Our analysis: because automatic judges can contain systematic bias, a procurement or validation file that relies on a single average score can give a skewed picture to whoever releases the model; that is why a CISO or validation lead now asks for separate reporting of output variation, judge bias and mutual agreement, with uncertainty intervals, before production use. Because substantively comparable prompts yield different answer distributions, a supplier figure from a single prompt variant can overstate performance for the controller; that is why a buyer includes a contract clause obliging the supplier to name the prompts, model versions and sampling settings used, and the own team repeats the test under alternative judges. Because the findings only hold within the models, tasks and languages studied, a generic bias percentage can give a lawyer a false sense of certainty about a concrete product; that is why a responsible party ensures the evaluation dataset contains representative cases as well as edge cases and documents that the outcome holds only for those conditions. Because every change in model, prompt or setting can shift the outcome, a silent model update can unnoticeably alter a previously approved workflow for a lawyer or doctor; that is why the team keeps prompts, model versions, judge configurations and raw outputs and tests again at every change. These governance steps belong in a broader approach to AI verification; for supplier claims it moreover holds that benchmark figures from the provider are not independent proof.

In short, from these studies, in our analysis, one rule follows: separate the output from the judge, measure both with uncertainty, and keep enough to repeat the test.

Sources: The article draws on the ICML study on bias in LLM-as-a-judge, an NLP+CSS workshop study on prompt perturbations and the arXiv preprint The Multilingual Artificial Hivemind.