Breaking

Arena Finds AI Judges Prefer Their Own Answers in 34,580-Verdict Study

September 29, 2026 at 12:19 EDT

Foundation Models
Research & Papers

Arena says a new analysis of 34,580 valid model verdicts found that AI judges often prefer their own answers, even when the source model names are hidden. The result is a warning for teams using LLM-as-a-judge systems: model consensus can diverge sharply from human preference.

Sept. 29, 2026 · Evaluation study

AI judges have their own taste

A 12-model review found that automated judges favored their own answers and aligned with other AI judges more often than with human voters.

34,580

valid judge verdicts

model judges

1,460

Text Arena battles

393

unanimous AI calls

Self-preference gap

Share of battles where the model's own answer was chosen.

58%

AI judge

34%

Human voters

AI agreement ran ahead of human agreement

When winners were selected, model judges sided with other models more often than with people.

79.4%

Other AI judges

56.9%

Human voters

Why teams still use AI judges

They can review many model outputs quickly and make evaluation less dependent on slow manual review.

Why the result matters

A judge can favor itself, favor a model family, and force a winner when people would call the battle close.

Operational takeaway

hide author namesswap answer ordertrack draw ratescompare with humans

Continue reading

The rest of this article is for AI News Blitz readers. Choose an option below to keep reading.