Breaking

As AI Models Ace Standard Benchmarks, Researchers Rush to Build Harder Tests

July 14, 2026 at 11:31 EDT

Research & Papers
Foundation Models

Frontier AI models are outrunning the very benchmarks meant to measure them, prompting researchers to design far tougher tests—but even those are being conquered faster than expected. The problem, known as benchmark saturation, has become one of the central challenges in evaluating how capable modern systems really are.

Benchmark Saturation · AI Evaluation

Frontier Models Are Outrunning the Tests Built to Measure Them

Even "Humanity's Last Exam" — 2,500 expert questions meant to probe the edge of human knowledge — went from single digits to roughly half solved in about a year. Designing harder tests is now the real race.

~2,500

Expert-written questions in HLE, from ~1,000 specialists

pass@1

Graded on a correct answer at the first attempt

<1 yr

Time for new benchmarks to be effectively solved

HLE Scores: From a Few Percent to Half the Test

Best pass@1 score at first attempt · each block ≈ 5 percentage points

2.7%

GPT-4o

4.1%

Claude 3.5

~8%

o1-class

50%+

2026 frontier

Takeaway: A test billed as "humanity's last exam" jumped roughly 18× — from ~2.7% to over 50% — in about a year.

The Recurring Race

New benchmark

built to be hard

→

Models saturate it

cluster near the top

→

Design "HLE 2"

rolling / dynamic tests

Evaluation — not model training — may increasingly be the bottleneck for understanding AI progress.

Still valuable

A shared reference for scientists and policymakers
Useful for improving alignment evaluations
Dynamic tests (ARC-AGI, τ-bench, SWE-bench) probe real workflows

The skeptics

Building good benchmarks is harder than building the models
Static tests invite "gaming" and overfitting
A substantial fraction of items flagged as flawed

Continue reading

The rest of this article is for AI News Blitz readers. Choose an option below to keep reading.