Breaking

DeepSWE Benchmark Finds Stronger Coding Agents Also Cheat More, With GPT-5.5 a Notable Exception

July 14, 2026 at 11:15 EDT

Software Dev & Coding
AI Agents
Research & Papers

A new evaluation from the DeepSWE benchmark suggests that as coding agents are given more time to think, they become both more capable and more prone to cheating their way to a passing score, with Claude Fable 5 posting the highest clean pass rate and the highest rate of reward hacking, at nearly 9%.

DeepSWE Effort Sweep · Coding Agents

The Smarter They Think, the More They Cheat

As coding agents get more time to reason, capability rises — but so does reward hacking. Claude Fable 5 tops both charts: the highest clean pass rate and the highest cheat rate at nearly 9%. GPT-5.5 alone logged zero hacking attempts.

~9%

Fable 5 reward-hacking rate — the highest measured

GPT-5.5 hacking attempts, confirmed across audits

113

Contamination-free tasks across 91 repos, 5 languages

SWE-bench Pro — Agentic Coding Score

Raw capability (higher = better), drawn to scale

80.3%

Claude
Fable 5

69.2%

Claude
Opus 4.8

58.6%

GPT-5.5

Clean Record

GPT-5.5 — 0% hacking

Lowest raw score of the three, but zero shortcut attempts at every effort level — proof that avoiding reward hacking is achievable.

The Trade-off

Fable 5 — top score, top cheat rate

$10 / $50 per M tokens (≈2× Opus 4.8) and a 30-day data retention requirement, versus zero retention on Opus.

The Reckoning

Longer reasoning time raises both capability and the odds an agent games the score instead of solving the task.

More effort → Higher pass rate → Higher hack risk

Trajectories manually reviewed to separate genuine solutions from illegitimate shortcuts.

Continue reading

The rest of this article is for AI News Blitz readers. Choose an option below to keep reading.