Breaking

Cognition Launches FrontierCode Leaderboard to Measure Whether AI-Written Code Is Actually Mergeable

July 17, 2026 at 17:23 EDT

Software Dev & Coding
Research & Papers
Foundation Models

Cognition, the company behind the Devin coding agent, has launched a public leaderboard for FrontierCode, a benchmark that scores AI models not on whether their code passes tests but on whether a human maintainer would actually merge it into production. The live page publishes full scores — including newly added results for Grok 4.5 and Inkling — alongside the complete methodology and sample tasks.

July 7, 2026 · Cognition

"Would a maintainer merge it?" — FrontierCode raises the bar for AI code

A new public leaderboard scores coding models not on whether their code passes tests, but on whether a human maintainer would actually ship it to production. Even the strongest frontier models top out in the low double digits.

81%

cut in misclassification rate vs. prior benchmarks

major open-source repositories behind the tasks

40h+

each maintainer spent authoring a single task

5×

runs per task, across reasoning-effort settings

Still far from a production-quality bar

Scores on the hardest "Diamond" subset (v1.0). Height shows the merge-ready pass rate — the remaining gray represents what a maintainer would reject or rework.

13.4%

Claude
Opus 4.8

6.3%

GPT-5.5

4.7%

Gemini
3.1 Pro

3.8%

Kimi K2.6
best open src

Dotted line reference: 100% = every pull request merge-ready. The leaders barely clear the floor.

Broader task sets — Opus 4.8

34.3%

Main (100 tasks)

51.8%

Extended (150 tasks)

Efficiency footnote

up to 4×

fewer output tokens from GPT-5.5 vs. Opus 4.8, at competitive scores

Six merge dimensions

Instead of one pass/fail signal, each pull request is graded across quality axes. Miss a blocker criterion and the run scores zero.

Behavioral correctness Regression safety Mechanical cleanliness Test correctness Scope discipline Overall code quality

Welcomed by developers

More realistic than test-only benchmarks. Maintainers from Celery, Budibase, uppy and Mattermost liken the grading to a tech lead reviewing a submission.

Open questions

Some observers raise reproducibility and score-variance concerns. The hardest "Diamond" subset was retired partly to reduce noise; v1.1 added cheating detection.

Continue reading

The rest of this article is for AI News Blitz readers. Choose an option below to keep reading.