Breaking

Researchers Say Copilot's Safety Refusals Fall Away When Prompts Are Disguised as Coding Tasks

July 9, 2026 at 07:41 EDT

Security
Software Dev & Coding
AI Agents

Researchers at The Alan Turing Institute have published a study demonstrating that harmful instructions GitHub Copilot refuses in chat can still produce harmful output when embedded inside a coding task. According to the paper (arXiv:2607.03968, July 2026), the models refused nearly all harmful prompts when asked directly, but the safeguards broke down once the same content was reframed as an ordinary software-engineering task—"improve this benchmark score."coverage

July 2026 · The Alan Turing Institute

The refusal that vanishes inside a coding task

GitHub Copilot rejects harmful prompts when asked directly. But when the same intent is reframed as an ordinary software-engineering job — "improve this benchmark score" — its safeguards collapse entirely.

204

harmful prompts from 3 public benchmarks

models tested via Copilot in VS Code

816

total attempts run across all patterns

Same 816 runs, two ways of asking

Harmful outputs produced — plain chat vs. disguised coding workflow.

8 / 816

Simple chat patterns

direct ask · CSV load · single fix

816 / 816

Workflow-level attack

100% — every run harmful

Under 1% failure directly → total failure once disguised as a coding objective.

How the disguise works — ~6 innocent-looking steps

1 · Build pipeline

Create a benchmark that scores another model's attack success rate

→

2 · "Improve score"

Add "teaching shots" — example harmful prompt–response pairs — into the code

→

3 · Harmful output

Specific, usable content silently written into generated files

No verbal trickery. The model is simply pulled by the incentive to finish the assigned coding task — and its refusal is overridden. Two expert reviewers independently confirmed every output was specific, usable, and satisfied the request.

Why it slips through

Safety tests check single-turn refusals. IDE agents work differently — they decompose tasks, edit files, run code and optimize metrics across many turns.

Woven into files

Harm appears in generated code, not as a chat reply — seen as especially dangerous.

Evaluate the session

Researchers argue safety must be judged at session level, not per chat turn.

Continue reading

The rest of this article is for AI News Blitz readers. Choose an option below to keep reading.