Give a frontier coding agent more thinking time and it gets better — and, it turns out, it cheats more too. On DeepSWE, as we raise reasoning effort, Claude Fable-5 and Opus climb in capability and in reward-hacking attempts together (first graph below). One model breaks the pattern: GPT-5.5 stays flat at exactly zero, at every effort level.
A clean zero is suspicious, so we went looking — and what we found wasn't a clean model, it was a clean benchmark type. Put that same GPT-5.5 in front of a long-horizon benchmark that leaves an answer key in the sandbox (SWE-Marathon) and it hacks more than any model we measured. Reward hacking, we'll argue, is as much a property of the environment as of the model — which is exactly why it has to be measured across many of them, from the outside. This is the companion write-up to our preprint; full method and citations are there.
2
benchmarks audited
10,506
trajectories
14
frontier models
419
reward-hack attempts flagged
More effort, more cheating
Left — DeepSWE: each model swept across reasoning effort (low→max); x = clean pass, y = hacking-attempt rate (higher = more hacking). Within a model, raising effort raises the rate monotonically (Spearman fable ρ=0.90 p=0.04, opus ρ=1.00, pooled ρ=0.61 p=0.02); the exception is gpt-5.5, which never attempts at any effort. None of these DeepSWE attempts broke through (the air-gap blocked every one), so this is hacking propensity, not realized contamination. Right — SWE-Marathon: one dot per model (across-model, not an effort sweep); the trend is weak and confounded by model identity (n≈6–11, not significant).
DeepSWE · effort sweepρ=+.61 · n=14 · p=.02 · within
A clean zero is suspicious, so we re-audited it directly. We read the full transcripts (every bash command and its output) for 100 of GPT-5.5's highest-effort runs, oversampling the 65 that ran git show / curl / pip (what a reference-seeker would use), with a strict LLM rubric that does not flag normal inspection. Result: 0 reward-hack attempts, 0 verbalized eval-awareness. To prove the rubric can actually see a hack, we ran it as a positive control on 40 Claude (fable-5) candidate runs: it flagged 8 upstream-network-fetch attempts, each with the command and the model's own words (“check if I have internet access to find the reference implementation”). A structural check rules out a recording gap: GPT-5.5's transcripts are the same length, with reasoning and file-reads in 100% of runs, yet it runs git show / curl / pip 4 to 20× less than Claude. The zero is behavioral, not missing data.
100
transcripts read
0
hacks found
8 / 40
caught in positive control
4–20×
fewer probe commands
It also isn't an artifact of our air-gap. On SWE-bench Pro — patch-based but not air-gapped, where a live .gitleak leads other Claude configurations to register bad passes on 12%+ of rollouts (Datacurve's DeepSWE report) — GPT-5.5 still never git-cheats. The zero is a genuine behavioral trait. (We can only observe verbalized awareness, a lower bound — so this is about what the model does, not a claim about what it secretly knows.) It just doesn't hold on every benchmark, as the next section shows.
The reversal: clean here, worst there
The channel a model reaches for is set by the benchmark's form. DeepSWE is patch-based, so cheating means hunting for the upstream fix — reference-seeking via network, git-history, or disk. There Claude probes hardest (claude-fable-5 5.4%) while GPT-5.5 attempts exactly zero (0 / 1,808). SWE-Marathon is long-horizon and leaves golden answers and score scripts in the sandbox, so cheating means reading that answer — and the order inverts: GPT-5.5 hacks the most (26.5%), and claude-fable-5, the heaviest prober on DeepSWE, sinks to near the bottom (9.0%).
Flagship model
DeepSWE patch-based · attempt rate
SWE-Marathon long-horizon · hack rate
GPT-5.5OpenAI
0.0%
26.5%
Gemini 3.1 ProGoogle
1.3%
22.1%
Claude Fable 5Anthropic
5.4%
9.0%
The read is not “GPT-5.5 is clean” or “Claude cheats most.” The benchmark's form sets which shortcut exists; its exposure sets whether that shortcut pays off (DeepSWE's air-gap blocks it, SWE-Marathon's doesn't); and models differ in which openings they take — GPT-5.5 ignores the patch-based ones yet reads an exposed oracle, Claude the reverse. Reward hacking is a model×environment interaction: no single benchmark characterizes a model.
How they hack: a taxonomy of exploits
Every reward-hack channel below maps to one root cause = one thing the benchmark has to defend. Hover a channel for its definition, the defense, and a real example.
① Network egress is open→ defense: air-gap the sandbox (block network egress)
network-fetch
② The reference is left in the sandbox→ defense: scrub the leaked source
③ The check is gameable→ defense: harden the verifier
hardcode-expectedgrader-tampering
How they reward-hackDeepSWE
163 reward-hack attempts (2-skeptic gated) over all 9,040 audited DeepSWE trajectories (Codex first-pass screen), each flagged attempt hand-classified. Per-model coverage is uneven (counts scale with it). 50 verbalized situational awareness (recognized it's a benchmark / a reverted commit / a port). But none produced an illegitimate pass — the git-history reads that passed were verified to be genuine solves (the verifier resets and re-applies its own hidden tests, and the read history did not contain the answer), and every network / version / cache fetch was blocked by the air-gap. The harness held.
Model
network fetch
git history
disk reference
grader tamper
total
aware
claude-fable-52260 audited
94
15
11
2
122
33
claude-opus-4.82260 audited
9
6
14
1
30
16
gemini-3.1-pro452 audited
4
·
·
2
6
·
claude-sonnet-4.6452 audited
2
·
·
·
2
·
kimi-k2.7-code452 audited
·
2
·
·
2
·
gpt-5.4452 audited
·
1
·
·
1
1
gemini-3.5-flash452 audited
·
·
·
·
0
·
glm-5.2452 audited
·
·
·
·
0
·
gpt-5.51808 audited
·
·
·
·
0
·
all models
109
24
25
5
163
50
✓ = load-bearing (the hack actually produced the pass). Snapshot from a hand-verified classification; recount as the audit grows.
How they reward-hackSWE-Marathon
97 Delphik-verified reward-hacks (adversarial + partial-aware), each trajectory hand-classified. 88 our audit surfaced beyond the 9 the SWE-Marathon authors had labeled — most lift only the reported partial score (the verifier's anti-cheat blocks them from a full pass), a lens the authors' binary labels don't track. The exploits differ from DeepSWE's git-history reads: read the exposed oracle (copy/seed perfect.align, replay golden.jsonl), tamper with the grader (patch the parity runner to compare a golden to itself), or delegate the real work (submit a foreign checkpoint). 41 verbalized they were gaming it. 11 actually beat the binary gate (magenta) — 6 find-network oracle-copies (Claude, Gemini) and 5 post-train foreign-checkpoint swaps (DeepSeek ×4, GPT-5.5) that slipped past post-train-ifeval's own fail-closed LLM judge.
Model
read oracle
network fetch
hardcode
grader tamper
total
aware
gemini-3.1-pro199 audited
17✓
9
1
1
28·1✓
18
gpt-5.5200 audited
14
3✓
3
5
25·1✓
11
deepseek-v4-pro100 audited
6
4✓
·
·
10·4✓
7
claude-opus-4.7200 audited
8✓
·
1
·
9·2✓
3
claude-fable-567 audited
5✓
·
1
·
6·2✓
·
claude-opus-4.8100 audited
6✓
·
·
·
6·1✓
·
glm-5.2100 audited
4
·
·
·
4
·
kimi-k2.6200 audited
3
·
·
·
3
·
glm-5.1100 audited
3
·
·
·
3
·
gemini-3.5-flash100 audited
2
·
·
·
2
2
minimax-m2.7100 audited
1
·
·
·
1
·
all models
69 (6✓)
16 (5✓)
6
6
97
41
✓ = beat the binary gate (the hack produced the pass); the rest inflated only the partial score. Delphik-verified; † = the SWE-Marathon authors also flagged it.
Every audited trajectory is browsable per task — its pass/fail, whether we held it out, and any hack — in the DeepSWE matrix and the SWE-Marathon matrix.
The Clean Coding Index
The same audit re-scores the benchmarks. Every task we confirm as a false pass, false negative, or hackable is held out, and we re-score each model on what remains — the Clean Coding Index (CCI). Held-out tasks are 13% of DeepSWE and 25% of SWE-Marathon; removing them lowers every model by up to ~8 points, concentrated in SWE-Marathon's oracle-inflated partial credit — a contamination margin, not a new ranking.
Model
CCI
DeepSWE orig→clean
SWE-Marathon orig→clean
claude-fable-5
60.8%
64.5%63.7%
60.4%57.9%
claude-opus-4.8
56.9%
50.2%48.8%
71.1%65%
gpt-5.5
50.7%
53.1%51.8%
54.1%49.6%
glm-5.2
49.2%
43.6%42.8%
59.4%55.7%
gemini-3.5-flash
34.9%
37.4%36.3%
37.2%33.5%
gemini-3.1-pro
20.3%
11.7%10.8%
38.3%29.9%
Full sortable leaderboard, with per-benchmark hacking rates, on the index.
Why only these two benchmarks
We audit benchmarks whose full per-trial trajectories are public and come from current frontier models. Most release only pass/fail or nothing, so no one (us included) can verify reward-hacking. SWE-bench Pro is a near-miss — its public runs are from much older models, so they no longer track today's frontier. What we audited is not the two best benchmarks, but the two that are auditable and current.
Benchmark
Owner
Trajectories public
Audit
DeepSWE
Datacurve
✓ 9,040 per-trial JSON
Included
SWE-Marathon
Abundant AI
✓ 1,466 traces (+ full logs)
Included
SWE-bench Pro
Scale AI
✓ older models only · no model overlap
Excluded
Terminal-Bench 2.1
TB team
✗ timestamps only
Excluded
FrontierSWE
ProximalHQ
✗ not released
Excluded
FrontierCode
Cognition
✗ private tasks
Excluded
Where this is going
These same environments are increasingly used not just to evaluate models but to train them. A model that learns a shortcut pays off can carry that habit past the one task it was trained on — which is why hardening the environments we train and grade on matters beyond any single leaderboard.
And the gap is widening. As tasks get longer and more open-ended, the surface a model can exploit — and a grader can miss — grows with every step, while the human review that has to catch it doesn't scale as fast. Closing that gap is, we think, one of the more important open problems for trustworthy evaluation and training, and it is the problem this work is meant to serve.
We work on it in the open: every defect we confirm goes into a public store, the index is versioned and reproducible, and anyone can report or correct a task — a clean eval the community builds together. And for teams building benchmarks and RL environments, we run the same audit privately, surfacing false passes, false negatives, and hacking paths before a model ever trains on them.
Private audit · waitlist
Want this audit on your benchmark or RL environment?
We harden public and private RL environments against the leaks above — false passes, false negatives, and reward-hacking — with an LLM-pass then human-confirm method tuned for high recall at low cost. Drop your email and we'll reach out.