Delphik · open eval
We audit every released trajectory of frontier coding benchmarks for reward hacking — where a model passes by exploiting a defect instead of solving the task — then re-score capability with the contaminated tasks held out: the Clean Coding Index.
Why only two? They are the only recent coding benchmarks that publish every agent trajectory — the raw material an audit needs. We chose for auditability, not quality (full benchmark comparison).
Capability (x) vs hacking-attempt rate (y — higher = more hacking).
Left — DeepSWE: each model swept across reasoning effort, so more effort means more hacking attempts.
Right — SWE-Marathon: one dot per model.
The rank inverts across the two: gpt-5.5 attempts zero on patch-based DeepSWE (and stays clean on patch-based SWE-bench Pro too, which isn't air-gapped), yet hacks long-horizon SWE-Marathon the most. The split tracks the task form, not a fixed model trait — so behavior has to be audited across many benchmarks.
Capability re-scored with contaminated tasks held out (orig → clean), and the hacking-attempt rate alongside. Click any header to sort.
| # | Model | Coding Index ▾ | Hacking Attempt % ↕ | DeepSWE ↗ | SWE-Marathon ↗ | ||
|---|---|---|---|---|---|---|---|
| pass rate orig→clean 13% held out | hacking attempt % | partial score orig→clean 25% held out | hacking attempt % | ||||
| 1 | claude-fable-5 Anthropic | 60.8% | 8.3% | 64.5% 63.7% 7.6% mini-swe-agent · 5 cfg | 60.4% 57.9% 9% Claude Code · 1 cfg | ||
| 2 | claude-opus-4.8 Anthropic | 56.9% | 5.8% | 50.2% 48.8% 1.6% mini-swe-agent · 5 cfg | 71.1% 65% 10% Claude Code · 1 cfg | ||
| 3 | gpt-5.5 OpenAI | 50.7% | 13.3% | 53.1% 51.8% 0% mini-swe-agent · 4 cfg | 54.1% 49.6% 26.5% Codex, Terminus 2 · 2 cfg | ||
| 4 | glm-5.2 Z.ai | 49.2% | 3.5% | 43.6% 42.8% 0% mini-swe-agent · 1 cfg | 59.4% 55.7% 7% Claude Code · 1 cfg | ||
| 5 | gemini-3.5-flash | 34.9% | 6% | 37.4% 36.3% 0% mini-swe-agent · 1 cfg | 37.2% 33.5% 12% Gemini CLI · 1 cfg | ||
| 6 | gemini-3.1-pro | 20.3% | 11.8% | 11.7% 10.8% 1.5% mini-swe-agent · 1 cfg | 38.3% 29.9% 22.1% Gemini CLI, Terminus 2 · 2 cfg | ||
Read the full analysis → the audit method, who hacks where (provider × environment), how we verified GPT-5.5's zero, and the full reward-hack taxonomy.
Why this exists. Capable models increasingly exploit benchmark tasks instead of solving them. Labs curb this in training, but the fix doesn't carry to a new benchmark — so we measure it from the outside, from public trajectories alone (method).
How it runs. Living and versioned, not a one-off. A task is held out once we confirm a defect — false pass, false negative, or hackable — and restored once the benchmark fixes it; a benchmark retires when it saturates and newer ones take its place.
Part of an open loop. Every defect we confirm here lands in our open defect store, next to the defects we track across many other benchmarks (browse them on Delphik). Hit a false pass, false negative, or reward hack we missed? Report it and it flows into the same store. This is a clean eval the whole community builds together — open data, correctable by anyone.
Private audit · waitlist
We harden public and private RL environments against the leaks above — false passes, false negatives, and reward-hacking — with an LLM-pass then human-confirm method tuned for high recall at low cost. Drop your email and we'll reach out.