Coding Index · DeepSWE

DeepSWE

datacurve-ai/deep-swe · 113 tasks · by Datacurve. Each row is one (model, effort) unit, 4 trials per cell. Tasks with a confirmed false positive / false negative are held out of the score.

coding index →benchmark: DeepSWE v1.1 · upstream20 model×effort units · 113 tasks

Agent · Model	clean	misalign		narwhals-rolling-window-suite ↗	true-myth-iterable-collection-combinators ↗	koota-entity-snapshot-rollback ↗	psd-tools-blend-range-api ↗	go ⚠		sql-formatter-bigquery-pipe-formatting ↗	go ⚠	httpx-deterministic-cookie-store ↗	returns-validated-error-accumulation ↗	testem-per-launcher-reports ↗	vitest-duration-sharding ↗	abs-module-cache-flags ↗go ⚠	ytt-jsonpath-query-api ↗go ⚠	anko-default-function-arguments ↗go ⚠	task-task-graph-export ↗go ⚠	abs-stepped-slices ↗go ⚠	obsidian-linter-scoped-ignore-markers ↗	wazero-multi-module-snapshots ↗go ⚠	aiomonitor-task-snapshots-diff ↗	goreleaser-retry-publish-auditing ↗go ⚠	yjs-map-conflict-detection ↗	dynamodb-toolbox-lazy-recursive-schemas ↗		kysely-window-grouping-helpers ↗	ts-pattern-match-each ↗	wasmi-trap-coredumps ↗	adaptix-name-mapping-aliases ↗	go-genai-streamed-function-args ↗go ⚠	tomlkit-toml-table-converters ↗	cattrs-partial-structuring-recovery ↗	ofetch-per-origin-circuit-breaker ↗	dynamodb-toolbox-conditional-attribute-requirements ↗	textual-richlog-follow-state ↗		geo-shapeindex-serialization ↗go ⚠		arcane-drift-detection-baselines ↗go ⚠	opa-rego-rule-profiling ↗go ⚠	opa-template-string-reconstruction ↗go ⚠	httpx-multipart-response-parsing ↗	numba-stencil-boundary-modes ↗	query-persist-restored-query-state ↗	tengo-callable-instance-isolation ↗go ⚠	kombu-single-active-consumer-priority ↗	scc-bounded-memory-spilling ↗go ⚠	skrub-duration-encoding ↗	pebble-durability-wait-apis ↗go ⚠	scriggo-method-declarations ↗go ⚠	dasel-html-document-format ↗go ⚠	fastapi-deprecation-response-headers ↗	go-git-worktree-merge-conflicts ↗go ⚠	mobly-grouped-test-barriers ↗	tengo-destructuring-bindings ↗go ⚠	bandit-incremental-cache-control ↗	go-critic-doc-link-checker ↗go ⚠	sqlite-utils-safe-import-checkpoints ↗			igel-persist-feature-schema ↗	boa-hierarchical-evaluation-cancellation ↗	fastapi-implicit-head-options ↗	bandit-interprocedural-taint-checks ↗	ipython-session-bundle-replay ↗	mnamer-daemon-watch-lifecycle ↗	kgateway-consistent-hash-policy ↗go ⚠	claude-code-by-agents-recursive-delegation ↗	fd-deterministic-multi-key-sorting ↗	cliffy-config-file-parsing ↗	prometheus-typed-label-sorting ↗go ⚠	textual-kitty-key-phases ↗	python-statemachine-state-data-scoping ↗	awilix-async-container-initialization ↗	langchain-request-coalescing ↗		eicrud-keyset-pagination-cursor ↗	kea-atomic-signal-selectors ↗	mashumaro-flattened-dataclass-fields ↗	clack-async-autocomplete-options ↗	arktype-json-schema-refs-dependencies ↗	dateutil-rfc5545-timezone-interop ↗	happy-dom-deterministic-intersectionobserver ↗				koota-pair-relation-tracking ↗	pwntools-tube-multiplexing ↗	expr-try-catch-errors ↗go ⚠	koota-composite-trait-aspects ↗	sqlfmt-create-table-ddl-formatting ↗	helm-array-merge-strategies ↗go ⚠	katex-multicolumn-array-spans ↗	obsidian-linter-link-format-conversion ↗	ink-grid-box-layout ↗	optique-conditional-option-dependencies ↗	pest-character-class-coalescing ↗	testem-bail-on-test-failure ↗	vulture-persistent-analysis-cache ↗	csstree-shorthand-expansion-compression ↗	kombu-virtual-queue-dead-lettering ↗	meriyah-explicit-resource-declarations ↗	oxvg-structural-selector-preservation ↗	prometheus-transactional-reload-status ↗go ⚠	quill-shared-toolbar-focus ↗	effect-sse-httpapi-streaming ↗	superjson-error-stack-serialization ↗	termenv-preserve-ansi-resets ↗go ⚠	koota-deferred-mutation-buffer ↗	gql-incremental-graphql-delivery ↗	updo-policy-alerting ↗go ⚠	bandit-structured-nosec-directives ↗	obsidian-linter-auto-table-of-contents ↗
mini-swe-agent · claude-fable-5 xhigh	70.3	9.7% 44/452	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 2/4	✓ 4/4	✓ 4/4	FP✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 2/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	◐ 3/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	✓ 4/4	◐ 2/4	✓ 4/4	◐ 2/4	◐ 2/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 3/4	◐ 2/4	✓ 4/4	◐ 2/4	◐ 3/4	FN◐ 2/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 3/4	◐ 3/4	◐ 2/4	◐ 1/4	✓ 4/4	✓ 4/4	◐ 1/4	◐ 3/4	✓ 4/4	◐ 1/4	◐ 3/4	✗ 0/4	◐ 1/4	◐ 1/4	◐ 3/4	◐ 3/4	◐ 2/4	✓ 4/4	✓ 4/4	✗ 0/4	◐ 3/4	FN◐ 1/4	✓ 4/4	✗ 0/4	◐ 2/4	◐ 3/4	◐ 1/4	✗ 0/4	✓ 4/4	◐ 1/4	◐ 1/4	✗ 0/4	◐ 1/4	◐ 3/4	✗ 0/4	✗ 0/4	◐ 1/4	✗ 0/4	◐ 2/4	◐ 2/4	◐ 1/4	◐ 2/4	✗ 0/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 1/4
mini-swe-agent · claude-fable-5 max	66.8	10.2% 46/452	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	FP✓ 4/4	✓ 4/4	✓ 4/4	FP✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 2/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	✓ 4/4	◐ 2/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 2/4	◐ 2/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 3/4	◐ 2/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 3/4	◐ 3/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	◐ 1/4	✓ 4/4	FN◐ 1/4	✓ 4/4	◐ 3/4	✓ 4/4	◐ 2/4	◐ 1/4	◐ 3/4	◐ 3/4	◐ 2/4	✗ 0/4	◐ 3/4	✓ 4/4	◐ 2/4	◐ 3/4	✓ 4/4	✗ 0/4	◐ 2/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 2/4	◐ 2/4	◐ 2/4	✓ 4/4	✓ 4/4	✗ 0/4	✓ 4/4	◐ 2/4	✓ 4/4	✗ 0/4	◐ 2/4	✗ 0/4	◐ 1/4	✗ 0/4	◐ 2/4	✗ 0/4	◐ 1/4	◐ 2/4	◐ 3/4	✓ 4/4	✗ 0/4	◐ 1/4	✗ 0/4	◐ 1/4	◐ 3/4	◐ 1/4	✗ 0/4	◐ 1/4	◐ 2/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4
mini-swe-agent · gpt-5.5 xhigh	66	0% 0/452	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	FP✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 3/4	◐ 2/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	◐ 2/4	◐ 3/4	◐ 3/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 2/4	◐ 3/4	✓ 4/4	◐ 3/4	◐ 2/4	✓ 4/4	✓ 4/4	◐ 3/4	◐ 3/4	◐ 3/4	✓ 4/4	◐ 3/4	✓ 4/4	◐ 3/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	◐ 3/4	✓ 4/4	✓ 4/4	✗ 0/4	✓ 4/4	✓ 4/4	◐ 2/4	◐ 1/4	✓ 4/4	◐ 2/4	◐ 2/4	◐ 3/4	◐ 1/4	✓ 4/4	◐ 3/4	◐ 2/4	✓ 4/4	◐ 2/4	✗ 0/4	◐ 1/4	◐ 3/4	◐ 3/4	✗ 0/4	◐ 2/4	◐ 2/4	✗ 0/4	◐ 3/4	◐ 1/4	◐ 1/4	◐ 1/4	◐ 1/4	◐ 1/4	✓ 4/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 1/4	◐ 3/4	◐ 1/4	◐ 1/4	✗ 0/4	◐ 2/4	✗ 0/4	✓ 4/4	◐ 1/4	◐ 1/4	✗ 0/4	◐ 2/4	✓ 4/4	◐ 1/4	◐ 2/4	✗ 0/4	◐ 1/4	✗ 0/4	✗ 0/4
mini-swe-agent · claude-fable-5 high	65	8.4% 38/452	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	✓ 4/4	FP✓ 4/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	✓ 4/4	◐ 3/4	◐ 2/4	✓ 4/4	✓ 4/4	◐ 3/4	◐ 3/4	◐ 3/4	◐ 3/4	✓ 4/4	✓ 4/4	FP◐ 3/4	✓ 4/4	◐ 3/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 2/4	◐ 3/4	◐ 3/4	◐ 3/4	◐ 2/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 2/4	✓ 4/4	FN◐ 2/4	FN◐ 2/4	◐ 3/4	◐ 3/4	✓ 4/4	✓ 4/4	◐ 2/4	✗ 0/4	✓ 4/4	◐ 1/4	◐ 2/4	✓ 4/4	◐ 3/4	◐ 3/4	✓ 4/4	◐ 1/4	◐ 2/4	◐ 1/4	◐ 3/4	✗ 0/4	◐ 3/4	◐ 2/4	◐ 2/4	◐ 1/4	◐ 2/4	FP◐ 1/4	FP◐ 2/4	FN◐ 2/4	◐ 3/4	✗ 0/4	◐ 3/4	◐ 2/4	◐ 1/4	◐ 1/4	◐ 1/4	✗ 0/4	◐ 1/4	✗ 0/4	◐ 2/4	◐ 2/4	✗ 0/4	◐ 2/4	◐ 2/4	✗ 0/4	◐ 2/4	✗ 0/4	◐ 2/4	✗ 0/4	✗ 0/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4
mini-swe-agent · gpt-5.5 high	63.5	0% 0/452	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 3/4	FP✓ 4/4	◐ 3/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 3/4	◐ 3/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	◐ 2/4	◐ 3/4	✓ 4/4	◐ 3/4	◐ 2/4	◐ 3/4	✓ 4/4	✓ 4/4	◐ 3/4	◐ 2/4	◐ 2/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	◐ 3/4	✓ 4/4	◐ 1/4	◐ 1/4	◐ 3/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 2/4	◐ 2/4	◐ 1/4	◐ 2/4	◐ 2/4	✓ 4/4	◐ 2/4	◐ 1/4	◐ 2/4	✗ 0/4	◐ 3/4	◐ 2/4	◐ 1/4	◐ 3/4	◐ 1/4	◐ 1/4	◐ 1/4	◐ 3/4	◐ 3/4	✗ 0/4	◐ 1/4	◐ 3/4	◐ 1/4	◐ 1/4	◐ 1/4	✗ 0/4	◐ 2/4	◐ 3/4	◐ 2/4	◐ 3/4	◐ 1/4	✗ 0/4	◐ 1/4	◐ 1/4	◐ 3/4	◐ 3/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 2/4	✗ 0/4	◐ 2/4	◐ 1/4	◐ 1/4	◐ 3/4	◐ 1/4	◐ 1/4	◐ 1/4	◐ 2/4	✗ 0/4	◐ 2/4	✗ 0/4	✗ 0/4
mini-swe-agent · claude-fable-5 medium	61.3	6.9% 31/452	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	FP◐ 3/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 1/4	◐ 3/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	FN◐ 3/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	◐ 3/4	✓ 4/4	✓ 4/4	◐ 3/4	◐ 3/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 3/4	◐ 1/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 2/4	◐ 3/4	✓ 4/4	◐ 2/4	◐ 2/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	◐ 3/4	◐ 2/4	◐ 2/4	✓ 4/4	◐ 3/4	◐ 2/4	◐ 3/4	◐ 3/4	◐ 3/4	◐ 3/4	◐ 3/4	◐ 2/4	◐ 1/4	✗ 0/4	✗ 0/4	◐ 1/4	✓ 4/4	✓ 4/4	◐ 3/4	◐ 3/4	✓ 4/4	◐ 3/4	◐ 1/4	FN◐ 3/4	◐ 3/4	◐ 1/4	◐ 3/4	◐ 3/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 2/4	◐ 3/4	◐ 1/4	◐ 2/4	✗ 0/4	◐ 3/4	✗ 0/4	✓ 4/4	✗ 0/4	◐ 3/4	◐ 1/4	◐ 2/4	✗ 0/4	◐ 2/4	◐ 1/4	✗ 0/4	✗ 0/4	◐ 1/4	✗ 0/4	✗ 0/4	◐ 2/4	◐ 2/4	✗ 0/4	✗ 0/4	◐ 2/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4
mini-swe-agent · claude-fable-5 low	55.3	2.7% 12/452	◐ 3/4	✓ 4/4	◐ 3/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	✓ 4/4	FP✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✗ 0/4	◐ 3/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 2/4	✓ 4/4	◐ 2/4	◐ 3/4	◐ 3/4	◐ 3/4	✓ 4/4	✓ 4/4	◐ 3/4	◐ 3/4	✓ 4/4	✓ 4/4	◐ 3/4	◐ 1/4	◐ 3/4	◐ 3/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	◐ 2/4	◐ 3/4	◐ 1/4	✓ 4/4	✓ 4/4	◐ 2/4	◐ 1/4	◐ 2/4	◐ 2/4	◐ 3/4	◐ 3/4	◐ 1/4	◐ 2/4	✓ 4/4	◐ 2/4	◐ 2/4	◐ 3/4	◐ 3/4	FN◐ 3/4	◐ 3/4	◐ 2/4	◐ 1/4	✗ 0/4	◐ 3/4	✗ 0/4	◐ 2/4	◐ 3/4	◐ 2/4	◐ 1/4	◐ 2/4	◐ 2/4	◐ 2/4	◐ 2/4	◐ 2/4	✓ 4/4	✓ 4/4	◐ 3/4	◐ 2/4	◐ 3/4	◐ 3/4	✗ 0/4	✗ 0/4	◐ 3/4	FP◐ 1/4	FP◐ 2/4	✗ 0/4	◐ 3/4	✗ 0/4	◐ 3/4	✗ 0/4	◐ 2/4	✗ 0/4	◐ 2/4	✗ 0/4	◐ 1/4	✗ 0/4	◐ 1/4	◐ 1/4	◐ 1/4	✗ 0/4	◐ 3/4	✗ 0/4	✗ 0/4	◐ 3/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 2/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 1/4	✗ 0/4
mini-swe-agent · claude-opus-4.8 max	55	4.6% 21/452	✓ 4/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	FP✓ 4/4	◐ 3/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	◐ 2/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	◐ 2/4	◐ 3/4	◐ 1/4	◐ 3/4	✓ 4/4	◐ 3/4	✓ 4/4	◐ 2/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✗ 0/4	✓ 4/4	◐ 3/4	✓ 4/4	◐ 1/4	◐ 1/4	◐ 3/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	◐ 2/4	✓ 4/4	◐ 1/4	◐ 3/4	✓ 4/4	◐ 3/4	✓ 4/4	✓ 4/4	◐ 2/4	◐ 2/4	◐ 2/4	✓ 4/4	✓ 4/4	✗ 0/4	◐ 3/4	FN◐ 1/4	◐ 2/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 3/4	◐ 3/4	◐ 2/4	◐ 1/4	◐ 3/4	◐ 3/4	◐ 2/4	◐ 2/4	✗ 0/4	◐ 1/4	◐ 1/4	FN◐ 2/4	◐ 2/4	✗ 0/4	✗ 0/4	◐ 2/4	◐ 2/4	◐ 1/4	◐ 1/4	◐ 1/4	◐ 3/4	✗ 0/4	◐ 1/4	◐ 2/4	✗ 0/4	◐ 2/4	◐ 1/4	◐ 3/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 1/4	✗ 0/4	◐ 3/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4
mini-swe-agent · claude-opus-4.8 xhigh	53.3	2% 9/452	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 2/4	FP✓ 4/4	✓ 4/4	FP◐ 2/4	◐ 3/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 2/4	◐ 3/4	✓ 4/4	✓ 4/4	◐ 3/4	◐ 3/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 1/4	◐ 1/4	✓ 4/4	◐ 2/4	✓ 4/4	◐ 1/4	FN✗ 0/4	✓ 4/4	◐ 3/4	✓ 4/4	✗ 0/4	◐ 3/4	◐ 1/4	◐ 3/4	◐ 3/4	◐ 3/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 1/4	◐ 3/4	◐ 2/4	✓ 4/4	◐ 3/4	◐ 1/4	◐ 1/4	FN◐ 1/4	FN◐ 1/4	◐ 1/4	◐ 3/4	◐ 3/4	◐ 3/4	◐ 2/4	◐ 3/4	◐ 3/4	◐ 1/4	◐ 2/4	◐ 1/4	◐ 1/4	◐ 2/4	✗ 0/4	✗ 0/4	◐ 2/4	✓ 4/4	✗ 0/4	◐ 1/4	✗ 0/4	◐ 1/4	◐ 1/4	◐ 2/4	✗ 0/4	✗ 0/4	◐ 1/4	◐ 3/4	✗ 0/4	◐ 3/4	◐ 1/4	◐ 1/4	◐ 2/4	◐ 2/4	◐ 2/4	◐ 2/4	✗ 0/4	◐ 1/4	◐ 1/4	◐ 2/4	✗ 0/4	◐ 1/4	◐ 2/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4
mini-swe-agent · gpt-5.5 medium	53	0% 0/452	FP✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	FP✓ 4/4	◐ 3/4	✓ 4/4	◐ 2/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	✓ 4/4	◐ 3/4	◐ 2/4	◐ 3/4	◐ 2/4	✓ 4/4	◐ 3/4	✓ 4/4	◐ 2/4	✓ 4/4	✓ 4/4	◐ 1/4	◐ 2/4	◐ 1/4	✗ 0/4	◐ 3/4	✓ 4/4	◐ 3/4	◐ 2/4	◐ 2/4	✓ 4/4	✓ 4/4	◐ 2/4	◐ 1/4	◐ 3/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	✓ 4/4	◐ 3/4	◐ 1/4	◐ 3/4	✗ 0/4	◐ 1/4	✓ 4/4	✗ 0/4	◐ 3/4	✓ 4/4	◐ 2/4	✓ 4/4	◐ 2/4	◐ 3/4	FN◐ 2/4	✗ 0/4	◐ 2/4	✓ 4/4	✗ 0/4	◐ 3/4	✓ 4/4	✓ 4/4	◐ 1/4	✗ 0/4	✗ 0/4	◐ 3/4	◐ 1/4	◐ 3/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 3/4	✗ 0/4	◐ 1/4	◐ 1/4	◐ 3/4	◐ 2/4	◐ 3/4	✗ 0/4	✗ 0/4	◐ 1/4	✗ 0/4	◐ 1/4	◐ 2/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 2/4	◐ 1/4	◐ 1/4	◐ 1/4	✗ 0/4	◐ 1/4	◐ 3/4	✗ 0/4	◐ 3/4	✗ 0/4	◐ 1/4	◐ 1/4	◐ 3/4	✓ 4/4	✗ 0/4	◐ 2/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4
mini-swe-agent · gpt-5.4 xhigh	50.5	0.2% 1/452	◐ 3/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 3/4	◐ 3/4	✓ 4/4	✓ 4/4	◐ 2/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	◐ 2/4	✓ 4/4	◐ 3/4	◐ 2/4	◐ 2/4	◐ 3/4	◐ 3/4	◐ 1/4	◐ 2/4	◐ 1/4	◐ 3/4	◐ 2/4	◐ 3/4	◐ 2/4	✓ 4/4	◐ 2/4	◐ 2/4	✓ 4/4	✓ 4/4	◐ 3/4	◐ 2/4	✓ 4/4	✓ 4/4	◐ 1/4	◐ 2/4	◐ 2/4	◐ 3/4	◐ 2/4	◐ 2/4	✓ 4/4	◐ 1/4	✓ 4/4	◐ 1/4	◐ 2/4	✓ 4/4	✗ 0/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 3/4	◐ 1/4	✗ 0/4	◐ 3/4	✓ 4/4	✓ 4/4	◐ 2/4	◐ 1/4	◐ 1/4	◐ 3/4	✗ 0/4	✓ 4/4	◐ 3/4	✓ 4/4	◐ 2/4	✗ 0/4	◐ 3/4	◐ 2/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 1/4	◐ 3/4	◐ 2/4	✗ 0/4	✓ 4/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 3/4	◐ 1/4	◐ 2/4	✗ 0/4	◐ 3/4	◐ 1/4	✗ 0/4	◐ 1/4	✗ 0/4	◐ 1/4	✓ 4/4	✗ 0/4	✗ 0/4	◐ 1/4	◐ 1/4	✗ 0/4	◐ 3/4	◐ 2/4	✗ 0/4	✗ 0/4	◐ 2/4	◐ 1/4	◐ 1/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4
mini-swe-agent · claude-opus-4.8 high	50.2	1.1% 5/452	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	◐ 3/4	✓ 4/4	✓ 4/4	✓ 4/4	FP◐ 2/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 3/4	◐ 2/4	◐ 3/4	✓ 4/4	✓ 4/4	◐ 1/4	✓ 4/4	◐ 2/4	✓ 4/4	◐ 3/4	✓ 4/4	✓ 4/4	◐ 2/4	◐ 3/4	✓ 4/4	◐ 3/4	✓ 4/4	◐ 3/4	◐ 3/4	✓ 4/4	◐ 3/4	✓ 4/4	✗ 0/4	◐ 2/4	✓ 4/4	◐ 2/4	◐ 3/4	◐ 1/4	✓ 4/4	◐ 3/4	◐ 1/4	◐ 3/4	◐ 3/4	◐ 2/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 1/4	◐ 1/4	◐ 2/4	◐ 2/4	✓ 4/4	◐ 1/4	◐ 2/4	FN◐ 2/4	◐ 2/4	✓ 4/4	◐ 2/4	◐ 2/4	◐ 2/4	◐ 3/4	◐ 1/4	✓ 4/4	◐ 1/4	◐ 1/4	✗ 0/4	◐ 1/4	◐ 3/4	✗ 0/4	✗ 0/4	◐ 2/4	FN◐ 3/4	✗ 0/4	◐ 2/4	✗ 0/4	◐ 2/4	◐ 2/4	◐ 2/4	✗ 0/4	✗ 0/4	◐ 1/4	◐ 1/4	✗ 0/4	◐ 2/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 3/4	◐ 2/4	◐ 1/4	✗ 0/4	✗ 0/4	◐ 1/4	✗ 0/4	✗ 0/4	◐ 1/4	◐ 2/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 1/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4
mini-swe-agent · claude-opus-4.8 medium	48	0.4% 2/452	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	✓ 4/4	◐ 3/4	FP◐ 2/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 2/4	◐ 3/4	✓ 4/4	✓ 4/4	◐ 1/4	◐ 1/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	◐ 2/4	✓ 4/4	◐ 3/4	◐ 2/4	◐ 3/4	✗ 0/4	◐ 2/4	◐ 3/4	◐ 3/4	◐ 3/4	◐ 1/4	✓ 4/4	✓ 4/4	✗ 0/4	✓ 4/4	◐ 1/4	✓ 4/4	◐ 1/4	◐ 2/4	✓ 4/4	✗ 0/4	◐ 1/4	◐ 1/4	◐ 2/4	◐ 3/4	◐ 1/4	◐ 3/4	FN◐ 1/4	FN✗ 0/4	◐ 3/4	◐ 1/4	◐ 1/4	◐ 2/4	✗ 0/4	✗ 0/4	◐ 3/4	◐ 3/4	✗ 0/4	◐ 3/4	◐ 1/4	◐ 2/4	◐ 1/4	✗ 0/4	✗ 0/4	◐ 2/4	◐ 2/4	◐ 3/4	✗ 0/4	◐ 1/4	◐ 2/4	✗ 0/4	✗ 0/4	◐ 1/4	◐ 1/4	✗ 0/4	✗ 0/4	◐ 3/4	◐ 2/4	✗ 0/4	◐ 1/4	◐ 2/4	✗ 0/4	◐ 1/4	✗ 0/4	◐ 1/4	◐ 1/4	◐ 2/4	◐ 1/4	◐ 1/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4
mini-swe-agent · glm-5.2 max	42.8	0% 0/452	✓ 4/4	✓ 4/4	◐ 2/4	✓ 4/4	◐ 3/4	◐ 3/4	✓ 4/4	✓ 4/4	FP◐ 3/4	◐ 3/4	✓ 4/4	◐ 3/4	◐ 3/4	✓ 4/4	◐ 2/4	◐ 2/4	◐ 2/4	◐ 1/4	◐ 2/4	◐ 2/4	◐ 1/4	◐ 3/4	◐ 3/4	◐ 2/4	◐ 2/4	◐ 2/4	◐ 2/4	◐ 3/4	✓ 4/4	◐ 3/4	◐ 2/4	◐ 3/4	◐ 1/4	◐ 2/4	◐ 1/4	◐ 3/4	✗ 0/4	◐ 2/4	◐ 3/4	✓ 4/4	◐ 3/4	✗ 0/4	✗ 0/4	✓ 4/4	◐ 2/4	✗ 0/4	✓ 4/4	◐ 1/4	◐ 1/4	◐ 2/4	✓ 4/4	✓ 4/4	✓ 4/4	✗ 0/4	◐ 2/4	✗ 0/4	◐ 3/4	◐ 1/4	FN◐ 2/4	FN◐ 1/4	◐ 3/4	◐ 1/4	◐ 2/4	◐ 2/4	◐ 3/4	◐ 1/4	✗ 0/4	◐ 2/4	◐ 1/4	◐ 2/4	◐ 1/4	✗ 0/4	◐ 1/4	◐ 3/4	◐ 2/4	FN◐ 1/4	◐ 3/4	◐ 2/4	◐ 2/4	◐ 1/4	◐ 1/4	✗ 0/4	◐ 1/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 1/4	✗ 0/4	◐ 1/4	◐ 1/4	◐ 1/4	✗ 0/4	◐ 2/4	◐ 2/4	◐ 1/4	◐ 1/4	✗ 0/4	✓ 4/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 1/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 2/4	✗ 0/4	✗ 0/4	✗ 0/4
mini-swe-agent · claude-opus-4.8 low	37.5	0% 0/452	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	FP◐ 3/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 3/4	◐ 3/4	✓ 4/4	◐ 3/4	◐ 1/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 1/4	◐ 3/4	◐ 1/4	✓ 4/4	◐ 2/4	◐ 2/4	◐ 1/4	✓ 4/4	✗ 0/4	◐ 2/4	✗ 0/4	✓ 4/4	◐ 1/4	✓ 4/4	◐ 3/4	◐ 3/4	◐ 2/4	✓ 4/4	◐ 3/4	◐ 1/4	◐ 3/4	◐ 3/4	◐ 2/4	◐ 1/4	✓ 4/4	◐ 1/4	◐ 2/4	◐ 1/4	◐ 1/4	◐ 1/4	✗ 0/4	✗ 0/4	◐ 1/4	✗ 0/4	◐ 2/4	◐ 3/4	◐ 1/4	FN◐ 2/4	◐ 2/4	◐ 1/4	◐ 1/4	◐ 1/4	◐ 2/4	✗ 0/4	◐ 1/4	✗ 0/4	◐ 3/4	✗ 0/4	◐ 2/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 1/4	◐ 1/4	◐ 3/4	✗ 0/4	✗ 0/4	◐ 1/4	✗ 0/4	✗ 0/4	◐ 1/4	◐ 1/4	◐ 3/4	✗ 0/4	◐ 2/4	✗ 0/4	✗ 0/4	◐ 1/4	◐ 2/4	✗ 0/4	✗ 0/4	◐ 1/4	✗ 0/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 1/4	✗ 0/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4
mini-swe-agent · gemini-3.5-flash medium	36.3	0% 0/452	✓ 4/4	◐ 2/4	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	◐ 2/4	◐ 3/4	FP◐ 3/4	◐ 3/4	✓ 4/4	✓ 4/4	◐ 3/4	◐ 3/4	✓ 4/4	◐ 2/4	◐ 2/4	✓ 4/4	✓ 4/4	✗ 0/4	◐ 3/4	◐ 2/4	◐ 3/4	✗ 0/4	◐ 3/4	◐ 3/4	✓ 4/4	◐ 2/4	✗ 0/4	◐ 2/4	◐ 3/4	◐ 2/4	◐ 3/4	◐ 1/4	✗ 0/4	◐ 1/4	✓ 4/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 2/4	✗ 0/4	◐ 1/4	◐ 1/4	◐ 1/4	◐ 2/4	◐ 2/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 2/4	◐ 3/4	◐ 3/4	◐ 2/4	✗ 0/4	◐ 2/4	◐ 1/4	◐ 1/4	◐ 1/4	◐ 3/4	◐ 1/4	◐ 1/4	◐ 1/4	✓ 4/4	✗ 0/4	✗ 0/4	◐ 3/4	◐ 3/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 1/4	◐ 1/4	◐ 2/4	◐ 2/4	◐ 1/4	✗ 0/4	◐ 1/4	◐ 3/4	✗ 0/4	◐ 3/4	◐ 1/4	FP◐ 2/4	✗ 0/4	◐ 1/4	✗ 0/4	✗ 0/4	◐ 1/4	✗ 0/4	✗ 0/4	◐ 2/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 1/4	◐ 1/4	◐ 1/4	✗ 0/4	✗ 0/4	◐ 2/4	✗ 0/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 1/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4
mini-swe-agent · claude-sonnet-4.6 high	29	0.4% 2/452	✓ 4/4	✓ 4/4	✓ 4/4	✓ 4/4	✗ 0/4	◐ 3/4	◐ 1/4	◐ 3/4	FP◐ 3/4	◐ 3/4	◐ 2/4	✓ 4/4	◐ 3/4	◐ 3/4	◐ 2/4	◐ 1/4	◐ 1/4	✗ 0/4	✗ 0/4	◐ 3/4	✗ 0/4	◐ 1/4	◐ 3/4	◐ 3/4	◐ 1/4	◐ 3/4	◐ 3/4	✓ 4/4	◐ 1/4	✗ 0/4	◐ 3/4	◐ 3/4	✗ 0/4	◐ 1/4	◐ 3/4	✗ 0/4	◐ 1/4	◐ 1/4	✗ 0/4	✗ 0/4	◐ 1/4	◐ 2/4	◐ 1/4	◐ 2/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 2/4	◐ 1/4	◐ 1/4	◐ 1/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 3/4	◐ 1/4	FN◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 1/4	✗ 0/4	◐ 3/4	◐ 3/4	◐ 3/4	◐ 1/4	✗ 0/4	◐ 2/4	◐ 2/4	◐ 1/4	◐ 2/4	◐ 1/4	✗ 0/4	◐ 1/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 1/4	✗ 0/4	◐ 2/4	✗ 0/4	◐ 3/4	✗ 0/4	✗ 0/4	◐ 2/4	◐ 3/4	✗ 0/4	◐ 3/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 2/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4
mini-swe-agent · kimi-k2.7-code	27.8	0.7% 3/452	◐ 3/4	◐ 3/4	◐ 3/4	◐ 1/4	◐ 2/4	✓ 4/4	◐ 3/4	◐ 1/4	FP✓ 4/4	◐ 1/4	◐ 2/4	◐ 1/4	◐ 2/4	◐ 3/4	◐ 2/4	◐ 3/4	◐ 2/4	◐ 3/4	◐ 1/4	✓ 4/4	✓ 4/4	◐ 3/4	✓ 4/4	◐ 1/4	◐ 3/4	✗ 0/4	◐ 1/4	✗ 0/4	✗ 0/4	✓ 4/4	◐ 1/4	◐ 2/4	◐ 2/4	◐ 2/4	◐ 1/4	◐ 1/4	✓ 4/4	◐ 3/4	◐ 2/4	◐ 1/4	◐ 2/4	◐ 1/4	◐ 2/4	◐ 2/4	◐ 1/4	◐ 1/4	◐ 2/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 1/4	◐ 3/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 1/4	◐ 1/4	◐ 1/4	✗ 0/4	✗ 0/4	◐ 1/4	◐ 1/4	✗ 0/4	✗ 0/4	◐ 1/4	✗ 0/4	◐ 2/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 1/4	✓ 4/4	◐ 2/4	◐ 1/4	✓ 4/4	✗ 0/4	◐ 1/4	✗ 0/4	◐ 2/4	✗ 0/4	✗ 0/4	◐ 3/4	✗ 0/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 1/4	✗ 0/4	◐ 2/4	◐ 1/4	◐ 1/4	✗ 0/4	✗ 0/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 2/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4
mini-swe-agent · gpt-5.5 low	24.5	0% 0/452	✓ 4/4	✓ 4/4	✓ 4/4	◐ 2/4	◐ 2/4	FP✓ 4/4	◐ 2/4	◐ 1/4	FP◐ 2/4	◐ 3/4	✗ 0/4	✗ 0/4	◐ 1/4	◐ 3/4	◐ 2/4	◐ 1/4	◐ 1/4	✓ 4/4	◐ 1/4	◐ 3/4	✗ 0/4	✓ 4/4	◐ 2/4	◐ 1/4	◐ 2/4	◐ 2/4	◐ 1/4	✗ 0/4	✗ 0/4	◐ 2/4	✗ 0/4	◐ 2/4	◐ 2/4	✗ 0/4	◐ 3/4	✗ 0/4	◐ 1/4	✓ 4/4	◐ 3/4	✗ 0/4	✗ 0/4	◐ 3/4	✗ 0/4	◐ 3/4	✗ 0/4	◐ 3/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 1/4	✗ 0/4	✗ 0/4	✓ 4/4	◐ 1/4	◐ 3/4	✗ 0/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	✓ 4/4	◐ 2/4	◐ 2/4	◐ 1/4	✗ 0/4	◐ 1/4	◐ 1/4	✗ 0/4	✗ 0/4	◐ 3/4	✗ 0/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 1/4	✗ 0/4	✗ 0/4	✓ 4/4	FP◐ 2/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 1/4	✗ 0/4	✗ 0/4	◐ 3/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 3/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4
mini-swe-agent · gemini-3.1-pro high	10.8	1.5% 7/452	◐ 1/4	✗ 0/4	◐ 3/4	✗ 0/4	✓ 4/4	◐ 1/4	✗ 0/4	◐ 2/4	FP◐ 3/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 1/4	◐ 1/4	✗ 0/4	◐ 1/4	◐ 2/4	✗ 0/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✓ 4/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 1/4	◐ 2/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 2/4	✗ 0/4	◐ 3/4	✗ 0/4	◐ 1/4	◐ 2/4	✗ 0/4	◐ 1/4	✗ 0/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 1/4	✗ 0/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 3/4	✗ 0/4	✗ 0/4	◐ 2/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 2/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 1/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	◐ 1/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4	✗ 0/4

FP false positive · weak verifier — a wrong solution passed (task under-tested). Held out of the score.

FP⚡ false positive · reward-hack — a load-bearing exploit caused the pass. Held out of the score.

FN false negative — a correct solution was failed. Held out of the score.

misalign — reward-hack behaviour that did not cause the pass (solved anyway). Counted.

go ⚠ Go test-isolation harness flaw. DeepSWE compiles each package's test files together and resets only files named in test.patch, so an agent-authored test that redeclares a hidden gold-test symbol breaks the package build and scores a correct solution 0. This latent flaw affects all 35 Go tasks. We held out the 6 where it produced a confirmed false negative (exhaustive 9040-trial sweep) and mark the rest go ⚠; their negative results may understate correct, test-writing models. The other Go fails are verified genuine failures, so we do not hold the whole Go subset out.