SWE-bench Verified vs Pro: why one says 88% and the other says 64%

What the gap actually tells you

If you have spent any time inside a procurement conversation in the last twelve months, you have seen a slide with a SWE-bench score on it. The number is almost always SWE-bench Verified, almost always close to or above 80%, and almost always presented as if it predicts how the model will behave on your codebase. It does not. The April 2026 leaderboard for SWE-bench Pro — the harder, larger, less gameable variant — has Claude Opus 4.7 at the top with 64.3%. The same model scores 87.6% on Verified. That is a 23.3 percentage-point gap on the same model, the same week, on benchmarks built from the same underlying GitHub corpus.

The gap is not a bug. It is the most informative single data point a builder can carry into a vendor evaluation, because it tells you how much of the headline score is real general code reasoning and how much is scaffold engineering, prompt patterns and overfitting to a 500-instance evaluation set the entire industry has been optimising against for two years.

The two benchmarks, side by side

Both benchmarks pull from real GitHub issues paired with the human-written test patches that resolve them. The model has to produce a code change that turns the failing tests green. The differences are in scale, filtering and how easily a clever scaffold can game the format.

Why the gap exists — three reasons

The 23-point delta is not random noise. It decomposes into three structural causes that every builder should be able to recite.

1. Test-suite filtering. Verified was hand-filtered to remove tickets where the test patch was wrong, the description was unclear, or the task was unsolvable in the snapshot. That filtering removes a category of failure that real-world tickets routinely contain — ambiguous repro steps, flaky tests, environment-dependent assertions. Pro keeps more of those, so a model that depends on tidy problem statements degrades.
2. Scaffold overfitting. A chunk of benchmark progress in 2024-2025 was scaffolding and prompt engineering specific to Verified's patterns, not general improvement in code reasoning. That is a direct quote of the framing in the morphllm coverage of SWE-bench Pro, and it is the single most uncomfortable sentence in modern code-LLM marketing. Pro breaks scaffolds that were tuned to the Verified instance distribution.
3. Distribution drift. Verified is a fixed slice of mostly Python repos from a defined snapshot of GitHub. Pro stretches that distribution. Models that learnt the shape of Django bug fixes do less well on the longer tail.

Verified leaderboard: the April 2026 snapshot

The full Verified leaderboard is the easiest place to see how compressed the top ten has become. Ten frontier models sit inside a ten-point band, which means the marginal cost of a one-point gain has become enormous — and almost certainly does not translate one-for-one into production behaviour.

For comparison, Mistral Devstral 2 — a developer-focused open-weight release — scores 72.2% on the same Verified benchmark, which is a reminder that the headline gap between top-tier proprietary and the strongest open-weight tier is much smaller than vendor decks usually claim. We covered the broader open-weight picture in the Claude Opus 4.7 long-context piece, and the practical agent-vs-agent comparison sits in the Cursor Composer 2 vs Claude Code piece.

What 'gameability' looks like in practice

'Gameable' is one of those words that benchmark people throw around without unpacking. Concretely, here is what it means inside a SWE-bench evaluation harness.

• File-localisation shortcuts. Verified instances tend to have a small number of files where the patch lives. A scaffold that aggressively narrows attention to a few candidate files lifts the score without improving the underlying reasoning. Pro distributions punish that shortcut.
• Test-aware prompting. Some scaffolds quietly include the failing test in the prompt, turning a debug task into a fill-in-the-blank task. Verified's clean test patches make that especially effective.
• Retry-and-vote loops. Running the same task 32 times and voting the dominant patch is cheap on a 500-instance benchmark and outright cheating on a deployment, but it shows up as a Verified bump.
• Repository-specific finetuning. When a benchmark is fixed, you can over-train on its repo style. Pro is larger and harder to over-train against, by construction.

None of these tactics make a model better at solving your tickets. They make it better at solving the 500 tickets in Verified. The Pro number strips most of that polish away.

How to read benchmark numbers in a pitch deck or contract

This is the section to forward to whoever is signing the cheque. Whether the deal is a Mumbai studio quoting an enterprise client in Singapore, or a London consultancy putting a vendor through a procurement gate at a UK insurer, the rules are the same.

1. Treat Verified as a floor, not a forecast. A model below 70% Verified is unlikely to do well on your repo. A model above 80% Verified might. The score does not differentiate well at the top.
2. Always ask for the Pro number. If the vendor cannot produce one, that is information. If the gap is wider than 25 points, that is also information.
3. Specify the harness in the contract. 'The model under SLA is Claude Opus 4.7 at the official Anthropic API endpoint, evaluated with the openhands scaffold version X' is a contractable sentence. 'The model scores 87.6% on SWE-bench' is a marketing sentence.
4. Build a bridge clause. Tie any productivity claim to a custom evaluation on a slice of your real backlog. Twenty closed PRs from the last quarter is enough to embarrass the worst vendors and reward the honest ones.
5. Be honest in your own deck. If you are a UK or Indian builder pitching to international clients, quote the Pro number and explain the gap. Sophisticated buyers reward technical honesty; the unsophisticated ones will be confused either way.

What benchmark would actually predict how a model handles your codebase?

The honest answer is none of the public ones. Verified, Pro, Multilingual and Lite are all useful as triage filters and as signals of frontier movement, but none of them sees your authentication middleware, your bespoke build tooling, the half-deprecated ORM you cannot replace this quarter, or the way your senior engineers actually write tickets. A custom evaluation of twenty real closed pull requests from your last six months — diff redacted, issue text as input — is the single highest-leverage hour an engineering lead can spend in a model selection.

For a related comparison of how scope changes a benchmark conversation entirely, Cursor Composer 2 reports 73.7% on SWE-bench Multilingual, which is a different evaluation set and not directly comparable to a Verified score on a single-language slice. Cross-benchmark comparisons are the second-most common error in vendor decks, after quoting Verified as a productivity prediction.

The takeaway for buyers and Builders

The 23-point gap between Claude Opus 4.7 on Verified and Pro is not a problem to be solved. It is a measurement that finally makes the benchmark conversation honest. If your work involves quoting numbers — in a pitch, in a procurement document, in an SLA — quote the Pro number when you can, explain the gap when you have to, and build a bridge clause to a custom evaluation in every contract that hinges on a productivity claim. Buyers are getting more sophisticated, slowly. Builders who treat them as if they are will win the deals worth winning.

Primary sources for this piece: the official SWE-bench leaderboard at swebench.com and the Verified split; aggregator views at llm-stats.com and benchlm.ai; and the SWE-bench Pro framing and leaderboard at morphllm.com/swe-bench-pro and labs.scale.com/leaderboard/swe_bench_pro_public.