SWE-bench Pro Explained
What SWE-bench Pro measures, how it differs from SWE-bench Verified, why contamination resistance matters, and which models lead agentic coding today.
SWE-bench Pro is the harder successor to SWE-bench Verified, the benchmark that measures whether an AI model can resolve real GitHub issues end to end. While SWE-bench Verified made agentic coding famous, its public tasks became a victim of the field's success: frontier labs could tune against them, and the score gap between "great at benchmarks" and "great at your codebase" kept growing. SWE-bench Pro exists to close that gap with harder tasks and contamination resistance.
What SWE-bench Pro Actually Tests
Like the original, each task gives the model a repository and an issue description, and the model must produce a patch that passes the repository's hidden tests. SWE-bench Pro raises the difficulty in three ways:
- Larger, messier repositories. Tasks draw on bigger, more realistic codebases - including commercial and enterprise code - where the fix requires understanding code far from the issue's symptoms.
- Multi-file, multi-step changes. Many tasks require coordinated edits across several files, not the single-function patches that dominated the original benchmark.
- Contamination resistance. Held-out private tasks mean a model cannot score well by having memorized public test cases during training, which makes cross-lab comparisons more honest over time.
SWE-bench Verified vs SWE-bench Pro
| SWE-bench Verified | SWE-bench Pro |
|---|---|
| Human-validated subset of public GitHub issues | Harder split with private holdout tasks |
| Mostly single-repo, well-scoped issues | Commercial-grade repos, multi-file fixes |
| Risk of training-data contamination | Designed to resist memorization |
| Scores in the 30-50% range defined the 2024 frontier | Substantially lower scores at the same capability level |
The practical rule: a 49% SWE-bench Verified score (Claude 3.5 Sonnet's famous October 2024 result - see our historical breakdown) and a 30% SWE-bench Pro score are not comparable numbers. Within one benchmark version, comparisons are meaningful; across versions, they are not.
Who Leads Agentic Coding Right Now
The table below is from the RankLLMs dataset (last verified September 2026): SWE-bench Verified accuracy and Terminal Bench performance for the current agentic coding leaders, with blended per-million-token API pricing. Full rankings and pillars are on the leaderboard.
| Model | Provider | SWE-bench Verified | Terminal Bench | API $/1M |
|---|---|---|---|---|
| Claude Fable 5 | Anthropic | 84.5 | 88.0 | $14.44 |
| Claude Opus 5 | Anthropic | 81.8 | 85.4 | $7.22 |
| GPT-5.6 Sol | OpenAI | 81.2 | 88.8 | $7.78 |
| DeepSeek-V4-Pro-0813 | DeepSeek | 81.0 | 80.3 | $1.74 / $3.48 |
| Kimi K3 | Moonshot AI | 80.5 | 85.7 | $4.33 |
| Qwen3.8 Max | Alibaba Qwen | 79.4 | 86.6 | $2.02 |
| Gemini 3.7 Flash | 78.6 | 85.8 | $0.75 / $3.75 | |
| GLM-5.3 | Z.ai | 77.8 | 88.3 | $1.73 |
Scores per the RankLLMs composite dataset; see our methodology for sources and verification dates.
How to Use SWE-bench Pro Scores When Choosing a Model
- Compare within the same harness. Agentic scores depend on scaffolding and step budgets; only compare runs with published evaluation details.
- Weight cost per solved task, not per token. A cheaper model that retries twice can cost more than an expensive one that lands the fix first - our best coding LLM guide does this math.
- Prefer contamination-resistant results for trend claims. For "is the new release actually better," Pro-style held-out results are the ones to trust.