SWE-bench
In short
SWE-bench is a benchmark that tests whether an AI model can fix real bugs in real software. Each task is a genuine GitHub issue, and the fix only counts if the project's own tests pass afterwards.
- Also called
- SWE-bench Verified, SWE-bench Pro, Software engineering benchmark
- Topic
- Testing and benchmarks
- Level
- Some background helps
- First described
- Introduced by Carlos E. Jimenez, John Yang and colleagues in October 2023, in "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?"1.
SWE-bench was released in October 2023 by researchers at Princeton and the University of Chicago1. It has 2,294 problems drawn from real GitHub issues and the pull requests that fixed them, across 12 popular Python projects1.
The model gets the code and the issue text, and must edit the code to fix it. It never sees the tests. Its change passes only if tests that failed before now pass, and existing tests still pass2.
At launch, the best model, Claude 2, solved just 1.96% of issues1. In August 2024 OpenAI released SWE-bench Verified, a subset of 500 problems checked by software engineers, after finding some original tasks were impossible or unfairly tested2. Verified became the standard coding score in model launches.
It has since run into trouble. In February 2026 OpenAI said it had stopped reporting SWE-bench Verified. It audited 138 hard problems and found 59.4% had flawed tests or descriptions, and every frontier model it tested could reproduce some original fixes from memory, a sign they had seen them in training2. OpenAI now recommends SWE-bench Pro2.
SWE-bench Pro, from Scale AI, has 1,865 problems from 41 repositories, with some kept private or drawn from commercial code to resist leakage3. On Scale's public leaderboard in October 2026, the top listed score was 61.5%4.
Example
When an AI coding tool claims it can "fix bugs on its own", SWE-bench-style tests are the usual evidence: did the model's change make a real project's failing tests pass2?
Why it matters
Coding is one of the most commercially important uses of AI, and SWE-bench scores are the main public evidence of progress. Its contamination problems show why a high score can overstate real ability.
Questions people ask
What is the difference between SWE-bench and SWE-bench Verified?
SWE-bench is the original set of 2,294 problems1. Verified is a 500-problem subset that OpenAI had human engineers review in 2024 to remove unfair or impossible tasks2.
Why did OpenAI stop using SWE-bench Verified?
In February 2026 it said many remaining tasks had flawed tests, and that frontier models had seen some problems and solutions during training, so score gains no longer reflected real coding ability2.
What is SWE-bench Pro?
A harder successor from Scale AI with 1,865 problems from 41 repositories, partly held out or drawn from commercial code to reduce leakage into training data3.
Related terms
Sources
- Jimenez et al. (2023), SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, arXiv:2310.06770
- OpenAI (February 2026), Why SWE-bench Verified no longer measures frontier coding capabilities
- Deng et al. (2025), SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?, arXiv:2509.16941
- Scale AI, SWE-Bench Pro leaderboard (viewed 11 October 2026)