Public benchmark · 1 November 2026

First on SWT-Bench Verified

What SWT-Bench measures

SWT-Bench is a public benchmark from researchers at ETH Zurich (the paper). Each task is a real issue from an open-source Python project, with the repository exactly as it was when the issue was filed. The agent has to add a test to the project’s own test suite that fails on that code and passes once the issue is fixed. The benchmark checks the test against the fix the project’s developers actually shipped.

The Verified set has 433 issues from 12 projects: Django, SymPy, Matplotlib, Sphinx, scikit-learn, Astropy, xarray, pytest, Pylint, Requests, seaborn and Flask.

It reports two scores. Success rate (S) is the share of issues where the test fails before the real fix and passes after it; the leaderboard ranks by it. Coverage change (ΔC) is how much of the code changed by the real fix the test runs.

The leaderboard

SWT-Bench Verified, as listed on swtbench.com on 4 November 2026

‡ Script mode: the test is a standalone script instead of a test in the project’s suite. Every other entry here writes unit tests. PatchTwin+Guard has the highest coverage change on the board. See the full leaderboard

How bugstack ran it

The entry is bugstack’s tester, the step that reproduces a bug in your repository before the fixer changes any code. Claude Sonnet 5.5 writes candidate tests, and Claude Opus 5.5 picks the one to keep. Each issue ran in an isolated sandbox with the project’s own dependencies installed, the same way a bug in your code does.

It was one run over all 433 issues, with about $177 of model spend in total, roughly 41 cents an issue.

We scored the run ourselves with the official harness and got 92.6% (401 of 433). The leaderboard shows the maintainers’ own scoring, 91.5%, and that is the number we quote. Scores move a little between runs of the harness because some tests in these projects are flaky.

Check it yourself

• Our predictions, evaluation reports and full model traces on GitHub
• The maintainers’ evaluation logs on ETH Zurich’s server
• The SWT-Bench Verified leaderboard

What it means for your code

Every bug fix bugstack proposes starts with this step: a test that fails on your current code, for the bug’s own reason. Only when that test exists does the fixer write a change, and the same test has to pass with the fix while the rest of your suite stays green. SWT-Bench shows how often the tester gets that first test right on real issues in large, mature codebases.

SWT-Bench covers Python projects and scores the tester alone. The fixer has its own exam: on planted bugs sealed before we tuned it, 43 of the 44 fixes it proposed held. See how well it works