SWE-bench: can it fix the bug?
SWE-bench was published in 2023 by researchers at Princeton. It is built from 2,294 real problems, each one a GitHub issue and the change that fixed it, taken from 12 popular Python projects.
Each task works like this:
- The AI system gets the project as it was when the issue was filed, and the issue text.
- It writes a change.
- The project’s own tests are run. The tests that failed because of the bug must now pass, and the tests that already passed must still pass.
SWE-bench Verified is a 500-task subset in which people checked each task: that the issue is clear, the tests are correct, and the problem is solvable. It is the version most often quoted.
SWT-Bench: can it prove the bug?
SWT-Bench comes from researchers at ETH Zurich and is built from the same kind of real issues. It asks a different question.
- The AI system gets the project and the issue text.
- It writes a test.
- The test is run twice: on the code before the real fix, where it must fail, and on the code after the real fix, where it must pass.
A test that does both has reproduced the bug. The Verified set has 433 issues.
Why the second one matters
Fixing a bug and proving a bug are different skills, and the second supports the first.
A fix with no test is a claim. A fix with a test that failed before and passes after is evidence. If a system can reliably write that test, every fix it proposes can be checked by a machine, and by you.
That is why Bugstack, which does autonomous software maintenance, writes the test first. Its tester is first on the SWT-Bench Verified leaderboard: it reproduced 91.5% of the 433 issues, as scored by the benchmark’s maintainers.
The two side by side
| SWE-bench | SWT-Bench | |
|---|---|---|
| Question | Can it fix the bug? | Can it write a test that proves the bug? |
| What the system produces | A code change | A test |
| How it is scored | The project’s tests pass after the change | The test fails before the real fix and passes after |
| Source | Real issues from 12 Python projects | Real issues from Python projects |
| Verified set | 500 tasks | 433 tasks |
What neither one tells you
- How a system does on your code. Both use well-known open-source Python projects. Your codebase is different.
- Anything about other languages. Both are Python.
- Whether the system finds problems on its own. In both, the issue is handed to it.
- What it costs or how long it takes.
A benchmark score is a reason to try something on your own repository, and the trial is what tells you.
Questions
Is a higher SWE-bench score always better?
It means more tasks solved on that set. It does not account for cost or speed. Read who scored it and whether the work is public.
Why does Bugstack quote SWT-Bench and not SWE-bench?
Because SWT-Bench scores the step Bugstack depends on most: reproducing a bug with a failing test before any fix is written. The fixer has its own exam, which we publish alongside.
Can I check Bugstack’s result?
Yes. The leaderboard shows the maintainers’ own scoring, and our predictions, evaluation reports and model traces are public on GitHub.