Why the question is hard to answer
“Reliable” depends on what the code was asked to do and who looked at it afterwards. A model asked to build a feature has no way to know whether the feature is what you wanted. A model asked to fix one specific crash has a much easier job: the crash either stops or it does not.
That is why Bugstack, which does autonomous software maintenance, works on problems with a clear pass or fail: a crash, a failing build, a failed deploy, a package with a known vulnerability. Each one comes with its own check.
What we measured
We ran two measurements and published both.
| Measurement | What was tested | Result |
|---|---|---|
| SWT-Bench Verified, a public benchmark | Writing a test that reproduces a real bug: 433 real issues from 12 open-source Python projects | 91.5% reproduced, the top score on the leaderboard, scored by the benchmark’s maintainers |
| Our own exam, sealed before we tuned the fixer | Proposing a fix for a planted bug | 43 of 44 proposed fixes held (8 October 2026) |
| The larger set we tuned on | The same, across TypeScript, Python, Go and Ruby | 245 of 252 proposed fixes held |
“Held” is decided by a script, with no model and no person involved. It applies the proposed fix and triggers the bug again. The fix holds when the error is gone, the response matches the healthy one, and the project’s typecheck and tests pass. A fix that only edits tests never counts.
What those numbers do and do not say
They say that for bugs with a reproducing test, an AI system can write the test and the fix correctly most of the time.
They do not say AI code is reliable in general. Three limits matter:
- The bugs on our exam were planted by us, in our own code. A planted bug is a fair test of the fixer, and it is still not the same as every bug in every codebase.
- The benchmark scores the test-writing step, which is the tester’s job. The fixer has its own exam, above.
- One fix in 44 did not hold. That is why a new repository starts with a person merging every change.
What makes AI-written code safe to ship
The model matters less than what stands between its output and your main branch. Four things do the work:
- A test that fails first. Before any fix is written, a test has to fail on the current code, for the bug’s own reason. A test that fails because of a typo or a bad import does not count.
- The same test passing after. The fix has to turn that test green and leave every other test green.
- Your own CI. The pull request runs through the checks you already have.
- A person, until you decide otherwise. On Bugstack’s Review level, which is where every repository starts, you merge each pull request yourself.
Bugstack proves each bug fix in an isolated sandbox before the pull request opens. When a bug cannot be reproduced, the pull request says so, and you decide what to do with it.
Questions
Does AI-generated code have more bugs than human code?
It depends on what checks it. Code from any author that ships without a test or a review has more bugs than code that goes through both. The useful question is whether a failing test, your CI and a reviewer stand between the code and production.
Can I trust an AI to fix a bug without reading the fix?
Start by reading every one. Bugstack opens each fix as a pull request with the cause, the change and the test, and new repositories start on Review, where you merge. Autopilot is there for when you have seen enough fixes hold to let proven ones merge on their own.
How do I know a benchmark result is real?
Check who scored it and whether you can see the work. Bugstack’s SWT-Bench score is the maintainers’ own scoring, and our predictions, evaluation reports and model traces are public.