Any tool can write a patch. The hard part of fixing a bug is knowing the patch is right: that it fixes the bug you have, not a nearby one, and that it does not break something else. Engineers know this as the difference between "I changed the code" and "I proved the change". A good engineer reproduces the bug first, writes a test that fails because of it, fixes the code, and watches the test go green.
bugstack does exactly that, every time, on its own. This post walks through how: what runs, where it runs, what counts as proof, and what never does.
Why "the CI passed" is not proof
The usual answer to "how do you know the AI's fix works?" is "the tests pass". That answer is weaker than it sounds.
- Green can mean nothing ran. If no existing test touches the code path that crashed, the suite passes before the fix and after it. It tells you the fix broke nothing the tests know about. It says nothing about whether the bug is gone.
- CI can be silent. A pipeline that timed out, was never configured for that repository, or skipped a job can read as success if you only look for "not red". We learned this ourselves: an earlier version of our own pipeline once recorded a timed-out CI run as proof.
- CI can be flaky. A test that fails one run in ten turns a single green run into a coin toss.
Proof needs two observations, not one: the bug happening on the old code, and not happening on the new code, measured by the same test. That is a red-to-green reproduction. bugstack collects both, in an environment it controls, before you ever see the pull request.
The rule: a test that fails for the bug's own reason
A reproduction test only proves something if it fails because of the bug. A test that fails on an import mistake, a typo, or a missing fixture is red, but the red is about the test, not your code. Fix the bug and that test still fails; or worse, fix the typo along the way and it "passes", and the patch gets credit it never earned.
So bugstack is strict about what counts:
- The failure must match the bug. The test has to fail with the incident's own error: the same exception type, and a stable fragment of its message when there is one. A
KeyErrorin production must be aKeyErrorin the test, not anImportError. - Generic errors need more. A bare
Error,Exceptionor Gopanicsays too little on its own, so the test's failure must also carry a distinctive piece of the original message. If the error cannot tell a right failure from a wrong one, bugstack does not pretend: nothing is marked as proven. - A bug described in words gets a marker. When there is no stack trace, because the bug came from a GitHub issue or a support ticket, the failing assertion must print a marker that bugstack checks for in the test output. A failure anywhere else does not count, and neither does a test whose own source happens to contain the marker.
- Unknown is never proof. A run that timed out, could not install, or never started proves nothing. It is recorded as unknown, with the reason, and never as a pass.
Step 1: can we run your tests?
When you connect a repository, bugstack checks that it can run your tests at all. It installs your dependencies and runs your whole suite on the default branch with the network cut. If every test passes, the repository is sandbox-ready, and the project settings page says so. If a test already fails on your default branch, bugstack tells you which one, because a suite that is already red cannot tell a fixed bug from a broken one.
This check is also what makes later proof cheap to trust: bugstack knows your suite was green on its own, with no network, before any fix touched it.
Step 2: an isolated copy, with the network cut
Every fix gets its own sandbox: a fresh, isolated copy of your repository. It runs on infrastructure bugstack operates, never on the servers that run bugstack's own application. The rules are simple, and each one is enforced and tested:
- An empty environment. The sandbox starts with nothing inherited: none of bugstack's keys, tokens or host variables, and none of your secrets either. Your tests run the way a new contributor's would on a clean machine.
- Network for installing, then none. While dependencies install, the only places the sandbox can reach are your package registries, through a proxy that allows those hosts and refuses everything else. Before the first test runs, the network is cut completely: no outbound connections, no DNS, only loopback. Nothing the code does during a test can reach the outside world.
- No way out of the box. Root inside the sandbox is not root on the host. The sandbox cannot mount filesystems, switch users or change its root, and it cannot gain privileges it was not given. Two sandboxes cannot see each other's files or processes.
- Hard limits. Time, memory, processes and disk are capped, so a runaway test cannot take anything else down.
- Gone when done. When the fix is finished, the sandbox is destroyed. What is kept is the result and its logs, with secrets scrubbed.
Step 3: the Tester writes the failing test
The Tester is the agent whose only job is the reproduction. It does not write tests blind from a stack trace. It reads the source the error comes from, finds your nearest existing tests, follows their layout and conventions, and, where it makes sense, extends an existing test file instead of adding a new one.
Then it runs the test on your current code, inside the sandbox. The test must fail, and fail for the bug's own reason, by the rules above. If it passes, or fails for some other reason, the Tester rewrites it, this time with what the sandbox printed, the relevant source and your own tests in front of it. It gets up to three attempts. A test that never fails for the right reason is not used as proof.
Step 4: the Fixer, in the same sandbox
With a test that reproduces the bug, the Fixer writes the change. It runs in the same sandbox, so it pays for the install once and works against exactly the code the test just failed on.
Now the reproduction test must pass, and every other test in your suite must stay green. If either does not hold, no pull request opens. The Fixer tries again, with the sandbox's output as feedback, and every check that applies to a first attempt applies to the retry. Every attempt starts from the original commit: whatever an earlier attempt wrote is put back first, so attempts never stack on each other.
The whole fix runs on a budget: up to three tests, up to three fixes, and thirty minutes of wall-clock time, model calls included. A fix that cannot meet the bar inside the budget does not become a pull request on the strength of a guess.
Step 5: proof you can read
When a fix is proven, the pull request says so in plain language: the reproduction test, the error it failed with before the change, and how many of your other tests still pass. The test itself is part of the pull request, so the bug stays fixed: if anyone reintroduces it, your own suite catches it.
The fix page in the dashboard shows the same evidence as a Proof panel: Failed before, Passes after, and the number of other tests still passing, along with how long it took, that the network was cut, and what the test printed before the change. A run that did not finish never reads as a pass there.
Step 6: checked again before it merges
Code moves while a pull request waits. So when your CI goes green on a bugstack pull request, bugstack runs the proof again, against the commit the pull request would merge into: the reproduction test must still fail there with the original error, and pass with the change, with your suite intact and the network cut.
This is what decides merging. On Autopilot, a fix merges only with this proof. On Review, every fix waits for a person anyway, and the proof is right there in the pull request when they look. Your own CI still runs on every pull request, as a second, independent check.
Bugs described in words
Not every bug arrives with a stack trace. Some arrive as a GitHub issue, a report filed in the dashboard, or a customer's support ticket. bugstack handles these the same way: the Tester writes a test from the description and the code it points at, and the sandbox has to show it failing before anything is fixed. If the bug cannot be reproduced, bugstack says so plainly, with what it tried. It never guesses a fix for a bug it could not see.
What the sandbox is not
A sandbox is a faithful copy of your code and your tests, not your production system. Passing there means the fix is proven against the bug as your code expresses it. That is why bugstack keeps watching after a fix ships: the error stopping in production is the final word. And the sandbox runs your tests, not your traffic; it is not a load test or a staging environment.
What it runs today
The sandbox runs Python repositories (pytest), JavaScript and TypeScript (Vitest, Jest and Mocha) and Go (go test). Ruby is next, along with test suites that need a database or other services alongside them.
Frequently Asked Questions
Does the sandbox have access to my secrets?
No. Each sandbox starts from an empty environment. It receives none of bugstack's keys or tokens and none of your secrets, and the network is cut before any test runs.
Does the reproduction test end up in my repository?
Yes. A proven fix's pull request includes the test that reproduced the bug, so your own suite guards against the bug coming back.
What happens if bugstack cannot reproduce the bug?
Nothing is marked as proven. On Autopilot the fix does not merge on its own, and for a bug described in words, bugstack says plainly that it could not reproduce it instead of guessing a fix.
Why run the proof again before merging?
Because the code may have changed while the pull request waited. bugstack proves the fix again against the commit it would merge into, so the proof is about the code that actually ships.
bugstack is an autonomous maintenance department for your software. It watches your app and your repository, proves each fix in an isolated sandbox, and sends it to you as a pull request you control. Sign up with GitHub (a card is required) to start bugstack Pro: $100 a month with $100 of usage credit every month, plus a one-time $100 welcome bonus. Or book a call and we set it up with you. Get started.