# How to read an AI agent benchmark

[Learn](/learn) › How agents work

# How to read an AI agent benchmark

A benchmark score for an AI coding agent tells you something only if you know who scored it, what was being tested, and whether you can see the work. Five questions sort the meaningful numbers from the rest. We use our own result as the example, including the parts that count against us.

## The five questions

### 1\. Who did the scoring?

A score a company gave itself is weaker than one given by the people who run the benchmark. Look for a public leaderboard with independent scoring.

Bugstack’s case: we scored our own run at 92.6%. The maintainers scored it at 91.5%. We quote 91.5%.

### 2\. What exactly was tested?

A product has many parts, and a benchmark usually tests one. Find out which.

Bugstack’s case: [SWT-Bench](/learn/what-is-swe-bench) scores the tester, the part that reproduces a bug with a failing test. It does not score the fixer. The fixer has a separate exam.

### 3\. Can you see the work?

A number with nothing behind it cannot be checked. Look for the predictions and the logs.

Bugstack’s case: our predictions, evaluation reports and model traces are public on GitHub, and the maintainers’ evaluation logs are on their server.

### 4\. Is it the best on every measure?

Leaderboards often report more than one number. A system can lead on one and not another.

Bugstack’s case: SWT-Bench reports a success rate and a coverage figure. Bugstack is first on success rate. Another entry has the highest coverage.

### 5\. How close is the test to your situation?

The projects, the language and the kind of task all matter.

Bugstack’s case: the benchmark is 433 issues from 12 open-source Python projects. It says nothing direct about a private TypeScript codebase.

## What a self-run number is worth

Not nothing. A company’s test on its own code can be more relevant to its product than a public benchmark, as long as it says exactly what it did.

Our fixer exam is self-run: bugs we planted in our own code, in a set sealed before we tuned the fixer, graded by a script. Of 44 fixes proposed, 43 held. We say “planted” and “our own code” every time we quote it, because those words change what the number means.

## Signs a number is being oversold

-   No date.
-   No mention of who scored it.
-   “Up to” in front of it.
-   One number for a whole product.
-   No link to anything.

## What to do with a score

Use it to decide what to try. Then try it on your own code. Bugstack does [autonomous software maintenance](/autonomous-software-maintenance), and the test that matters is whether its fixes hold in your repository. That is what the 14-day trial is for.

## Questions

### Are AI agent benchmarks reliable?

The good ones are reliable about what they measure: a fixed set of tasks, scored by running code. They are unreliable as a prediction of how a tool does on your work, which is why a trial matters more.

### What is a good SWE-bench or SWT-Bench score?

It moves quickly, so compare against the current leaderboard and not a fixed bar. Check the date on any score you are shown.

### Why do two scores for the same run differ?

Some tests in real projects are flaky, so two runs of the same grading can differ slightly. That is the reason our own scoring and the maintainers’ scoring are about a point apart.

## Related

-   [What is SWE-bench, and what is SWT-Bench?](/learn/what-is-swe-bench)
-   [Is AI-generated code reliable?](/learn/is-ai-generated-code-reliable)
-   [Can AI fix bugs on its own?](/learn/can-ai-fix-bugs)
-   [What is autonomous software maintenance?](/autonomous-software-maintenance)

## See it work on your own code.

$100 a month

Includes $100 of usage credit every month. Start with a 14-day trial: $50 of credit and a first security audit on us. A card is required.

[Get started](https://dashboard.bugstack.ai/login) [Book 15 minutes with the founder](https://cal.com/mason-bachmann-1atgdq/15min)