What a SWE-bench score really measures

A high SWE-bench score doesn't mean a model will fix most of your bugs. The number measures something narrower.

SWE-bench gives a model a real GitHub issue and the repo, and asks for a patch. The patch passes only if the project's own tests pass. SWE-bench Verified is the 500-task subset that human annotators screened for problems. OpenAI released it in August 2024.

A good score tells you the model can read an unfamiliar codebase and make a change that passes real tests. That's a lot.

It doesn't tell you much else. The tasks come from open-source Python projects, so your stack may behave differently. Scores depend heavily on the agent harness around the model, and cost or number of attempts usually isn't in the headline. And a patch can pass the tests and still be wrong, because tests don't cover everything.

I'd use public scores to build a shortlist, then run 10 of your own real issues before choosing.

What's the first thing you check when a new model tops a coding leaderboard?