How to read a score with an @k on it

'Resolved@11: 70.7%' and 'Resolved@1: 41.7%' come from the same model. A score with an @k on the end needs reading carefully.

The k is the number of attempts. Resolved@1 means the agent got it on the first try. Resolved@11 means at least one of eleven tries worked. In the FC-SWE paper, a 4B model scores 41.7% on SWE-bench Verified at one attempt, 52.8% at two, and 70.7% at eleven.

All three are true. They answer different questions.

1. 1@1 is what a user feels if the agent has one shot.
2. 2@k tells you how good the best of k tries is, which matters if you can verify the result and retry cheaply, for example with a test suite.
3. 3The gap between them tells you how much a verifier is worth.

Three questions to ask whenever you see one:

1. 1Who or what chooses the winning attempt? Real tests, or a human who already knows the answer?
2. 2What does each extra attempt cost in time and money?
3. 3Is the comparison fair? A model with 11 tries against another with 1 isn't a comparison.

For your own agent, pick the k that matches how you'll run it, and report that one.

When your team quotes a benchmark score, do you know which k it is?