GPQA Diamond in 198 questions

GPQA Diamond is on almost every frontier model card. What is it, and why do its scores need a careful read?

GPQA is a set of hard multiple-choice questions in biology, physics and chemistry, written by PhD-level experts. They're meant to be Google-proof: skilled non-experts with web access still struggled. The Diamond subset keeps the questions where experts agreed on the answer and most non-experts got it wrong. It has 198 questions.

That makes it a decent test of deep scientific reasoning, since a quick search won't solve it.

A few reasons to read the scores carefully. With 198 questions, a single question is about half a percentage point, so small gaps between models are mostly noise. Frontier scores have climbed high enough that the test separates models less than it used to. And multiple choice rewards test-taking skill as well as scientific ability.

I'd treat it as one signal for hard science-style reasoning, never the deciding vote. If your work isn't graduate-level science, a model's score here tells you very little about your use case.

Which benchmark do you trust when you pick a model for work?