Reading a Humanity's Last Exam score

Humanity's Last Exam has a dramatic name, so a score on it is easy to over-read.

HLE is a multi-modal benchmark of closed-ended academic questions, built to show where current models still fall short. Closed-ended means each question has an answer that can be checked, so grading can be automatic.

It shows up on model cards. DeepSeek's V4.1-Flash-1017 card lists 42.6 on HLE without tools, next to 56.3 for Opus 5 and 44.5 for GPT-5.6 Sol in the same table. Those are the labs' own numbers, not an independent run.

Three things to check before you compare two HLE numbers:

1. 1With tools or without? A result that allows search and code execution isn't comparable to one that doesn't. The card above says 'without tools', so read that column against that column.
2. 2Who ran it? A number a lab reports for its own model, with its own prompt, isn't the same as a third-party evaluation.
3. 3Which version of the dataset? A revised release, HLE-Verified, came out in February 2026, so check that two scores used the same one.

What it tells you: how a model does on hard, expert-level questions with a single checkable answer.

What it doesn't: whether it writes good code for you, or handles a long messy task. A test of expert questions and a test of useful work are different tests.

Which benchmark do you trust most when it's about your actual work?