What LiveCodeBench measures, and where it misleads

Models that ace old coding benchmarks may have seen the answers. LiveCodeBench was built to make that harder.

The problem is contamination. A benchmark made of problems published in 2021 is probably in the training data of a model released in 2025. A high score can mean memory, not skill.

LiveCodeBench's approach is to keep collecting new problems over time. They come with release dates, so you can score a model only on problems published after its training cutoff. That's the whole trick, and it's a good one.

It also measures more than writing code. Beyond generation, the paper covers self-repair (fixing code after seeing an error), code execution (predicting what a program outputs) and test output prediction.

Where it helps: judging raw problem-solving on fresh, self-contained tasks.

Where it stops: these are contest-style problems. They say little about reading a messy codebase, using your framework, or writing a change that fits your team's conventions. That's closer to what SWE-bench measures.

A practical habit: when you read a coding score, ask what date the problems are from, and what date the model's data ends.

When did you last check a benchmark's release date against a model's cutoff?