What ARC measures, and where it misleads

Most benchmarks test what a model already knows. ARC tests how quickly it can learn something it has never seen.

ARC, the Abstraction and Reasoning Corpus, comes from a 2019 paper by François Chollet, 'On the Measure of Intelligence'. The paper argues that intelligence is better measured as skill-acquisition efficiency: how well you pick up new skills from limited experience, not how many skills you've been trained on.

The tasks are small colored-grid puzzles. You see a few examples of an input grid and its output grid, work out the rule, and apply it to a new input. There's no language, no trivia and no library of facts to lean on. Each puzzle has its own rule.

Why it's useful: it's hard to solve by memorizing, which makes it a counterweight to benchmarks that saturate.

Where to be careful: ARC-style puzzles test one narrow kind of reasoning. Excelling at them doesn't mean a model will manage your messy business process. And how a score was reached matters. Whether it took one attempt or thousands, and how much compute it used, changes what the number means.

When you read an ARC result, check the attempts and the compute before the percentage.

Do you think skill-acquisition efficiency is the right way to measure intelligence?