The vocabulary is most of a small model

In a small language model, the biggest part isn't the transformer. It's the vocabulary.

A new paper puts numbers on it. In Gemma 3 270M the embedding table holds 168M of the 268M parameters, about 63%. Compressing the rest of the model doesn't help much, because the table usually stays at 16-bit.

The authors replaced the learned table with a frozen 16 MB table of 512-bit codes, derived once from a pretrained embedding. Then they trained the same model from scratch on 2.05B tokens, with and without the trick.

What they report:

1. 1The learned table buys 0.20 nats of validation loss for 2.6x the trainable parameters.
2. 2On LAMBADA, HellaSwag and PIQA the frozen-table model is better or tied. Only ARC-Easy favors the learned table.
3. 3With a ternary body, the whole model is a 46 MB file that generates 615 to 740 tokens a second on a laptop CPU, in 62 MB of memory.
4. 4It accepts 10,000 new words after training for a 3% cost, where the learned table collapses by 43%.

It's one study at small scale, and the weights are for non-commercial research only. But it's a good reminder to ask what each part of a model is buying you.

If you ship small or on-device models, where do your megabytes go?