Agents break speculative decoding. History is why

Agent workloads break speculative decoding, and a new paper says the culprit is the history.

Speculative decoding speeds up inference. A small draft model guesses the next few tokens and the big model checks them in one pass. When the guesses are accepted, you get several tokens for the price of one.

Agents are a bad fit. Their contexts are long and keep changing, with many turns of tool calls. The DLite authors (Shanghai Jiao Tong University) report that existing draft models lose up to 80% of their acceptance length under these workloads, and that keeping the draft model's history cache around adds heavy system overhead.

Their answer is to stop carrying the history. DLite reuses hidden states from the target model and predicts only locally, so the draft model solves a small problem again and again instead of a growing one. A light sequential head restores the dependencies inside each block of tokens.

They report acceptance lengths of 2 to 6 across general, long-context and agentic workloads, and up to 70% better than the previous state of the art, DSpark.

If you run your own inference for agents, this is the layer where cost hides: not the model, the loop around it.

Are you measuring tokens per task, or only tokens per second?