Freemium: Draft Models Outrun the Giants

Hello!

Welcome to today’s edition of Business Analytics Review!

Large language models still generate text one token at a time, even when the hardware sitting underneath them could process dozens of candidates in parallel. The result is a stubborn latency tax that grows with every extra parameter. Teams keep buying faster GPUs, only to discover that the sequential bottleneck remains almost unchanged. Speculative decoding flips that equation by letting a cheap, small draft model race ahead and propose multiple tokens while the expensive target model checks them all at once. Output quality stays identical to ordinary autoregressive decoding; wall-clock speed often doubles or triples.

The core insight is simple yet counter-intuitive: most of the compute cost in inference is memory movement, not arithmetic. Once the target model’s weights are loaded, verifying five or ten candidate tokens costs almost the same as verifying one. A well-chosen draft model therefore turns idle silicon cycles into accepted tokens instead of wasted bandwidth.