Freemium: Memory Bottleneck Broken by Tiling

Hello!
Welcome to today’s edition of Business Analytics Review!

Standard self-attention in transformers looks compute-heavy on paper, yet on real GPUs it starves for data. An A100 offers hundreds of teraflops but only about 2 TB/s of high-bandwidth memory throughput. When sequence length hits a few thousand tokens the full N-by-N attention matrix forces repeated round-trips through that slow HBM, so the arithmetic units sit idle most of the time. The result is quadratic memory growth, training that crawls on long contexts, and inference bills that scale with every extra token of context window. Approximate methods tried to escape the quadratic trap by sacrificing exactness; they rarely delivered wall-clock wins because they ignored the real bottleneck: data movement, not FLOPs.

FlashAttention flipped the diagnosis. By treating the GPU memory hierarchy as a first-class constraint and redesigning the algorithm around it, the same exact attention can run several times faster while using memory that grows only linearly with sequence length. The practical payoff is longer contexts, cheaper training runs, and models that finally stay inside a single GPU’s memory budget.