GPU Glossary
/
Performance·2 min read

Memory-bound

A kernel is memory-bound when its speed is limited by the GPU's memory bandwidth rather than its arithmetic throughput.

A memory-bound kernel is limited by the memory bandwidth of the GPU. It spends its time waiting for data to move, not doing arithmetic.

Roofline diagrams like this one help identify whether a program is bottlenecked by compute, memory bandwidth, or something else. Adapted from Williams, Waterman, and Patterson (2008).
Roofline diagrams like this one help identify whether a program is bottlenecked by compute, memory bandwidth, or something else. Adapted from Williams, Waterman, and Patterson (2008).

Which bandwidth?#

Specifically, memory-bound kernels are limited by the bandwidth between GPU RAM and the local cache of the Streaming Multiprocessors. That's because the problems that matter for GPU performance generally have working sets much larger than any higher level of the memory hierarchy.

Characteristics#

Memory-bound kernels have low arithmetic intensity (few operations per byte moved) relative to the ridge point of their roofline model.

Strictly, memory-boundedness is defined only for a single kernel, as part of the roofline model. With some squinting, it generalizes to the many kernels that make up a typical workload.

Example: LLM decoding#

Large language model inference is often memory-bound during decode (output generation), when the weights must be loaded once per forward pass. Unless multi-token prediction or speculative decoding is used, that's once per output token, which makes the minimum inter-token latency (time per output token) easy to calculate.

  • Model: 500B parameters in 16-bit precision, so 1 TB of weights.
  • GPU: a single GPU with 10 TB/s of memory bandwidth.
  • Result: the weights can be loaded once every 100 ms, a lower bound on inter-token latency.

Key idea Batching inputs together increases the floating point operations done per parameter loaded (the arithmetic intensity) linearly, in principle up to the point of compute-boundedness, without adding latency. Throughput therefore improves linearly with batch size.

For more on LLM inference, see Modal's LLM Engineer's Almanac.