GPU Glossary
/
Performance·2 min read

Compute-bound

A kernel is compute-bound when its speed is limited by the arithmetic bandwidth of the CUDA Cores or Tensor Cores.

A compute-bound kernel is limited by the arithmetic bandwidth of the CUDA Cores or Tensor Cores. Making it faster means doing less arithmetic or using faster arithmetic units, not moving data more efficiently.

In this roofline diagram, kernels underneath the blue line are compute-bound. Adapted from Williams, Waterman, and Patterson (2008).
In this roofline diagram, kernels underneath the blue line are compute-bound. Adapted from Williams, Waterman, and Patterson (2008).

Characteristics#

Compute-bound kernels have high arithmetic intensity: many arithmetic operations per byte of memory loaded or stored. Utilization of the arithmetic pipes is their limiting factor.

Strictly, compute-boundedness is defined only for a single kernel, as part of the roofline model. With some squinting, it generalizes to the many kernels that make up a typical workload.

Examples#

  • Large diffusion model inference is generally compute-bound.
  • Large language model inference is often compute-bound during batch prefill (prompt processing), when each weight can be loaded into shared memory once and reused across many tokens.

Worked example: minimum inter-token latency#

Here is a simple estimate, inspired by kipperrii's Transformer inference arithmetic, of the minimum latency between tokens (inter-token latency, or time per output token) for compute-bound Transformer inference.

  • Model: 500B parameters in 16-bit precision, so 1 TB of weights.
  • Work: about one trillion floating point operations per batch element (one multiply and one accumulate per parameter).
  • GPU: one petaFLOP/s of arithmetic bandwidth for 16-bit matrix math.

Assuming compute-boundedness, the minimum inter-token latency is one millisecond per batch element.

For this GPU to be compute-bound at batch size one, it would need a memory bandwidth of 1 PB/s, enough to load all 1 TB of weights in one millisecond.

Key idea Real memory bandwidths are in the TB/s range, so batches of hundreds of inputs are needed to provide enough arithmetic intensity for execution to be compute-bound.

For more on LLM inference, see Modal's LLM Engineer's Almanac.