GPU Glossary
/
Performance·3 min read

Roofline Model

A simple visual performance model that shows whether a program is limited by memory bandwidth or by arithmetic bandwidth.

The roofline model is a simplified, visual model of performance. It lets you quickly see whether a program is bound by memory bandwidth or by arithmetic bandwidth.

Kernels left of the ridge point are limited by memory bandwidth; kernels to the right are limited by arithmetic bandwidth. Adapted from Williams, Waterman, and Patterson (2008), which introduced the model.
Kernels left of the ridge point are limited by memory bandwidth; kernels to the right are limited by arithmetic bandwidth. Adapted from Williams, Waterman, and Patterson (2008), which introduced the model.

The two roofs#

Two hardware-derived "roofs" put a ceiling on achievable performance:

  • Compute roof: the peak rate of the target hardware (CUDA Cores or Tensor Cores), i.e. its arithmetic bandwidth.
  • Memory roof: the peak memory throughput of the target hardware, i.e. its memory bandwidth.

Both are drawn on a plane with arithmetic intensity (operations per byte) on the x-axis and performance (operations per second) on the y-axis.

  • The compute roof is a horizontal line at height equal to the arithmetic bandwidth.
  • The memory roof is a slanted line whose slope is the memory bandwidth. Slope is rise over run, so its units are (operations/second) ÷ (operations/byte) = bytes per second.

Reading the diagram#

A kernel's x-coordinate tells you immediately whether it is fundamentally compute-bound (beneath the flat roof) or memory-bound (beneath the slanted roof). In practice kernels rarely touch either roof, because of overhead.

The ridge point is where the slanted and flat roofs meet. Its x-coordinate is the minimum arithmetic intensity needed to escape the memory bottleneck.

Key idea The further left the ridge point, the easier it is to reach peak performance. Memory has scaled poorly relative to compute, so ridge points have generally moved right over time.

The roofs need to be derived only once per subsystem. They do vary by subsystem, not just by system: Tensor Cores have more FLOPS than CUDA Cores, for example.

NVIDIA's Nsight Compute tool for kernel performance engineering performs roofline analysis automatically for profiled kernels.

What the model leaves out#

The roofline model is deceptively simple. System latencies, for instance, appear nowhere in it; only bandwidths and throughputs do. It is simple because it is highly opinionated, and understanding those opinions and the reasoning behind them is key to applying it well.

Origins#

Samuel Williams, Andrew Waterman, and David Patterson introduced the roofline model in a 2008 paper, in response to several hardware scaling trends that have shaped system architecture before and since:

  • Latency lags bandwidth. As Patterson observed in a famous 2004 paper, across compute, memory, and storage, a linear improvement in latency has historically come with a quadratic improvement in bandwidth. This suggested future systems would be throughput-oriented, like GPUs.
  • The memory wall. Compute subsystems (like processor cores) have long scaled performance much faster than memory subsystems like caches and DRAM. Wulf and McKee popularized this as the "memory wall" in 1994.
  • The end of Dennard scaling. In the early 2000s, Dennard scaling (rising clock speed at equal power) ended, mainly because the fixed leakage current of transistors created power draw and heat dissipation problems. Rising clock speeds had favored general-purpose, latency-oriented CPUs over special-purpose hardware. Moore's Law (rising transistor count per chip) did not slow down at the same time. With abundant transistors but scarce power, the answer was hardware specialization: splitting computers into components specialized for distinct tasks. A well-documented example is the Pixel Visual Core image co-processor, covered in chapter 7 of the sixth edition of Hennessy and Patterson's Computer Architecture.

Together, these trends correctly suggested to the authors that future systems would be throughput-oriented, and that memory bandwidth would be the primary performance bottleneck among the bandwidths involved. Applications seeking peak performance would need high operational intensity for the hardware's specialized operations. For GPUs, that means high arithmetic intensity for Tensor Cores: very large matrix multiplications.