Performance Bottleneck
The single resource in a system that limits how fast tasks can be completed, and so the target of performance optimization.
A performance bottleneck is whatever limits the rate at which a system completes tasks, just as the neck of a bottle limits how fast liquid can be poured. Finding and removing bottlenecks is what performance optimization is.
The optimization loop#
The textbook approach is:
- Find the bottleneck.
- Elevate it until it is no longer the bottleneck.
- Repeat on the new bottleneck.
This loop is formalized in, for instance, Eliyahu Goldratt's "Theory of Constraints", which helped spread the Toyota approach to manufacturing worldwide and from there into software engineering and operations.
Bottlenecks in GPU kernels#
In a talk for Jane Street, Horace He split the work done by GPU kernels into three categories:
- Compute: floating point operations on CUDA Cores or Tensor Cores.
- Memory: moving data through the memory hierarchy.
- Overhead: everything else.
So GPU kernel bottlenecks fall into three main categories:
| Kind | Limited by | Example |
|---|---|---|
| Compute-bound | Arithmetic bandwidth of compute units | Large matrix-matrix multiplication |
| Memory-bound | Bandwidth of memory subsystems | Large vector-vector multiplication |
| Overhead-bound | Latency | Small array operations |
Roofline model analysis quickly tells you whether a program is bottlenecked by compute (arithmetic bandwidth) or by memory bandwidth.
Other bottlenecks#
Any resource can become a bottleneck. Power ingress and heat egress, for example, can and do hold some GPUs below their theoretical peak. NVIDIA describes a 4% end-to-end speedup from redirecting power from the L2 cache to the Streaming Multiprocessors, and Horace He shows that matrix multiplication performance depends on the input data, via the power demanded by transistor switching.
Key idea Any resource can be the bottleneck, but compute and memory are the most important resources and the most common bottlenecks.
Related#
- Roofline model: visual tool for spotting compute vs. memory bottlenecks
- Compute-bound and memory-bound: the two main bottleneck regimes
- Overhead: the "everything else" category