Memory Bandwidth
The maximum rate, in bytes per second, at which data can move between levels of the memory hierarchy.
Memory bandwidth is the maximum rate at which data can be transferred between levels of the memory hierarchy. It is the theoretical maximum throughput for moving data, in bytes per second, and it sets the slope of the "memory roof" in a roofline model of the hardware.
Which bandwidth matters#
A complete system has many memory bandwidths, one between each pair of adjacent levels of the memory hierarchy.
The most important is the bandwidth between GPU RAM and the register files of the Streaming Multiprocessors (SMs). The working sets of most kernels fit only in GPU RAM, not anywhere higher in the hierarchy, so this is the bandwidth primarily used in roofline modeling of GPU kernel performance.
Typical numbers#
Contemporary GPUs have memory bandwidths measured in terabytes per second. B200 GPUs, for example, have a (bidirectional) bandwidth of 8 TB/s to their HBM3e memory.
Key idea Memory bandwidth is much lower than the arithmetic bandwidth of a GPU's Tensor Cores, which pushes up the arithmetic intensity at the roofline's ridge point.
Representative numbers for NVIDIA data center GPUs from the Ampere through Blackwell Streaming Multiprocessor architectures:
| System (compute / memory) | Arithmetic bandwidth (TFLOPs/s) | Memory bandwidth (TB/s) | Ridge point (FLOPs/byte) |
|---|---|---|---|
| A100 80GB SXM BF16 TC / HBM2e | 312 | 2 | 156 |
| H100 SXM BF16 TC / HBM3 | 989 | 3.35 | 295 |
| B200 BF16 TC / HBM3e | 2250 | 8 | 281 |
| H100 SXM FP8 TC / HBM3 | 1979 | 3.35 | 592 |
| B200 FP8 TC / HBM3e | 4500 | 8 | 562 |
| B200 FP4 TC / HBM3e | 9000 | 8 | 1125 |
Related#
- Memory-bound: kernels limited by this bandwidth
- Arithmetic bandwidth: the compute-side counterpart
- Roofline model: where memory bandwidth forms the slanted roof
- Memory hierarchy: the levels data moves between