GPU Glossary
/
Performance·2 min read

Arithmetic Bandwidth

The peak rate at which a system can perform arithmetic, measured in operations (usually FLOPs) per second.

Arithmetic bandwidth is the peak rate at which a system can perform arithmetic work. It is the theoretical maximum throughput for arithmetic operations per second, and it sets the height of the "compute roof" in a roofline model of the hardware.

Which bandwidth matters#

A complete system has many arithmetic bandwidths, one for each group of hardware units that executes arithmetic.

On many GPUs, the most important has been the floating point bandwidth of the CUDA Cores. GPUs generally offer more bandwidth for floating point than for integer operations. The key idea of the Compute Unified Device Architecture (CUDA) is that the CUDA Cores and supporting systems give GPU applications a unified computing interface, unlike earlier GPU architectures.

Recent GPUs weaken that unity with Tensor Cores. They perform only matrix multiplication, but at much higher arithmetic bandwidth than CUDA Cores.

Key idea A good rule of thumb is a 100:1 ratio between Tensor Core and CUDA Core bandwidth, which makes Tensor Core arithmetic bandwidth the one that matters most for kernels that want maximum performance.

Typical numbers#

Contemporary GPUs have Tensor Core arithmetic bandwidths measured in petaFLOPS: quadrillions of floating point operations per second. B200 GPUs, for example, reach nine PFLOPS on 4-bit floating point matrix multiplications.

Representative numbers for NVIDIA data center GPUs from the Ampere through Blackwell Streaming Multiprocessor architectures:

System (compute / memory) Arithmetic bandwidth (TFLOPs/s) Memory bandwidth (TB/s) Ridge point (FLOPs/byte)
A100 80GB SXM BF16 TC / HBM2e 312 2 156
H100 SXM BF16 TC / HBM3 989 3.35 295
B200 BF16 TC / HBM3e 2250 8 281
H100 SXM FP8 TC / HBM3 1979 3.35 592
B200 FP8 TC / HBM3e 4500 8 562
B200 FP4 TC / HBM3e 9000 8 1125