Peak Rate
The theoretical maximum rate at which a hardware system can complete work, computed from its hardware specifications.
Peak rate is the theoretical maximum rate at which a hardware system can complete work. For a GPU, it is the performance you would get if every execution unit ran at full capacity with perfect efficiency.
It assumes no resource constraint gets in the way: no limits from registers, memory bandwidth, synchronization barriers or anything else that could become a bottleneck.
Why it matters#
Peak rate is the yardstick for all achieved performance:
- It sets the compute-bound "roof" in a roofline analysis.
- It is the denominator of the utilization fraction in pipe utilization metrics.
- It is the ultimate arbiter of GPU utilization.
NVIDIA engineers often call it the "speed of light": the limit on program speed imposed by physics.
Computing it: H100 FP32#
Peak rate comes straight from each GPU architecture's fixed hardware specs. Take an NVIDIA H100 GPU:
- 132 SMs × 128 FP32 cores = 16,896 cores.
- Each core can issue one single-precision fused multiply-add (
FMA) per clock, and an FMA counts as 2 floating point operations. That's 16,896 × 2 = 33,792 FLOPs per clock (see instructions per clock). - The compute clock runs at up to 1980 MHz (million clocks per second) when using the FP32 cores.
So the peak rate is 33,792 × 1,980 million ≈ 66,908 billion FLOPS, or 66.9 TFLOPS.
Key idea That number exactly matches the Peak FP32 TFLOPS (non-Tensor) figure in NVIDIA's H100 whitepaper. Advertised peaks are just hardware arithmetic.
Related#
- Roofline model — uses peak rate as its compute roof.
- Pipe utilization — reports achieved work as a fraction of peak.
- Arithmetic bandwidth — the peak rate for arithmetic.