Arithmetic Intensity

Arithmetic intensity is the ratio of arithmetic operations to memory operations in a kernel. A kernel with high arithmetic intensity performs many arithmetic operations per byte it loads.

Why it matters#

Modern GPUs have far more arithmetic bandwidth than memory bandwidth, so the most efficient kernels have high arithmetic intensity. When elevating a memory bottleneck, you can often shift work from the memory subsystem to the compute subsystem: less memory traffic, more load on the arithmetic units.

Two examples:

• Compression. Compressing data in global memory means fewer bytes to transfer, at the cost of extra decompression work on the compute units. If you were memory-bound, this can improve performance. It also raises FLOPs per byte moved, so arithmetic intensity goes up.
• Gradient checkpointing. The backpropagation algorithm creates long-lived intermediates (activations) that generally must be stored in global memory during the forward pass and read back during the backward pass. Sometimes it is faster to store only some of them and recompute the rest (gradient checkpointing), which increases arithmetic intensity.

How it scales with problem size#

Different algorithms have different operational and memory complexities, so their arithmetic intensity scales differently. An algorithm with O(1) operations and O(N) memory has O(1/N) arithmetic intensity scaling; one with O(N) operations and O(1) memory has O(N) scaling.

Kernel	FLOPs	Bytes moved	Arithmetic intensity	Scaling
SAXPY y = ax + y	2N	12N	1/6	O(1)
Single-precision real FFT	5/2 N log(N)	16N	5/32 log(N)	O(log(N))
SGEMM C = A @ B + C	2N³	16N²	N/8	O(N)

Key idea Matrix multiplication is O(N³) in operations and O(N²) in memory, so its arithmetic intensity grows linearly, O(N). That makes it easy to map onto arithmetic-intensity-oriented hardware, and is a key secret behind the success of matrix-multiplication-based machine learning, like neural networks, over the past few decades.

See the roofline model article for why hardware has evolved this way. For arithmetic intensity applied to Bahdanau attention as used in Transformers, see this paper by Zadouri, Strauss, and Dao.

Ridge points of recent GPUs#

The minimum arithmetic intensity needed to be compute-bound, i.e. past the roofline's ridge point, is a fixed property of a system, so it only needs to be derived once. The table lists ridge points for recent NVIDIA data center GPUs. The highest ridge point has risen from the Ampere to Hopper to Blackwell Streaming Multiprocessor architectures.

System (compute / memory)	Arithmetic bandwidth (TFLOPs/s)	Memory bandwidth (TB/s)	Ridge point (FLOPs/byte)
A100 80GB SXM BF16 TC / HBM2e	312	2	156
H100 SXM BF16 TC / HBM3	989	3.35	295
B200 BF16 TC / HBM3e	2250	8	281
H100 SXM FP8 TC / HBM3	1979	3.35	592
B200 FP8 TC / HBM3e	4500	8	562
B200 FP4 TC / HBM3e	9000	8	1125

Related#

• Roofline model: plots performance against arithmetic intensity
• Compute-bound and memory-bound: the regimes on either side of the ridge point
• Arithmetic bandwidth: the numerator of the ridge point