GPU Glossary
/
Performance·2 min read

Latency Hiding

Masking the latency of slow operations by running many of them concurrently, so execution units stay busy while individual operations wait.

Latency hiding masks long-latency operations by running many of them concurrently. It is how GPUs keep throughput high despite long instruction latencies.

How GPUs hide latency#

Performant GPU programs interleave the execution of many threads. When one warp stalls on a slow memory operation, the GPU immediately switches to issuing instructions from another eligible warp.

This keeps all the execution units busy at once. While one warp runs matrix multiplication on the Tensor Cores, another might do arithmetic on the CUDA Cores (say, quantizing or dequantizing matrix multiplicands), and a third could be fetching data through the load/store units.

A worked example#

Consider this simple instruction sequence in Streaming Assembler:

LDG.E.SYS R1, [R0]        // memory load, 400 cycles
IMUL R2, R1, 0xBEEF       // integer multiply, 6 cycles
IADD R4, R2, 0xAFFE       // integer add, 4 cycles
IMUL R6, R4, 0x1337       // integer multiply, 6 cycles

Run sequentially, it takes 416 cycles. Assume we can issue one instruction per cycle. By Little's Law, running 416 threads concurrently lets us still finish the sequence once per cycle on average, hiding the memory latency from whatever consumes R6.

Threads aren't the unit of instruction issue, though; warps are. Each warp has 32 threads, so this fragment needs 416 ÷ 32 = 13 warps.

Key idea When latency is hidden successfully, the GPU's scheduler keeps enough warps in flight and switches between them whenever one stalls, so the execution units never sit idle waiting for slow operations.

For a deep dive into latency hiding on pre-Tensor Core GPUs, see Vasily Volkov's PhD thesis.