GPU Glossary
/
Performance·1 min read

Overhead

Time during which the GPU does no useful work because it is waiting to receive work.

Overhead latency is time spent doing no useful work. When a kernel is bottlenecked on compute or memory, the GPU is at least working as fast as it can; during overhead, it is waiting to be given work.

Host overhead#

Overhead often comes from CPU-side bottlenecks that keep the GPU from receiving work fast enough:

  • CUDA API calls add on the order of 10 μs per kernel launch.
  • Frameworks like PyTorch or TensorFlow spend time deciding which kernel to launch, which can take many microseconds.

This is often called "host overhead", though the term isn't fully standardized.

Key idea CUDA Graphs, which bundle many device-side kernels into a single host-side launch, are a common fix for host overhead.

For more, see the CUDA Techniques to Maximize Concurrency and System Utilization talk at GTC 2025.

Memory and communications overhead#

"Memory overhead" or "communications overhead" is latency spent moving data between CPU and GPU or between GPUs. When communication bandwidth is the limiting factor, though, it's often better to think of it as memory-boundedness where the "memory" is distributed across machines.