Performance
The concepts and metrics you need to understand why a GPU program is fast or slow.
People use GPUs because their programs are too slow on ordinary hardware. That changes what "done" means.
In most software, like a database or a web server, correctness comes first. If the program loses data or returns wrong answers, it has failed, and performance is often an afterthought.
On GPUs, correctness is usually fuzzier. Outputs are often only "correct" to some number of significant bits, or only for well-behaved inputs. Correctness is still required, but it is not enough. If the program isn't faster per second, per dollar or per watt than the alternative, it has failed: GPUs are too hard to program and too expensive to run to justify anything less. NVIDIA sums this up as "performance is the product".
This section defines the terms you need to reason about GPU performance. Roughly, it should cover every term you meet when you debug a kernel in Nsight Compute.
Key idea Every slow kernel is limited by something: compute, memory bandwidth, latency or overhead. Start with performance bottlenecks and the roofline model to find out which.
- Performance BottleneckThe single resource in a system that limits how fast tasks can be completed, and so the target of performance optimization.
- Roofline ModelA simple visual performance model that shows whether a program is limited by memory bandwidth or by arithmetic bandwidth.
- Compute-boundA kernel is compute-bound when its speed is limited by the arithmetic bandwidth of the CUDA Cores or Tensor Cores.
- Memory-boundA kernel is memory-bound when its speed is limited by the GPU's memory bandwidth rather than its arithmetic throughput.
- Arithmetic IntensityThe ratio of arithmetic operations to memory traffic in a kernel, usually measured in operations (FLOPs) per byte moved.
- OverheadTime during which the GPU does no useful work because it is waiting to receive work.
- Little's LawA queueing law stating that the concurrency needed to hide latency equals latency multiplied by throughput.
- Memory BandwidthThe maximum rate, in bytes per second, at which data can move between levels of the memory hierarchy.
- Arithmetic BandwidthThe peak rate at which a system can perform arithmetic, measured in operations (usually FLOPs) per second.
- Latency HidingMasking the latency of slow operations by running many of them concurrently, so execution units stay busy while individual operations wait.
- Warp Execution StateThe overlapping states (active, eligible, stalled, selected) that describe what a warp running a kernel is doing on a given cycle.
- Active CycleA clock cycle in which a Streaming Multiprocessor has at least one active warp resident, whether eligible or stalled.
- OccupancyThe ratio of active warps on a device to the maximum number of warps that could be active.
- Pipe UtilizationHow close each execution pipe inside an SM gets to its peak rate while it is actively running warps.
- Peak RateThe theoretical maximum rate at which a hardware system can complete work, computed from its hardware specifications.
- Issue EfficiencyThe fraction of cycles on which a warp scheduler issues an instruction from an eligible warp.
- SM UtilizationThe percentage of time the GPU's Streaming Multiprocessors spend executing instructions.
- Warp DivergenceWhen threads in the same warp take different execution paths because of control flow, so some must be masked off.
- Scoreboard StallA stall where an instruction can't issue because it depends on the result of an earlier instruction that hasn't finished.
- Branch EfficiencyHow often all threads in a warp take the same path at a conditional; high branch efficiency means little warp divergence.
- Memory CoalescingA hardware technique that serves several logical global memory reads with one physical access, so memory bandwidth is used fully.
- Bank ConflictWhen threads in a warp access different addresses in the same shared memory bank at once, forcing their accesses to be serialized.
- Register PressureThe situation where the register file becomes a performance bottleneck, limiting how many threads an SM can run.