SM Utilization
The percentage of time the GPU's Streaming Multiprocessors spend executing instructions.
SM utilization measures the percentage of time that Streaming Multiprocessors (SMs) are executing instructions. It is a finer-grained version of the familiar GPU utilization number.
Compared with kernel utilization#
The kernel utilization reported by nvidia-smi is the fraction of time a kernel is running anywhere on the GPU. SM utilization instead reports the fraction of time all SMs spend executing kernels.
The difference can be huge. A kernel that uses only one SM, for instance because it has only one thread block, gets 100% GPU utilization while it runs. Its SM utilization is at most one over the number of SMs: under 1% on an H100 GPU.
As with GPU utilization, but unlike CPU utilization, SM utilization should be high, even up to 100%.
When it isn't enough#
SM utilization still doesn't show how well each SM's compute resources are used. If SM utilization is high but performance is still poor, check pipe utilization, which measures how effectively each SM uses its internal functional units.
Key idea High SM utilization with low pipe utilization means the kernel is spread across many SMs but isn't making full use of the hardware inside each one.
Related#
- Pipe utilization — the next, finer level of detail.
- Peak rate — the ultimate measure of utilization.
- Occupancy — how much work each SM holds at once.