GPU Glossary
/
Device Hardware·3 min read

Streaming MultiprocessorSM

The basic processor of an NVIDIA GPU, which executes instructions for many threads at once and holds their registers and caches.

Streaming Multiprocessors (SMs) are the units that run GPU programs. When we program GPUs, we produce sequences of instructions for the SMs to carry out. They are roughly analogous to CPU cores: they execute computations and hold the state those computations use, in registers and associated caches.

The internal architecture of an H100 GPU's Streaming Multiprocessors. GPU cores appear in green, other compute units in pink, scheduling units in orange, and memory in blue. Modified from NVIDIA's H100 white paper.
The internal architecture of an H100 GPU's Streaming Multiprocessors. GPU cores appear in green, other compute units in pink, scheduling units in orange, and memory in blue. Modified from NVIDIA's H100 white paper.

Simple processors, many threads#

Compared to CPU cores, SMs are simple, weak processors. Execution is pipelined within an instruction, as in almost all CPUs since the 1990s, but there is no speculative execution or instruction pointer prediction, unlike every contemporary high-performance CPU.

What SMs offer instead is the ability to execute far more threads in parallel:

AMD EPYC 9965 CPU H100 SXM GPU
Max power 500 W 700 W
Processors 192 cores 132 SMs
Parallel threads per processor 2 128 (4 Warp Schedulers × 32 threads)
Total parallel threads 384 128 × 132 > 16,000
Power per thread ≈ 1.25 W ≈ 5 cW

(EPYC 9965 specs.) On the H100, each Warp Scheduler can issue an instruction to 32 threads (a warp) per clock cycle. This is truly parallel: each of the 16,000+ threads can make progress on every clock cycle.

Even more concurrent threads#

SMs also support a large number of concurrent threads, meaning threads of execution whose instructions are interleaved. A single H100 SM can hold up to 2048 threads, split into 64 groups of 32. Across 132 SMs, that's over 250,000 concurrent threads.

CPUs can run many concurrent threads too. The difference is that an SM switches between warps in a single clock cycle, over 1000× faster than a CPU context switch, again thanks to the Warp Schedulers.

Why this matters: latency hiding#

The large pool of warps and the speed of switching between them let the SM hide latency from memory reads, thread synchronization and other expensive instructions. That keeps the arithmetic bandwidth of the CUDA Cores and Tensor Cores well utilized.

Key idea Latency hiding is the secret to GPUs' strength. CPUs hide latency with large hardware-managed caches and sophisticated instruction prediction; GPUs hide it by always having another warp ready to run.

That extra CPU hardware limits how much silicon area, power and heat budget CPUs can spend on computation.

GPUs dedicate more of their area to compute (green), and less to control and caching (orange and blue), than CPUs do. Modified from a diagram in Fabien Sanglard's blog, itself likely modified from a diagram in the CUDA C Programming Guide.
GPUs dedicate more of their area to compute (green), and less to control and caching (orange and blue), than CPUs do. Modified from a diagram in Fabien Sanglard's blog, itself likely modified from a diagram in the CUDA C Programming Guide.

The payoff is much higher throughput for programs where it is fairly easy for programmers to express cache behavior themselves, such as neural network inference or sequential database scans. For example, a matrix kernel can store a chunk of each input matrix in cache and keep it there long enough to compute the related outputs.