GPU Glossary
/
Device Hardware·2 min read

Warp Scheduler

The Streaming Multiprocessor unit that decides, on every clock cycle, which group of threads (warp) executes next.

The Warp Scheduler of a Streaming Multiprocessor (SM) decides which group of threads to execute on each clock cycle. These groups are called warps.

The internal architecture of an H100 SM. The Warp Scheduler and Dispatch Unit are shown in orange. Modified from NVIDIA's H100 white paper.
The internal architecture of an H100 SM. The Warp Scheduler and Dispatch Unit are shown in orange. Modified from NVIDIA's H100 white paper.

Warps are switched out every clock cycle, roughly once a nanosecond. This resembles the fine-grained thread-level parallelism of simultaneous multi-threading ("hyper-threading") in CPUs, but at a much larger scale.

Key idea Warp Schedulers can switch rapidly among many concurrent warps, running whichever one has its instruction's operands ready. This is key to GPUs' latency hiding.

Why GPU context switches are cheap#

A full CPU thread context switch takes a few hundred to a few thousand clock cycles, closer to a microsecond than a nanosecond. The CPU must save one thread's context and restore another's. Context switches also reduce locality, which raises cache miss rates and hurts performance further (see Mogul and Borg, 1991).

GPUs avoid both costs:

  • No data movement. Each thread has its own private registers allocated from the SM's register file, so nothing needs to be saved or restored.
  • Little cache disruption. The L1 caches can be entirely programmer-managed and are shared among the warps scheduled together onto an SM (see cooperative thread array). Switching warps therefore has much less impact on cache hit rates.

For how programmer-managed and hardware-managed caches interact on GPUs, see the "Maximize Memory Throughput" section of the CUDA C Programming Guide.

Warp Schedulers also manage the execution state of warps.