GPU Glossary
/
Device Software·2 min read

Warp

A group of 32 threads that are scheduled together onto one Streaming Multiprocessor and execute the same instruction in parallel.

A warp is a group of threads that are scheduled together and execute in parallel. Warps are the typical unit of execution on a GPU, and understanding how they are scheduled is central to GPU performance.

Scheduling and SIMT#

All threads in a warp are scheduled onto a single Streaming Multiprocessor (SM). An SM typically runs multiple warps: at minimum, all warps from the same Cooperative Thread Array, also known as a thread block.

Normally, all threads in a warp execute the same instruction in parallel. This is the "Single-Instruction, Multiple Thread" (SIMT) model. When threads in a warp split up to execute different instructions, called warp divergence, performance generally drops sharply.

Warp size is technically a machine-dependent constant, but in practice (and throughout this glossary) it is 32.

Stalls and latency hiding#

When a warp is issued an instruction, the result is usually not ready within a single clock cycle, so dependent instructions can't be issued yet. This is most obviously true for fetches from global memory, which usually go off-chip, but it also applies to some arithmetic instructions. The CUDA C++ Best Practices Guide has a table of results per clock cycle for specific instructions.

A warp whose next instruction is waiting on missing operands is stalled.

When several warps are scheduled on one SM, the Warp Scheduler doesn't wait for a stalled warp; it picks another warp to run. This latency hiding is how GPUs achieve high throughput and keep all their cores busy.

Key idea GPUs hide latency by switching between warps, so it often pays to keep many warps resident on each SM, ensuring there is always an eligible one to run.

Two related metrics:

  • Issue efficiency: the fraction of cycles on which a warp was issued an instruction.
  • Occupancy: the degree of concurrency in warp scheduling.

An implementation detail, not an abstraction#

Warps are not part of the CUDA programming model's thread hierarchy. They are a detail of how NVIDIA GPUs implement that model. In that respect they resemble cache lines on CPUs: a hardware feature you don't control directly and don't need for correctness, but which matters a lot for maximum performance.

Name and equivalents#

The name comes from weaving, "the first parallel thread technology", according to Lindholm et al., 2008.

Equivalents in other GPU programming models: