GPU Glossary
/
Device Software·1 min read

Shared Memory

The memory shared by the threads of a thread block, much smaller but much faster than global memory, stored in the SM's L1 data cache.

Shared memory is the level of the memory hierarchy that matches the thread block level of the thread hierarchy in the CUDA programming model. It is generally much smaller than global memory, but much faster in both throughput and latency.

Shared memory is the abstract memory associated with the thread block level (left, center) of the CUDA thread group hierarchy (left). Modified from diagrams in NVIDIA's CUDA Refresher: The CUDA Programming Model and the NVIDIA CUDA C++ Programming Guide.
Shared memory is the abstract memory associated with the thread block level (left, center) of the CUDA thread group hierarchy (left). Modified from diagrams in NVIDIA's CUDA Refresher: The CUDA Programming Model and the NVIDIA CUDA C++ Programming Guide.

A typical kernel pattern#

Because of this trade-off, a fairly typical kernel looks like this:

  1. Load data from global memory into shared memory.
  2. Perform many arithmetic operations on that data using the CUDA Cores and Tensor Cores.
  3. Optionally, synchronize threads within the thread block with barriers while doing those operations.
  4. Write data back to global memory, optionally using atomics to prevent races across thread blocks.

Key idea Stage data in shared memory so it is loaded from slow global memory once and reused many times.

Hardware#

Shared memory is stored in the L1 data cache of the GPU's Streaming Multiprocessor (SM).