GPU Glossary
/
Device Software·2 min read

Memory Hierarchy

The CUDA programming model's levels of memory, each shared by the threads at one level of the thread hierarchy: registers, shared memory and global memory.

In the CUDA programming model, each level of the thread hierarchy has its own block of memory shared by all threads in a group at that level. This is the memory hierarchy. The memory is used for coordination and communication, and it is managed by the programmer, not by the hardware or a runtime.

Shared memory and global memory are two levels of the memory hierarchy in the CUDA programming model (left), mapping onto the L1 data cache and GPU RAM respectively. Modified from diagrams in NVIDIA's CUDA Refresher: The CUDA Programming Model and the NVIDIA CUDA C++ Programming Guide.
Shared memory and global memory are two levels of the memory hierarchy in the CUDA programming model (left), mapping onto the L1 data cache and GPU RAM respectively. Modified from diagrams in NVIDIA's CUDA Refresher: The CUDA Programming Model and the NVIDIA CUDA C++ Programming Guide.
Thread level Memory Where it lives
Thread Registers The SM's register file
Thread block Shared memory The SM's L1 data cache
Thread block grid Global memory GPU RAM

Grid: global memory#

For a thread block grid, the shared memory is in the GPU's RAM and is called global memory. Access can be coordinated with atomic operations and barriers, but execution order across thread blocks is indeterminate.

Thread: registers#

For a single thread, the memory is a chunk of the Streaming Multiprocessor's (SM's) register file. Under the original semantics of the programming model, this memory is private to the thread. However, some instructions added to PTX and SASS for matrix multiplication on Tensor Cores share inputs and outputs across threads.

Block: shared memory#

In between, the shared memory for the thread block level is stored in each SM's L1 data cache. Managing this cache carefully, for example by loading data so it supports the maximum number of arithmetic operations before new data is loaded, is central to designing high-performance CUDA kernels.

Key idea Each level of threads gets matching memory, and fast kernels come from keeping data in the smaller, faster levels as long as possible.