GPU Glossary
/
Device Hardware·2 min read

Tensor Memory AcceleratorTMA

Hopper and Blackwell hardware that asynchronously copies multi-dimensional arrays from GPU RAM into shared memory, bypassing registers.

Tensor Memory Accelerators (TMAs) are specialized hardware in Hopper and Blackwell architecture GPUs that accelerate access to multi-dimensional arrays in GPU RAM.

The internal architecture of an H100 Streaming Multiprocessor (SM). The Tensor Memory Accelerator sits at the bottom of the SM, shared between the four sub-units. Modified from NVIDIA's H100 white paper.
The internal architecture of an H100 Streaming Multiprocessor (SM). The Tensor Memory Accelerator sits at the bottom of the SM, shared between the four sub-units. Modified from NVIDIA's H100 white paper.

The TMA loads data from global memory/GPU RAM into shared memory/L1 data cache, bypassing the registers/register file entirely.

Advantage 1: offloading address math#

The TMA hardware computes addresses for bulk affine memory accesses, i.e. accesses of the form addr = width * base + offset, for many bases and offsets concurrently. These are the most common accesses for arrays.

Offloading this work saves register file space, reducing "register pressure". It also reduces demand on the arithmetic bandwidth of the CUDA Cores. The savings are largest for large (KB-scale) accesses to arrays with two or more dimensions.

Advantage 2: asynchronous copies#

Key idea One thread can kick off a large TMA copy and immediately go back to other work.

A single CUDA thread triggers the copy and then rejoins its warp. That thread and others in the same thread block can later detect, asynchronously, that the copy has finished and operate on the results, as in a producer-consumer model.

Despite the name, the Tensor Memory Accelerator does not accelerate operations using Tensor Memory.

Further reading#