Device Hardware
The physical parts of a GPU, from the chip-wide memory down to the cores that do the arithmetic.
In NVIDIA's terms, the GPU is the device and the CPU that controls it is the host. This section covers the physical hardware on the device: the Streaming Multiprocessors that run your code, the cores and special units inside them, and the memory that feeds them.
If you are new to GPUs, start with the Streaming Multiprocessor. It is the GPU's basic building block, and most other terms in this section describe its parts.
- CUDA Device ArchitectureThe unified GPU hardware design behind CUDA, built from many identical, general-purpose Streaming Multiprocessors instead of a fixed graphics pipeline.
- Streaming Multiprocessor SMThe basic processor of an NVIDIA GPU, which executes instructions for many threads at once and holds their registers and caches.
- GPU CoreThe primary compute units inside a Streaming Multiprocessor, such as CUDA Cores and Tensor Cores, best thought of as pipes that transform data.
- Special Function Unit SFUHardware units in a Streaming Multiprocessor that accelerate certain arithmetic operations, notably transcendental functions like exp, sin and cos.
- Load/Store Unit LSUUnits in a Streaming Multiprocessor that dispatch requests to load data from, or store data to, the GPU's memory subsystems.
- Warp SchedulerThe Streaming Multiprocessor unit that decides, on every clock cycle, which group of threads (warp) executes next.
- CUDA CoreGPU cores that execute scalar arithmetic instructions, issued in groups (usually a 32-thread warp) by the Warp Scheduler.
- Tensor CoreGPU cores that operate on entire matrices with each instruction, providing the highest arithmetic throughput on NVIDIA GPUs since Volta.
- Tensor Memory Accelerator TMAHopper and Blackwell hardware that asynchronously copies multi-dimensional arrays from GPU RAM into shared memory, bypassing registers.
- Streaming Multiprocessor ArchitectureThe versioned design of a GPU's Streaming Multiprocessors, which determines which compiled SASS code it can run.
- Texture Processing Cluster TPCA pair of adjacent Streaming Multiprocessors, which since Blackwell maps onto the "CTA pair" level of the PTX thread hierarchy.
- Graphics/GPU Processing Cluster GPCA collection of Texture Processing Clusters plus a raster engine; since compute capability 9.0, thread block clusters are scheduled onto one GPC.
- Register FileThe Streaming Multiprocessor's fastest memory, made of 32 bit registers that hold data between operations by the cores.
- L1 Data CacheThe private on-chip SRAM memory of each Streaming Multiprocessor, partitioned among its thread blocks and largely managed by the programmer.
- Tensor MemoryA specialized, tightly restricted memory in the Streaming Multiprocessor of GPUs like the B200 that stores Tensor Core inputs and outputs.
- GPU RAMThe GPU's large, bottom-level DRAM memory, addressable by all Streaming Multiprocessors and used to implement global memory.