GPU Glossary
/
Device Software·3 min read

CUDA Programming Model

NVIDIA's model for writing massively parallel programs, built on a hierarchy of thread groups, a hierarchy of memories and barrier synchronization.

The CUDA programming model is a way of writing programs for massively parallel processors. Its abstractions are designed so that a program gets faster automatically when it runs on a GPU with more parallel hardware.

"CUDA" stands for Compute Unified Device Architecture, and the name covers three distinct things:

The vision for CUDA is laid out in the Lindholm et al., 2008 white paper. It is highly recommended reading: it is the original source of many claims, diagrams and even turns of phrase in NVIDIA's documentation.

Three key abstractions#

Per the NVIDIA CUDA C++ Programming Guide, the model rests on three abstractions:

The diagram below shows the execution and memory hierarchies and how they map onto device hardware.

Left: the abstract thread group and memory hierarchies of the CUDA programming model. Right: the hardware that implements them. Modified from diagrams in NVIDIA's CUDA Refresher: The CUDA Programming Model and the NVIDIA CUDA C++ Programming Guide.
Left: the abstract thread group and memory hierarchies of the CUDA programming model. Right: the hardware that implements them. Modified from diagrams in NVIDIA's CUDA Refresher: The CUDA Programming Model and the NVIDIA CUDA C++ Programming Guide.

Why it scales#

Together, these abstractions push programmers to write code that scales transparently as GPUs gain more parallel execution resources. Put provocatively, the model stops you from writing programs for CUDA-architected GPUs that fail to get faster when the user buys a newer NVIDIA GPU.

Thread blocks are the clearest example. Threads within a block can coordinate tightly, but coordination between blocks is limited. So blocks capture independent, parallelizable pieces of the program and can be scheduled in any order. In computer-architecture terms, the programmer "exposes" this parallelism to the compiler and hardware. On a GPU with more scheduling units (specifically, more Streaming Multiprocessors), more blocks simply run at once.

A CUDA program with eight blocks runs in four sequential steps (waves) on a GPU with two SMs, and in half as many steps on a GPU with twice as many SMs. Modified from the CUDA Programming Guide.
A CUDA program with eight blocks runs in four sequential steps (waves) on a GPU with two SMs, and in half as many steps on a GPU with twice as many SMs. Modified from the CUDA Programming Guide.

Key idea Because blocks must be independent, the same program fills a bigger GPU with more blocks at once, with no code changes.

How it is implemented#

Programmers reach the model through extensions to high-level CPU languages, such as the CUDA C++ extension of C++. Underneath, it is implemented by an instruction set architecture, Parallel Thread eXecution (PTX), and a low-level assembly language, Streaming Assembler (SASS). For example, the thread block level of the thread hierarchy is implemented in these languages as cooperative thread arrays.