GPU Glossary
/
Device Hardware·1 min read

CUDA Core

GPU cores that execute scalar arithmetic instructions, issued in groups (usually a 32-thread warp) by the Warp Scheduler.

CUDA Cores are GPU cores that execute scalar arithmetic instructions. They contrast with Tensor Cores, which execute matrix operations.

The internal architecture of an H100 SM. CUDA Cores and Tensor Cores are shown in green; note the Tensor Cores are larger and fewer. Modified from NVIDIA's H100 white paper.
The internal architecture of an H100 SM. CUDA Cores and Tensor Cores are shown in green; note the Tensor Cores are larger and fewer. Modified from NVIDIA's H100 white paper.

Scheduled in groups, not individually#

Unlike CPU cores, CUDA Cores generally aren't scheduled independently. The Warp Scheduler issues the same instruction to a group of cores at once, and each applies it to different registers.

These groups are commonly 32 wide, the size of a warp. On contemporary GPUs a group can contain as little as one thread, at a cost to performance.

A slippery term#

What counts as a "CUDA Core" varies across Streaming Multiprocessor architectures. Different generations mix 32 bit integer, 32 bit floating point and 64 bit floating point units differently. The term is best understood in contrast to early GPUs, which had a variety of much more specialized compute units mapped onto shader pipelines (see CUDA Device Architecture).

For example, the H100 white paper says each H100 Streaming Multiprocessor (SM) has 128 "FP32 CUDA Cores". That accurately counts the 32 bit floating point units, but it is double the number of 32 bit integer or 64 bit floating point units, as the diagram above shows.

Key idea To estimate performance, count the hardware units for the specific operation you care about, not "CUDA Cores".