GPU Glossary
/
Host Software·2 min read

cuDNN

NVIDIA's library of highly optimized GPU primitives for deep neural networks, such as convolution, attention and normalization.

cuDNN (CUDA Deep Neural Network) is NVIDIA's library of primitives for building GPU-accelerated deep neural networks. It provides highly optimized kernels for operations common in neural networks, including:

  • convolution
  • self-attention, including scaled dot-product attention (aka "Flash Attention")
  • matrix multiplication
  • various normalizations
  • poolings

Where it fits#

cuDNN is a key library at the application layer of the CUDA software platform, alongside its sibling library cuBLAS. Deep learning frameworks like PyTorch typically use cuBLAS for general-purpose linear algebra, such as the matrix multiplications at the core of dense (fully-connected) layers. They use cuDNN for more specialized primitives like convolutional layers, normalization routines and attention.

Operation graphs and fusion#

Modern cuDNN code expresses computations as operation graphs. You build them with the open source Python and C++ frontend APIs via the declarative Graph API (not to be confused with CUDA Graphs).

Given a graph of operations, cuDNN can analyze it and optimize it. The most important optimization is operation fusion: a sequence like Convolution + Bias + ReLU is merged ("fused") into one operation that runs as a single kernel.

Key idea Fusion reduces demand on memory bandwidth by keeping intermediate results in shared memory across a sequence of operations.

The frontends sit on top of a lower-level, closed source C backend, which exposes an API for legacy use cases or direct C FFI.

Kernel selection#

For each operation, cuDNN keeps multiple implementations. It uses internal heuristics (not publicly known) to pick the fastest one for the target Streaming Multiprocessor (SM) architecture, data types and input sizes.

History#

cuDNN first made its name accelerating convolutional neural networks on Ampere SM architecture GPUs. For Transformer networks on Hopper and especially Blackwell SM architectures, NVIDIA has tended to put more emphasis on the CUTLASS library.

See the official cuDNN documentation and the open source frontend APIs.

  • cuBLAS: sibling library for general linear algebra
  • CUTLASS: NVIDIA's favored toolkit for Transformer kernels on newer GPUs
  • Memory-bound: the problem operation fusion attacks