GPU Glossary
/
Host Software·2 min read

CUTLASS

NVIDIA's open-source CUDA C++ template library for building high-performance linear algebra kernels, especially matrix multiplications.

CUTLASS (CUDA Templates for Linear Algebra Subroutines and Solvers) is a library of abstractions for implementing high-performance linear algebra in CUDA kernels.

CUTLASS vs cuBLAS#

Like cuBLAS, CUTLASS is named after the Basic Linear Algebra Subprograms (BLAS) standard for low-level linear algebra routines. The difference:

  • cuBLAS is a library of ready-to-call routines.
  • CUTLASS is a toolkit for constructing kernels.

CUTLASS is mainly associated with the third level of the BLAS hierarchy: general matrix multiplications ("GEMMs").

Templates#

As the name says, CUTLASS is a collection of CUDA C++ template abstractions. Templates are C++'s implementation of parametric polymorphism, which you may know as generics in other languages: a polymorphic function is written once but works on inputs of different types.

Structure#

The core of modern CUTLASS is CuTe, which defines Layout and Tensor types for composably describing and manipulating tensors of data and threads. Don't confuse it with CuTe DSL, which exposes CuTe/CUTLASS templates through a Python Domain-Specific Language (DSL).

On top of CuTe, CUTLASS provides a header-only CUDA C++ library at three levels:

  • device: the whole device
  • kernel: a single kernel
  • collective: a group of threads, typically a thread block

At the collective level, matrix multiplications are usually split in two:

  • Mainloops express the core algorithm, such as the tiling strategy.
  • Epilogues describe post-processing, such as applying scaling factors or scalar non-linearities (popular in neural networks).

Why people use it#

CUTLASS is very commonly used to write some of the fastest kernels, especially matrix multiplications on recent Streaming Multiprocessor architectures. These kernels need careful programming of the Tensor Cores to get anywhere near peak performance.

CUTLASS is open source on GitHub. It also ships many high-performance open-source kernels written with CUTLASS, which are widely used as references in open-source kernel development.

To learn it, the tutorials by Jay Shah of Colfax International are highly recommended; they explain in detail how CUTLASS's key components achieve maximum performance. Be warned: like most C++ template metaprogramming, CUTLASS is not for the faint of heart.

  • CuTe: the layout and tensor library at CUTLASS's core
  • CuTe DSL: Python access to CuTe/CUTLASS
  • cuBLAS: ready-made GEMM routines
  • Tensor Core: the hardware CUTLASS kernels target