Architecture · Compute
Key Fundamentals
What lets AI compute work, and grow? A handful of technologies come up again and again. Here is each one: what it is, what it does, and why it matters, before you step inside the rack that brings them all together.
Compute · 01
GPU + HBM
The engine and the memory that feeds it.
- What it is
- The GPU performs the AI computation. HBM (high-bandwidth memory) is very fast memory stacked right beside it.
- What it does
- The GPU runs thousands of calculations at once; HBM hands it the model and the data to work on, fast enough to keep up.
- Why it matters
- An AI model reads huge amounts of data for every answer it produces. If memory cannot keep up, the GPU sits idle, however powerful it is.
- Kind
- Chips: the GPU, with stacks of HBM mounted beside it in the same package.
- Supplied by
- GPUsNVIDIAAMDHBMSK hynixSamsungMicron
In-house accelerators such as Google TPU and AWS Trainium do the GPU’s job in some clouds. HBM is never bought on its own: it arrives inside the GPU.
Inside the package: a Rubin GPU is two compute dies (single pieces of silicon) side by side on a silicon interposer that wires them to each other and to the HBM stacks around them. Each die is filled with streaming multiprocessors, whose tensor cores do the matrix maths. See it in Chip to Rack →