Evolution · 2.1
Why GPUs?
Why did GPUs (graphics processing units), not traditional CPUs (central processing units) or purpose-built AI chips, become the processors modern AI runs on? Four questions, one race, and the infrastructure that follows from the answer.
Question 1 of 4
Why are GPUs fundamentally different from CPUs?
A CPU spends its silicon on a few sophisticated s that make decisions quickly, one step after another. A GPU spends it on a great many simpler arithmetic units that do the same calculation on different data at the same time.
- Relatively few, powerful cores
- Optimised for low latency and flexible control
- Runs the operating system, application logic and complex instructions
- Strong on sequential, branching work
- Very many parallel arithmetic units
- Optimised for throughput
- Performs many similar operations concurrently
- Strong on large, regular mathematical workloads
ConceptualA picture of the design emphasis, not of real cores: the number of squares is not a core count.
Emphasis, not absolutes: modern CPUs have many cores and vector units, and modern GPUs run general-purpose programs. The difference is what each design spends its transistors on.
Explore deeper
Most of a CPU core’s area is control machinery: branch prediction, out-of-order execution and deep caches that hide the cost of unpredictable memory access. They make one unpredictable stream of instructions run fast.
A GPU groups threads that execute the same instruction together. When threads in a group take different branches, the group runs each branch in turn, which is why branch-heavy code runs poorly on it even though it can be written for it.
Graphics needed exactly this shape of work, the same calculation for millions of pixels and vertices each frame, which is why the architecture existed before AI wanted it.
Question 2 of 4
Why do AI models benefit so much from GPU processing?
A neural network is mostly : inputs multiplied by learned , layer after layer. Each output number is independent of its neighbours, so thousands can be computed at once. GPUs do that, feed it with high , and add hardware such as built for exactly this arithmetic.
| 1 | 0 | 2 |
| 2 | 1 | 0 |
| 0 | 1 | 1 |
| 1 | 2 | 1 |
| 1 |
| 2 |
| 1 |
| 0 |
| 0 |
| 0 |
| 0 |
12 multiply-adds · step 0 of 12
ConceptualToy weights and a single input. Each output is its own sum, independent of the others: that independence is what lets parallel hardware compute them together.
This toy layer needs a handful of multiply-adds. One layer of a large language model can need hundreds of millions for every token it produces, and the model has dozens of layers.
Training
The model processes examples again and again and adjusts its weights each time.
- Very large amounts of arithmetic, for weeks
- Memory for weights, gradients and optimiser state
- Fast links between many accelerators to share gradients
Inference
A trained model uses its weights to answer: a prediction, a classification, the next token.
- Low latency for each user
- Reading the weights quickly: often limited by memory bandwidth, not arithmetic
- Cost per request or per token at the volume actually served
Inference is not always dominated by large matrix multiplication. With small requests, the time spent moving weights from memory, handing work to the accelerator, and the serving software around the model can decide the speed.
Explore deeper
In a transformer, the weight matrices are fixed during inference, while every token produces fresh activations. Generating one token at a time reads the whole set of weights for little arithmetic, which is why token generation is usually limited by memory bandwidth.
Batching many users’ requests together lets one read of the weights serve many tokens, moving the bottleneck back toward arithmetic. That trade between per-user speed and total throughput is a serving decision, not a property of the chip.
Simulation
Compute Race: CPU vs GPU vs TPU
256 independent multiply-add tiles: any can be computed in any order.
CPU
256 units of work, ready
GPU
256 units of work, ready
TPU / dedicated AI accelerator
256 units of work, ready
- Multiply-add tile
- waiting
- running
- done
ConceptualIllustrative model in conceptual time units, built to show architectural principles. It is not a benchmark of any real CPU, GPU or TPU, and squares are units of work, not cores. Real results depend on the product, model, precision, batch size and software.
| CPU | GPU | TPU / dedicated AI accelerator | |
|---|---|---|---|
| Suitability here | A few groups at a time: correct and flexible, but the work queues up. | Many groups at once: the work spreads across the chip. | A fixed matrix-processing structure streams supported operations through efficiently; less flexible for anything else. |
| Parallelism | Tens of cores, wide vector units | Very many lightweight threads | A large matrix engine fed by a compiler |
| Flexibility | Runs anything | Programmable; weak on branch-heavy code | Supported operations only; others fall back or fail to compile |
| Memory | Large and cheap, lower bandwidth | HBM: high bandwidth, limited capacity | HBM plus on-chip memory near the arithmetic |
| Energy | Efficient for control work; wasteful on dense maths | Efficient per operation on parallel maths; high power per device | Most efficient on its target workload; wasted on anything else |
Question 3 of 4
Why did GPUs become dominant before purpose-built AI chips?
Because GPUs already existed, were sold in volume and could be programmed. Researchers could buy them and start, instead of waiting years for a dedicated chip to be designed and built. made them approachable; deep learning’s success made them essential; GPUs then grew AI-specific hardware of their own. Purpose-built accelerators arrived later and now compete for particular workloads.
1999
The GPU as a category
- What happened
- NVIDIA launches what it calls the first GPU and popularises the term for a dedicated graphics processor.
- Why it mattered
- Graphics accelerators existed before; this framed a separate processor category built for parallel arithmetic, sold in consumer volumes.
- Impact on AI infrastructure
- Volume made the hardware cheap and everywhere, years before anyone used it for AI.
Sources: NVIDIA
GPUs didn’t win because they were originally designed for AI. They became dominant because their architecture, programmability, availability and software ecosystem made them exceptionally useful for deep learning.
What was built after that, generation by generation: The evolution of AI infrastructure (2.2) →
Question 4 of 4
Are GPUs always the best choice for AI?
No. GPUs are versatile and backed by the most mature software, which makes them the safe default. Dedicated chips can give better performance, energy efficiency or economics on particular workloads. The right choice depends on what the system has to do.
| Processor | Primary strength | Typical applications |
|---|---|---|
| CPU | Flexible general-purpose processing | Application logic, orchestration, preprocessing |
| GPU | High-throughput parallel computation | AI training, inference, scientific computing |
| TPU (Tensor Processing Unit) | Specialised tensor processing | Supported training and inference workloads |
| NPU (neural processing unit) | Efficient neural-network processing | Mobile, edge and embedded AI |
| AI ASIC (application-specific integrated circuit) | Workload-specific acceleration | Specialised cloud and inference systems |
These categories overlap. A TPU is one kind of AI ; "" is a broad label that can describe a phone’s AI block or a data-centre accelerator. Treat the rows as tendencies, not separate species.
Choose your AI processor
Likely candidates
- GPUs
- Specialised training accelerators (e.g. TPU, Trainium)
Weeks of dense arithmetic across thousands of devices. Memory per device, the interconnect between them and the distributed-training software decide the result as much as the chip.
Watch: Whether your framework, model and parallelism strategy are supported on the accelerator, and whether capacity can actually be obtained.
How much each factor weighs here
- PerformanceDecisive
- CostDecisive
- Energy efficiencyImportant
- Memory capacityDecisive
- Software ecosystemDecisive
- Deployment flexibilitySecondary
- AvailabilityDecisive
A starting point, not a recommendation. Real selection needs benchmarks on your own workload and a total cost analysis: hardware, power, software effort, staff skills and what capacity you can actually get.
The full side-by-side, with where each stops being the right answer: CPUs, GPUs and TPUs compared (3.2) →
Why NVIDIA became so dominant
It’s not just the chip: the software ecosystem
Compiler, runtime and software ecosystem
NVIDIA CUDA, cuDNN, TensorRT; AMD ROCm; Google XLA for TPUs; AWS Neuron for Trainium and Inferentia
Why the processor reaches this layer: Where the lock-in actually lives: tuned kernels, collective-communication libraries and years of debugging tools.
Not every AI workload needs CUDA. AMD’s ROCm, Google’s XLA and JAX for TPUs, AWS Neuron and other toolchains run real production workloads. The cost of switching is the engineering effort to port, tune and support the model, not a technical impossibility.
From compute to infrastructure
Choosing GPUs chooses a data centre
AI requires large-scale computation
GPUs accelerate parallel mathematical operations
The arithmetic is only fast if data reaches it fast enough.
GPUs need high-bandwidth memory (HBM)
AI accelerators move large volumes of weights and intermediate data. HBM stops memory transfer from capping compute performance.
Multiple GPUs need fast interconnects
When a workload spans several GPUs, they exchange data constantly; NVLink and NVSwitch let a rack of them work as one.
Distributed AI systems need high-performance networking
Beyond one server or rack, InfiniBand or suitable Ethernet / RoCE (RDMA over Converged Ethernet, direct memory access across the network) fabrics carry the traffic between them.
High-density systems require advanced power and cooling
Dense accelerator racks draw far more power and give off far more heat than ordinary IT racks.
The modern AI data centre
Not every AI system needs HBM, NVLink, InfiniBand or liquid cooling. A single inference server or an edge device needs none of them; a large training cluster needs all of them. The requirements follow the architecture, scale and workload.
Knowledge check
Five questions
5 questions
Sources
Historical milestones rest on the sources below, read on 22 October 2026. Architectural explanations are general principles; the simulation and the chooser are illustrative models, not measurements of real hardware.
- Corporate timeline (1999 GPU, 2006 CUDA, 2012 AlexNet) · NVIDIA
- CUDA C++ Programming Guide · NVIDIA
- Krizhevsky, Sutskever, Hinton: ImageNet Classification with Deep Convolutional Neural Networks (NIPS 2012) · NeurIPS proceedings
- Jouppi et al.: In-Datacenter Performance Analysis of a Tensor Processing Unit (2017) · arXiv
- An in-depth look at Google’s first Tensor Processing Unit · Google Cloud
- NVIDIA launches Volta GPU platform (May 2017) · NVIDIA Newsroom
- NVIDIA Tesla V100 GPU Architecture whitepaper · NVIDIA
- Vaswani et al.: Attention Is All You Need (2017) · arXiv
- AWS Trainium · Amazon Web Services
- What is ROCm? · AMD