CPUs, GPUs and TPUs compared
The processor classes side by side: what each is built to optimise, where each stops being the right answer, and why the choice is as much a supply and lock-in decision as a technical one.
CPUs, GPUs and TPUs are routinely discussed as though they were three grades of the same thing, with the GPU as the fast one and the TPU as the faster one. They are not. They are three points on a single spectrum that runs from fully general and programmable to narrow and hard-wired, and every difference in performance, cost, tooling and operational risk follows from where a processor sits on that spectrum.
The axis that explains all of it
Silicon area is finite. Every transistor spent on making a processor good at anything is a transistor not spent on making it good at one thing. A CPU spends most of its area on control logic — branch prediction, out-of-order execution, deep cache hierarchies — all of which exist to make unpredictable, branch-heavy, latency-sensitive code run quickly. A domain-specific accelerator spends almost none of its area that way, because it assumes the work is predictable dense arithmetic. The GPU sits between them and is, in practice, the most useful compromise for most organisations.
Moving right buys efficiency on the workload the silicon was designed for, and pays for it in flexibility when the workload changes.
Side by side
| CPU | GPU | TPU-class accelerator | |
|---|---|---|---|
| Designed to optimise | Latency of a single instruction stream | Throughput across many parallel threads | Throughput and efficiency on dense matrix arithmetic |
| Silicon spent mostly on | Control logic, branch prediction, large caches | Arithmetic units, register files, memory bandwidth | Matrix units and on-chip memory close to them |
| Parallelism model | A few powerful cores; instruction-level parallelism | Thousands of lightweight threads in lockstep groups | Data flowing through a fixed arithmetic structure |
| Handles branching code | Very well — this is what it is for | Poorly; divergent branches serialise within a thread group | Effectively not at all |
| Memory system | Deep cache hierarchy over large, cheap DRAM | High-bandwidth memory, capacity-constrained | Large on-chip memory plus high-bandwidth memory |
| Programming model | Any language, any workload, decades of tooling | Vendor toolkits and kernels, plus every major ML framework | Compiler-driven from a framework graph; hand-written kernels are rarely an option |
| Typical AI role | Data loading, preprocessing, scheduling, orchestration, serving glue | Training and inference for almost everything | Training and inference at scale for stable, high-volume model shapes |
| Where it wins | Irregular logic, small data, single-request latency, everything that is not a tensor | Anything parallel, and anything whose shape may change next quarter | A known workload run continuously at scale, where efficiency per watt compounds |
| Where it loses | Dense linear algebra at scale — by orders of magnitude | Efficiency per watt against purpose-built silicon on its own workload | Any workload the compiler does not map well, and any research-stage model |
| Procurement reality | Commodity; several credible suppliers | Constrained supply, long lead times, allocation-driven | Often available only inside one provider's cloud, on that provider's terms |
| Switching cost | Low | Moderate — framework portable, kernels and tuning less so | High — the model, the compiler and the platform are coupled |
A conceptual orientation, not a benchmark. Characteristics vary considerably between products within each category, and the boundaries move with every generation — CPUs have gained matrix instructions, and accelerators have gained programmability.
Each in its own terms
CPU
The CPU is the processor that can do anything. It is unmatched at code that makes decisions — branching, pointer-chasing, coordinating other components — and comparatively poor at doing the same arithmetic to millions of numbers at once.
Architecture
- Few, powerful cores. Each core carries substantial machinery to extract parallelism from a single instruction stream automatically.
- Deep cache hierarchy. Designed to hide the latency of unpredictable memory access, at the cost of area and power.
- Large addressable memory. Host memory is far larger and far cheaper per unit than accelerator memory, which is why it holds the data pipeline.
- Vector and matrix extensions. Modern server CPUs include instructions aimed at inference; they narrow the gap for small models without closing it for large ones.
Engineering deep dive
In an AI cluster the CPU is not the compute — it is the machine that keeps the compute fed. Data loading, decoding, augmentation, tokenisation, checkpoint serialisation, collective library setup, container runtime and the scheduler agent all run on host cores. A host specified as an afterthought becomes the reason expensive accelerators sit idle, and the symptom presents as poor accelerator utilisation, which sends investigation in exactly the wrong direction.
Operational implications
Host-side problems are diagnosed by looking at what the accelerator is waiting for, not at the accelerator itself. Input pipeline stalls, storage latency, and CPU core count or memory bandwidth per accelerator are the usual candidates. Establishing a per-node baseline for steps per second on a known job makes this tractable; without one, every slowdown looks like a hardware fault.
GPU
A GPU trades single-thread speed for enormous parallel throughput. It runs thousands of lightweight threads over the same instructions on different data — which is precisely the shape of the linear algebra underneath neural networks.
Architecture
- Throughput over latency. Individual operations are slower than on a CPU; the aggregate is far higher.
- Lockstep execution groups. Threads execute in groups, so divergent branches within a group serialise. Irregular control flow therefore performs badly even when it is expressible.
- Dedicated matrix units. Successive generations have added fixed-function units for mixed-precision matrix multiplication, moving the GPU rightwards along the specialisation spectrum without giving up general programmability.
- High-bandwidth memory. Capacity is the binding constraint on model size within one device; bandwidth is usually the binding constraint on speed.
- A tightly coupled interconnect. Accelerator-to-accelerator links let several devices behave as one larger unit — see the section on scale-up versus scale-out.
Engineering deep dive
Operational implications
GPUs fail in ways that are visible but ambiguous: memory errors, thermal throttling, link degradation, driver and firmware mismatch after maintenance. Because a synchronous training job runs at the pace of its slowest participant, one degraded device slows an entire job rather than failing it — which is why per-device health and throughput baselining matters more than up/down status.
TPU-class accelerators
A TPU-class accelerator is silicon built for one job: dense matrix multiplication at scale. Removing the machinery needed for general-purpose work leaves more area for arithmetic and for memory placed close to it, which is where the efficiency comes from. The cost is that anything outside the intended workload runs badly, or does not run at all.
Architecture
- Organised around a matrix engine rather than around many independent programmable cores. Data flows through a fixed arithmetic structure instead of being fetched, computed and written back per instruction.
- Large on-chip memory to keep weights and activations near the arithmetic, reducing the number of expensive trips to external memory.
- Compiler-driven execution. The framework graph is compiled ahead of time onto the device. There is limited scope for hand-tuning, and operations the compiler cannot map fall back to a slower path or fail to compile.
- A purpose-built interconnect between devices, since these accelerators are designed to be deployed as large coupled fleets rather than as individual cards.
Engineering deep dive
The engineering trade is predictability against adaptability. Where the model architecture is stable, the operations are supported and the deployment is large enough that efficiency per watt compounds into real money, this class of device is genuinely hard to beat. Where the model is still changing — new attention variants, custom kernels, unusual shapes — the same properties become obstacles, and engineering time goes into making the compiler accept the model rather than into the model itself. Named examples in this category include Google's TPU, AWS Trainium and Inferentia, and several dedicated inference accelerators.
Operational implications
The other processors in the machine
A modern AI node contains more processor classes than the three usually discussed. Each exists because something in the system needed work taken off the host.
- DPU / SmartNIC
- A programmable processor on the network adapter. Handles networking, storage access, encryption and infrastructure services so those cycles are not taken from host cores, and provides an isolation boundary between tenant workload and infrastructure control.
- NPU
- A neural processing unit — an inference-oriented accelerator, usually integrated alongside a CPU rather than sold as a discrete card. Common in edge and client devices; increasingly present in server processors.
- FPGA
- Reconfigurable logic. The circuit is defined after manufacture, giving ASIC-like efficiency on the target function with the ability to change it later. Development effort and per-unit cost keep it to specialised roles — protocol handling, low-latency signal processing, prototyping silicon.
- Inference-specific ASIC
- Silicon designed for serving rather than training, typically optimising tokens per second per watt at low batch sizes. A narrower proposition than a training accelerator, and a more tractable one, because the workload is more stable.
- CPU matrix extensions
- Matrix instructions built into general-purpose CPU cores. They make small-model and low-volume inference viable on hardware already deployed, which is often the right answer for workloads that do not justify an accelerator at all.
Choosing between them
| Situation | Sensible default | Why | What would change the answer |
|---|---|---|---|
| Research and model development | GPU | Every framework, kernel and debugging tool targets it first; model shapes change weekly | Almost nothing — flexibility is the requirement |
| Large-scale training of a stable architecture | GPU, with a TPU-class option evaluated | Efficiency per watt starts to dominate once the workload stops changing | Compiler support for the model, and willingness to accept provider coupling |
| High-volume inference of one model | Evaluate accelerators seriously | A fixed, continuously-served workload is exactly what specialised silicon is for | Frequent model replacement, or volume too low to amortise the migration |
| Low-volume or intermittent inference | CPU | Accelerator capacity sitting idle costs more than slower inference does | A latency requirement the CPU cannot meet |
| Classical ML, feature engineering, data preparation | CPU | The work is branch-heavy and memory-bound rather than dense arithmetic | Datasets large enough that specific stages become GPU-accelerated |
| Edge or embedded deployment | NPU or integrated accelerator | Power and thermal budgets rule out discrete accelerators | A site with real power and cooling available |
Defaults for orientation. The actual decision depends on model architecture, volume, latency requirement, existing skills and what capacity can actually be obtained — usually in that order of underestimation.
Key Takeaways
- CPU, GPU and TPU-class accelerators are points on one spectrum from general-purpose to fixed-function; every other difference follows from where a device sits on it.
- The CPU is optimised for latency on branch-heavy work, the GPU for parallel throughput, and a domain-specific accelerator for efficiency on one narrow arithmetic pattern.
- The GPU is the default not because it is the most efficient device for any given model, but because the software ecosystem is built against it.
- A TPU-class accelerator pays off when the workload is stable, large and continuously served; the same properties make it a poor fit for anything still changing.
- The CPU never leaves the critical path — it loads, preprocesses, schedules and checkpoints, and an under-specified host presents as poor accelerator utilisation.
- Peak throughput figures across architectures are rarely comparable: check precision, sparsity assumptions, scope and batch size before believing any of them.
- Specialisation is a commercial decision as much as a technical one — it concentrates supply, pricing and roadmap risk in a single provider.