If you want to understand GPU for LLMs - why every AI company is spending billions on graphics cards to run large language models - the answer comes down to two things: parallel math and memory. This post explains both, without the handwaving.
What is a CPU, and what's it good at?
A CPU - Central Processing Unit - is the general-purpose brain of a computer. A modern CPU has between 8 and 128 cores. Each core is exceptionally powerful: it handles complex logic, branching decisions, memory management, and operating system tasks. It's built to move through one complicated problem at a time, very quickly.
Think of a CPU as a small team of elite surgeons. Each one can handle almost any case you throw at them. But there are only a handful of them, and they work sequentially on each patient.
What is a GPU, and why was it built?
A GPU - Graphics Processing Unit - was originally designed to render video game frames: millions of pixels, each one needing its colour and brightness computed 60 times per second. The key insight engineers had in the 1990s was that pixel calculations are mostly independent of each other. You don't need to know pixel A to compute pixel B.
So instead of a few powerful cores, GPUs are built with thousands of smaller, simpler cores - all running at the same time. A modern NVIDIA H100 GPU has over 16,000 CUDA cores. Each core is less capable than a CPU core, but together they can process thousands of operations in parallel.
Back to the analogy: if a CPU is a small team of elite surgeons, a GPU is a stadium of medical students - each one performing a simple task simultaneously. For the right kind of work, the stadium wins decisively. And running GPU for LLMs is exactly that kind of work.
CPU vs GPU architecture. CPUs have a handful of powerful cores built for complex, sequential tasks. GPUs have thousands of simpler cores designed to run the same operation across massive datasets simultaneously - which is what makes GPU for LLMs so effective.
Why GPU for LLMs comes down to matrix multiplication
A large language model like GPT-4 or Claude isn't doing mystical reasoning. Under the hood, it's performing an enormous number of matrix multiplications: taking a grid of numbers, multiplying it by another grid of numbers, and producing a result. This happens billions of times per forward pass - once for every token the model generates.
Matrix multiplication is a perfect fit for parallel computation. Each cell in the output matrix is computed independently from the others. You don't need cell (2,3) to compute cell (5,7). They can happen at exactly the same time - on thousands of GPU cores simultaneously. This is the core reason GPU for LLMs is not just a preference but a practical necessity at scale.
On a CPU, those calculations happen one after another. On a GPU, they happen all at once. For a model with 70 billion parameters generating a single token, the GPU completes in milliseconds what would take a CPU several seconds.
Matrix multiplication: sequential vs parallel. In a real LLM, this matrix has billions of values. The GPU's ability to compute every cell simultaneously is why GPU for LLMs produces responses in milliseconds rather than minutes.
VRAM: the GPU for LLMs bottleneck nobody talks about
Speed is only half the story. The other hard constraint when using a GPU for LLMs is memory - specifically, VRAM (Video RAM), the memory built directly into the GPU chip.
Before a model can run, its weights must be loaded into VRAM. A 7-billion-parameter model quantised to 4-bit precision takes roughly 4 GB of VRAM. A 70-billion-parameter model in the same format needs around 40 GB. The NVIDIA A100 has 80 GB of VRAM. A consumer RTX 4090 has 24 GB.
If the model doesn't fit in VRAM, two things happen: either the software offloads layers to CPU RAM (dramatically slower), or the model simply refuses to load. This is why "does it fit?" is often the first question engineers ask when evaluating a GPU for LLM deployments.
LLM VRAM requirements vs GPU capacity. The dashed lines mark the VRAM ceiling of two common GPUs. Any model above the line requires multiple GPUs, CPU offloading, or cloud infrastructure.
GPU for LLMs in practice: what the numbers mean
The GPU advantage for LLM inference is not marginal. It shapes every real production decision:
| Concern | CPU | GPU |
|---|---|---|
| Inference speed | Seconds per token | Milliseconds per token |
| Throughput | 1–2 concurrent users | Dozens to hundreds |
| Cost at scale | High (CPU time) | Low per token (amortised) |
| Large model support | Possible, very slow | Required for viable speed |
| Small models locally | Acceptable | Faster but not always necessary |
Running a 7B model on a CPU for personal experimentation? Acceptable. Running a 70B model in production serving hundreds of users simultaneously on CPU? The latency would make the product unusable. This is why teams evaluating GPU for LLMs in production almost never go back to CPU-only inference once they've seen the difference.
When does a CPU still win over a GPU for LLMs?
Not every LLM deployment needs a GPU. There are legitimate cases where CPU inference is the right call:
- Edge deployments - devices like phones or embedded systems where no GPU is available.
- Small, quantised models - a 1–3B model at 4-bit runs acceptably on a modern laptop CPU for low-traffic use cases.
- Cost-sensitivity at low volume - if you're running five queries per hour, you don't need a $20,000 H100.
- Pre/post-processing pipelines - everything outside the model inference step (parsing, routing, logging) runs fine on CPU.
The full GPU for LLMs inference pipeline
To tie it together, here's what happens every time a user sends a prompt to an LLM running on GPU infrastructure:
GPU for LLMs: the inference pipeline. Every token requires a full forward pass through all model layers on the GPU. The loop repeats until the model produces an end-of-sequence token or hits the output limit.
Key takeaways
- GPU for LLMs works because LLMs are a parallel math problem. Billions of matrix multiplications per token run simultaneously across thousands of GPU cores - something a CPU simply cannot match in throughput.
- VRAM is the constraint that actually limits what you can run. A GPU with fast compute but insufficient VRAM cannot serve large models at useful speeds. Model size (in GB) must fit within the GPU's VRAM capacity.
- Speed and throughput are the practical stakes. CPU inference is possible but impractically slow for most production use cases above 7B parameters. GPU inference turns seconds into milliseconds and serves orders of magnitude more concurrent users.
Frequently asked questions about GPU for LLMs
Can I run an LLM without a GPU?
Yes, but with significant tradeoffs. Small models (1–7B parameters) at 4-bit quantisation can run on a modern CPU at speeds usable for personal experimentation. For production deployments or models larger than 7B, CPU inference is typically too slow. See the llama.cpp project for the current state of the art in CPU-based LLM inference.
What GPU do I need to run Llama 3 70B locally?
A 70B model at 4-bit quantisation requires approximately 40 GB of VRAM. That exceeds the 24 GB in a single consumer RTX 4090, so you would typically need either a high-end workstation GPU (like an A100 or H100), multiple consumer GPUs running in tandem, or a cloud GPU instance.
What is the difference between GPU VRAM and regular RAM?
VRAM is memory physically located on the GPU itself, with extremely high bandwidth designed to feed thousands of parallel cores simultaneously. Regular system RAM communicates over a much slower bus. When using GPU for LLMs, model weights must live in VRAM during inference - moving them back and forth from system RAM is a major performance bottleneck.
Why do cloud providers charge so much for GPU instances?
High-end AI GPUs like the NVIDIA H100 cost USD $25,000–$40,000 each at retail. Data centres run them at high power draw, require specialised cooling, and face constrained supply. The per-hour cost reflects the capital expense and scarcity of the hardware - not a margin decision.
Will CPUs ever be fast enough to replace GPUs for LLM inference?
Unlikely in the near term. The architectural gap is intentional - CPUs prioritise sequential task flexibility, GPUs prioritise parallel throughput. Some specialised chips (like Apple's unified memory architecture or purpose-built AI accelerators) are narrowing the gap for specific model sizes, but the fundamental parallel nature of GPU for LLMs inference continues to favour massively parallel hardware.
