What this article is
This is a commentary on a benchmark that Christina Sørensen (Lead SWE at Vitvio, NixOS Steering Committee) published, in which she ran Gemma 3 inference on a 2016 Intel Xeon E5-2620 v4. The original write-up is the source link above — read it first; this piece only summarizes the takeaway for engineering leads thinking about cheap CPU inference.
The machine
- CPU: Intel Xeon E5-2620 v4 (Broadwell-EP, 2016) — 8 cores / 16 threads, base 2.1 GHz, AVX2.
- RAM: DDR4 ECC (the v4 generation moved Xeon to DDR4; the v3 was the last DDR3 family). The board's bandwidth is the relevant figure, not the size.
- GPU: none.
On paper this looks ten years out of date. In practice, for a single-user chat at home or in a homelab, it is fine for small-to-mid models.
Why it works at all: the memory wall
LLM inference on CPU is not bottlenecked by ALU throughput. It is bottlenecked by memory bandwidth — every generated token requires the model weights to flow from RAM through the cache hierarchy to the cores. The Xeon E5-2620 v4 has four DDR4 channels; that is enough to keep the cores fed for a 2–7B-parameter quantized model.
This is also why fancier modern CPUs do not always win by the factor their TDP suggests for this workload — they hit the same memory wall.
What Christina's run actually shows
The useful claim is not "old hardware is as good as new hardware." It is narrower and more interesting:
- Quantization changes the game: with a 4-bit quantized Gemma 3, the working set fits in a few GB, and DDR4 bandwidth on a 2016 board is enough to produce a steady stream of tokens.
llama.cppis CPU-first: contrary to a common misconception,llama.cppwas built to run on CPU and treats GPU acceleration as an optional path. It is the right tool for this kind of hardware.- Throughput is single-user-grade: the run targets one concurrent generation, not a production multi-tenant service. Throughput per second per dollar still favours modern GPUs for serving many users.
When this matters for a team
If you are a startup, you do not need an H100 to prototype an LLM feature. A spare CPU box — even a decade old — is enough to validate prompts, schema, and latency targets locally before any cloud spend. Once you need to serve real users, the calculus flips to GPUs. Use the cheap path during the part of the project where being wrong is normal; pay for the fast path once the question is no longer whether but how much.
Read the original
This is a summary. The actual measurements, the OS configuration, and the runtime details live in Christina's post. Open the source link above and read it; she does the work, we are just pointing at it.