DDR5 / Unified Memory Bandwidth & Local LLM Tokens/Sec Estimator
Autoregressive LLM token generation is memory-bandwidth bound: dividing effective memory bandwidth (e.g., 89.6 GB/s on dual-channel DDR5-5600 or 400 GB/s on M3/M4 Max) by active model weight size in GB yields a tight upper bound of ~1.8 to 8+ tokens/sec for 70B Q4 models.
Interactive Parameters & Live Verdict
Rule Diagnostics & Recommendations
Engineering Formula & Verification Notes
During single-batch decode, every active parameter weight must be read from memory once per generated token. Practical speed equals (Memory Bandwidth × Efficiency Factor) / Active Weight Footprint in GB.
Worked Example: MoE vs Dense Models on CPU/Unified Memory
A 70B dense model in Q4_K_M reads ~42 GB per token, yielding only ~1.6 tok/s on 96 GB/s DDR5. A Mixture-of-Experts model with 30B total parameters but only 3B active parameters reads ~1.8 GB per token, achieving ~38 tok/s on the exact same memory bus.
- Always use active parameters—not total parameters—for MoE token speed estimation.
- Prompt evaluation (prefill) is compute-bound, while generation (decode) is bandwidth-bound.
Common Bandwidth Estimation Mistakes
DDR5 rated MT/s often misleads builders who run single-channel DIMMs or ignore memory controller efficiency.
- Running 1 DIMM (single-channel) cuts memory bandwidth in half.
- Expecting 100% theoretical peak bandwidth; real LLM kernels typically achieve 65%–80%.
Decode Speed Tiers by Memory Architecture (8B Q4_K_M ~4.8 GB)
| Platform | Bandwidth | Est. 8B Q4 Speed |
|---|---|---|
| Dual-Channel DDR5-6000 | ~96 GB/s | ~14 tok/s |
| Apple M4 Pro Unified | ~273 GB/s | ~41 tok/s |
| RTX 4090 GDDR6X | ~1,008 GB/s | ~145+ tok/s |
Frequently Asked Technical Questions
Why does memory bandwidth matter more than core count for local LLM chat?
Batch-size-1 token decode performs very little arithmetic per byte loaded, making memory bus speed the primary bottleneck.
Do 4 DDR5 sticks run faster than 2 sticks on consumer CPUs?
No—consumer desktop CPUs are still dual-channel, and 4 DIMMs often force lower DDR5 clock speeds due to signal integrity.
How fast is considered comfortable for interactive chat?
10–15 tokens/sec matches comfortable human reading speed.