DDR5 / Unified Memory Bandwidth & Local LLM Tokens/Sec Estimator

Verified Nov 14, 2026 Quality 100/100 Confidence 96% 0 Evaluations
VERIFIED QUICK ANSWER

Autoregressive LLM token generation is memory-bandwidth bound: dividing effective memory bandwidth (e.g., 89.6 GB/s on dual-channel DDR5-5600 or 400 GB/s on M3/M4 Max) by active model weight size in GB yields a tight upper bound of ~1.8 to 8+ tokens/sec for 70B Q4 models.

Interactive Parameters & Live Verdict

Rule Diagnostics & Recommendations

    Engineering Formula & Verification Notes

    During single-batch decode, every active parameter weight must be read from memory once per generated token. Practical speed equals (Memory Bandwidth × Efficiency Factor) / Active Weight Footprint in GB.

    Worked Example: MoE vs Dense Models on CPU/Unified Memory

    A 70B dense model in Q4_K_M reads ~42 GB per token, yielding only ~1.6 tok/s on 96 GB/s DDR5. A Mixture-of-Experts model with 30B total parameters but only 3B active parameters reads ~1.8 GB per token, achieving ~38 tok/s on the exact same memory bus.

    • Always use active parameters—not total parameters—for MoE token speed estimation.
    • Prompt evaluation (prefill) is compute-bound, while generation (decode) is bandwidth-bound.

    Common Bandwidth Estimation Mistakes

    DDR5 rated MT/s often misleads builders who run single-channel DIMMs or ignore memory controller efficiency.

    • Running 1 DIMM (single-channel) cuts memory bandwidth in half.
    • Expecting 100% theoretical peak bandwidth; real LLM kernels typically achieve 65%–80%.

    Decode Speed Tiers by Memory Architecture (8B Q4_K_M ~4.8 GB)

    Decode Speed Tiers by Memory Architecture (8B Q4_K_M ~4.8 GB)
    Platform Bandwidth Est. 8B Q4 Speed
    Dual-Channel DDR5-6000 ~96 GB/s ~14 tok/s
    Apple M4 Pro Unified ~273 GB/s ~41 tok/s
    RTX 4090 GDDR6X ~1,008 GB/s ~145+ tok/s

    Frequently Asked Technical Questions

    Why does memory bandwidth matter more than core count for local LLM chat?

    Batch-size-1 token decode performs very little arithmetic per byte loaded, making memory bus speed the primary bottleneck.

    Do 4 DDR5 sticks run faster than 2 sticks on consumer CPUs?

    No—consumer desktop CPUs are still dual-channel, and 4 DIMMs often force lower DDR5 clock speeds due to signal integrity.

    How fast is considered comfortable for interactive chat?

    10–15 tokens/sec matches comfortable human reading speed.

    Did this interactive specification answer your scenario accurately?