LLM Speculative Decoding Speedup & Draft Model Efficiency Calculator
Speculative decoding speedup is calculated as: Speedup = 1 + E[# accepted tokens] / (1 + Cd/Cv), where Cd is drafting cost and Cv is verification cost. Speedup depends critically on acceptance rate (α ≥ 0.6 recommended for 2x+ gains), draft model efficiency, and batch size. Gains diminish at high batch sizes where GPU utilization is saturated. Small models (<3B) often experience slowdowns due to high relative drafting overhead.
Interactive Parameters & Live Verdict
Rule Diagnostics & Recommendations
Engineering Formula & Verification Notes
Calculated using the ratio of total time per cycle: (1 + E[accepted]) / (1 + Cd/Cv). Assumes verification cost is dominated by a single forward pass of the target model.
Worked Engineering Example
For a 7B target model with 25ms verification and a draft model taking 15ms to generate 2.5 tokens:
- Drafting Cost (Cd) = 15ms
- Verification Cost (Cv) = 25ms
- Speedup = (1 + 2.5) / (1 + 15/25) = 3.5 / 1.6 = 2.18x
Common Sizing & Compatibility Mistakes
Engineers often overlook the impact of batch size on hardware utilization.
- Using speculative decoding on small models (< 3B) often leads to slowdowns due to high relative drafting overhead.
- Ignoring KV cache pressure when increasing draft lengths.
- Assuming speedup scales linearly with batch size; it actually plateaus as GPU compute becomes saturated.
Reference Benchmarks
| Model Size | Typical Speedup | Bottleneck |
|---|---|---|
| Small (<3B) | 0.7x - 1.2x | Drafting Overhead |
| Medium (7B-14B) | 1.5x - 2.5x | Acceptance Rate |
| Large (>30B) | 2.0x - 3.5x | Memory Bandwidth |
Frequently Asked Technical Questions
Why does speedup drop at high batch sizes?
At high concurrency, the GPU is already compute-bound; the extra operations from the draft model compete for cycles, reducing throughput.
What is the ideal acceptance rate?
An acceptance rate (α) ≥ 0.6 is generally required to achieve consistent 2x+ speedups.
Does draft model quality matter?
Yes, the draft model distribution must closely match the target model to ensure high acceptance rates.