LLM API Request Batching Throughput vs Latency Trade-off Calculator
Evaluate the LLM inference trade-off between batch size, system throughput (tokens/sec), and request latency. Larger batches improve GPU utilization and tokens per second up to the hardware saturation knee, but increase per-request end-to-end latency. Note: The calculator uses simplified heuristic models for demonstration purposes as exact latency depends on specific hardware, model architecture, and KV cache management.
Interactive Parameters & Live Verdict
Rule Diagnostics & Recommendations
Engineering Formula & Verification Notes
This calculator models the core LLM inference trade-off between compute efficiency and queuing/decoding delay. Prefill phase merges matrix multiplications across batched prompts, while decode phase shares GPU memory bandwidth across concurrent sequences. A scaling dampener accounts for KV-cache contention and scheduler overhead as batch sizes scale.
Worked Engineering Example
Consider an MPT-7B model deployment processing 512 input tokens and 128 output tokens per request. At a single-request speed of 45 tokens/sec and a batch size of 16, the system merges prompt prefill into larger matrix multiplications, elevating aggregate token throughput while keeping per-request latency within acceptable interactive bounds.
- Batch size scaling increases arithmetic intensity and GPU utilization.
- Decoding overhead grows as every decode step iterates over active sequences in the batch.
Common Sizing & Compatibility Mistakes
Engineers often scale batch sizes infinitely assuming linear throughput gains. In practice, once memory bandwidth saturates and execution shifts to compute or memory bound bottlenecks, latency climbs exponentially while token throughput flattens.
- Failing to account for KV cache memory limits leading to sudden OOM crashes.
- Using high batch sizes for real-time chat applications where low TTFT and token generation speed are critical.
Inference Batching Benchmarks (A100 Reference)
| Deployment Tier | Batch Size | Throughput Multiplier |
|---|---|---|
| Real-time Chat | 1 to 4 | 1x (Baseline Latency) |
| Balanced Service | 16 to 32 | 4x - 8x Throughput |
| Batch Pipeline | 64 to 128 | 12x - 14x Throughput |
Frequently Asked Technical Questions
Why does increasing batch size increase request latency?
During the decode phase, the model streams weights from memory for every token step. All sequences in the active batch share this memory bandwidth, meaning tokens arrive progressively slower as concurrency increases.
What is the knee of the throughput-latency curve?
The knee represents the optimal operational point where GPU utilization is maximized before memory bandwidth saturation causes latency to climb faster than throughput gains.
How does prompt length affect prefill latency?
Longer prompts require larger matrix multiplications during the prefill phase, which increases Time to First Token (TTFT) regardless of batch size.