What model inference latency means
Inference latency is the elapsed time between receiving a request and delivering a usable result. End-to-end latency includes more than accelerator compute. It may include validation, tokenization, network transfer, queueing, model execution, post-processing, serialization, and client rendering.
For generative systems, users often experience two distinct delays. Time to first token measures responsiveness before streaming begins. Inter-token latency determines how smoothly text, audio, or another stream continues after the first result.
Formula used
The weighted cold-start contribution equals cold-start duration multiplied by cold-start probability. Sequential custom stages are added directly. Parallel stages in the same group contribute only their slowest stage.
Effective output tokens reflect early stopping. Decode overhead varies by decoding method, beam width, and structured-output constraints. Speculative decoding reduces decode time according to draft length and acceptance rate.
The queue model uses an Erlang-C approximation. It assumes approximately Poisson arrivals, exponential service times, and identical workers. Real systems may differ because requests have variable sequence lengths and scheduler behavior.
How to use this calculator
1. Choose a workload mode
Select the mode matching your application. Language-model mode separates prefill and decoding. Vision, speech, embedding, and batch modes emphasize compute and transfer estimates. Custom pipeline mode adds user-defined sequential and parallel stages.
2. Select model and hardware presets
Presets provide useful starting points, not vendor benchmarks. Replace all values with measured specifications whenever possible. Active parameters matter for sparse mixture-of-experts models because only selected experts process each token.
3. Enter throughput benchmarks
Prefill tokens per second and decode tokens per second should come from the same runtime, precision, batch size, and hardware configuration. Mixing values from different tests creates misleading results. Use production-like prompt lengths and output lengths.
4. Model concurrency and queueing
Enter the request arrival rate, workers, replicas, and devices. The calculator estimates queue utilization and waiting time. Capacity becomes unstable when demand reaches or exceeds aggregate service capacity.
5. Review memory
Memory includes weights, activations, KV cache, buffers, framework overhead, and accelerator runtime overhead. The KV-cache coefficient is intentionally configurable. Calibrate it from your architecture or profiler because implementations differ significantly.
6. Inspect percentiles and bottlenecks
Mean latency can hide poor user experiences. Review P95 and P99 estimates alongside the SLO threshold. The bottleneck card identifies the largest modeled stage, while recommendations suggest practical experiments.
Latency versus throughput
Latency measures the time for one request. Throughput measures completed work per unit time. Larger batches may improve throughput while increasing per-request waiting and processing latency. Production systems balance both metrics against cost and service objectives.
Prefill versus decoding
Prefill processes the full input prompt and can exploit parallel computation. Decoding generates tokens iteratively and is often limited by memory bandwidth. Long prompts increase prefill work and KV-cache memory. Long outputs increase total decode time.
P50 versus P99 latency
P50 describes the median experience. P99 describes a slow tail encountered by roughly one percent of requests. Tail latency grows with queue saturation, cold starts, retries, contention, variable sequence lengths, paging, and noisy shared infrastructure.
Cold versus warm starts
A warm worker already has its runtime, model weights, kernels, and caches ready. A cold worker may download artifacts, start a container, initialize frameworks, load weights, compile graphs, and initialize caches. Even rare cold starts can affect high-percentile latency.
How batching changes performance
Batching amortizes launch and scheduling overhead across requests. Dynamic batching waits briefly to collect compatible requests. Continuous batching inserts new work as existing sequences finish. Padding and sequence-length variation reduce efficiency when requests differ greatly.
How quantization changes performance
Lower precision reduces weight memory and memory traffic. It may also increase tensor-core throughput. Real gains depend on kernel availability, dequantization overhead, group size, calibration, and model architecture. Accuracy effects must be measured separately.
Common inference bottlenecks
Compute-bound workloads benefit from greater arithmetic throughput and optimized kernels. Memory-bound workloads benefit from quantization, faster memory, cache improvements, and reduced model size. Network-bound workloads benefit from regional placement, compression, streaming, and connection reuse.
Queue-bound systems need more capacity, better admission control, or reduced service time. Cold-start-bound systems need warm pools and slower scale-to-zero policies. Pipeline-bound systems need parallel execution, caching, or removal of redundant stages.
Worked example
Consider a seven-billion-parameter language model on one high-memory GPU. The request contains 1,024 input tokens and requests 256 output tokens. Prefill runs at 4,500 tokens per second, while decoding runs at 85 tokens per second.
Network and fixed overhead remain modest. Decode time dominates because generation is sequential. Enabling speculative decoding, limiting output length, or using a faster decoding runtime may improve the result more than reducing tokenization time.
Frequently asked questions
Is this a benchmark?
No. It is an analytical estimator. Use benchmark samples to calibrate the inputs and compare the modeled output with observed production behavior.
Why can TTFT be smaller than total latency?
TTFT ends when the first streamed token becomes available. Total latency includes all remaining generated tokens, final post-processing, serialization, network delivery, and client rendering.
Why does P99 rise near full utilization?
Small service-time variations create queues when workers remain busy. As utilization approaches one hundred percent, queue waiting can increase nonlinearly.
Does adding GPUs always reduce latency?
No. Multi-device inference adds communication and synchronization. It helps when one device lacks memory or when parallelism gains exceed interconnect overhead.
What is a good latency target?
The target depends on the application. Interactive search often needs fast responses. Long-form generation may tolerate higher total latency when TTFT and streaming remain responsive.
How should measured samples be entered?
Paste latency values in milliseconds. Use data from representative traffic, prompt lengths, batches, and regions. Avoid mixing cold and warm samples unless that mixture reflects production.
Can this model image or speech inference?
Yes. Select the relevant mode and enter compute, image resolution, or audio duration. Replace token-specific defaults with values appropriate for the workload.
What does memory utilization include?
It includes estimated weights, activations, KV cache, buffers, framework overhead, and accelerator overhead. It does not guarantee fragmentation-free allocation.
How is cost per request estimated?
The calculator allocates hourly hardware cost across modeled service time and batch size. Optional token prices are then added for external API or accounting scenarios.
Can I export the results?
Yes. Download CSV, export JSON, copy the summary, or use the print dialog to save a PDF report.
Why are presets generic?
Hardware and model performance changes quickly. Generic presets avoid claiming vendor-specific benchmarks and encourage users to enter measured values.