Formula Used
Raw throughput: work units ÷ total elapsed seconds.
Effective throughput: useful work units ÷ total elapsed seconds.
Scaling efficiency: measured total throughput ÷ ideal throughput × 100.
Latency capacity: concurrent requests ÷ average latency in seconds.
How to Use
Select the workload and throughput unit. Enter completed work and elapsed time. Add optional hardware, latency, training, token, energy, and benchmark details.
Choose a preset for faster entry. Review every populated metric after calculation. Save configurations to compare performance across systems.
Example Data
| Scenario | Work completed | Elapsed time | Batch | Devices | Expected base result |
|---|---|---|---|---|---|
| Image inference | 10,000 images | 50 seconds | 32 | 1 GPU | 200 images/s |
| API serving | 60,000 requests | 300 seconds | 1 | 2 GPUs | 200 requests/s |
| LLM generation | 500,000 tokens | 600 seconds | 8 | 4 GPUs | 833.33 tokens/s |
| Training | 1,280,000 samples | 3,200 seconds | 256 | 8 GPUs | 400 samples/s |
Understanding the Results
Throughput measures completed work during a fixed period. Latency measures time required for one request. Increasing batch size may raise throughput while increasing latency.
Raw throughput includes every completed unit. Effective throughput discounts work marked unusable or wasted. Active throughput isolates the model execution period.
Scaling efficiency compares measured performance with ideal linear scaling. Lower efficiency can indicate communication or synchronisation overhead. Benchmark identical workloads before choosing production hardware.
Common Bottlenecks
Slow storage can starve accelerators during training. Network transfer can limit distributed model scaling. Excessive queueing can hide otherwise fast inference.
Small batches may underuse available parallel hardware. Very large batches can exceed memory capacity. Profile each stage before buying additional devices.
Frequently Asked Questions
What is model throughput?
Model throughput is completed work per time unit. Common units include samples, requests, images, frames, batches, or tokens. Higher values indicate more processing capacity.
How is throughput different from latency?
Latency measures one operation's response time. Throughput measures total completed work over time. A system can improve throughput while latency increases.
Should warm-up time be included?
Include warm-up for end-to-end operational measurements. Exclude it for steady-state kernel benchmarks. Report both methods when comparing published results.
Why does batch size affect throughput?
Larger batches improve accelerator parallelism and amortise overhead. Memory limits eventually reduce practical batch growth. Measure latency and throughput together.
What is effective throughput?
Effective throughput counts only useful completed work. Failed, discarded, or low-quality outputs can be excluded. This better represents delivered system value.
How do I calculate multi-device efficiency?
Multiply single-device throughput by the device count. Divide measured total throughput by that ideal value. Multiply the result by one hundred.
What repeated runs should I use?
Run identical benchmarks after stable warm-up. Collect enough runs to expose normal variation. Report central values and percentile results.
Can tokens per second compare language models?
Tokens per second helps compare similar generation workloads. Prompt length and output length strongly affect results. Use matching settings for fair comparisons.
How can I estimate required devices?
Divide target throughput by measured single-device throughput. Round the result upward to a whole device. Then adjust for expected scaling losses.
Does higher throughput always mean better performance?
No single metric defines overall system quality. Cost, latency, accuracy, energy, and reliability also matter. Choose metrics aligned with production requirements.