Calculation results
Sample results
| # | Original | Working | Index | Quantized | Error | |Error| | Relative | Error % | Clipped | Binary | Hex |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | -0.8200000 | -0.8200000 | 23 | -0.8164063 | 0.0035937 | 0.0035937 | -0.0043826 | -0.4383 | No | 00010111 | 17 |
| 2 | -0.3500000 | -0.3500000 | 83 | -0.3476563 | 0.0023437 | 0.0023437 | -0.0066964 | -0.6696 | No | 01010011 | 53 |
| 3 | 0.0000000 | 0.0000000 | 128 | 0.0039063 | 0.0039063 | 0.0039063 | 0.0000000 | 0.0000 | No | 10000000 | 80 |
| 4 | 0.1300000 | 0.1300000 | 144 | 0.1289063 | -0.0010938 | 0.0010938 | -0.0084135 | -0.8413 | No | 10010000 | 90 |
| 5 | 0.4900000 | 0.4900000 | 190 | 0.4882813 | -0.0017187 | 0.0017187 | -0.0035077 | -0.3508 | No | 10111110 | BE |
| 6 | 0.6300000 | 0.6300000 | 208 | 0.6289063 | -0.0010938 | 0.0010938 | -0.0017361 | -0.1736 | No | 11010000 | D0 |
| 7 | 0.9100000 | 0.9100000 | 244 | 0.9101563 | 0.0001562 | 0.0001562 | 0.0001717 | 0.0172 | No | 11110100 | F4 |
Charts
Original and quantized
Error by sample
Error histogram
Transfer curve
Formula used
How to use this calculator
- Select a single value, dataset, CSV, or waveform.
- Set bits, levels, range, quantizer, rounding, and dither.
- Review live errors, clipping, levels, charts, and statistics.
- Use comparison, fixed-point, ADC, image, and audio laboratories.
- Export CSV, JSON, charts, images, audio, or PDF reports.
Bit-depth comparison
| Bits | Mode | Levels | Step | MSE | RMSE | SQNR | Theoretical | ENOB | Storage |
|---|
Fixed-point Q-format
Floating-point approximation
ADC code converter
Image quantization
Original
Quantized
Audio quantization
Quantization guide
What quantization means
Quantization maps continuous or high-resolution values onto finite reconstruction levels. Sampling determines when measurements occur, while quantization determines which discrete number represents each measurement.
Uniform and non-uniform modes
Uniform quantizers use equal steps. Logarithmic, μ-law, and A-law modes allocate more detail near lower amplitudes. Mid-rise places zero on a threshold. Mid-tread includes zero as an output level.
Error, noise, and clipping
Ideal rounding error often stays near one-half step without overload. Clipping is separate overload distortion. Dither can reduce correlated artifacts while increasing broadband noise.
Bit depth and ENOB
Each additional ideal bit doubles level count. Full-scale sine SQNR rises about 6.02 dB per bit. Real systems lose performance through nonlinearity, jitter, thermal noise, and reference errors.
Fixed point, images, and audio
Fixed-point formats divide words into integer and fractional positions. Image quantization reduces colors or channel precision. Audio quantization reduces amplitude precision and often benefits from carefully chosen dither.
Example data table
| Scenario | Range | Bits | Levels | Approximate step | Use |
|---|---|---|---|---|---|
| Audio PCM | −1 to +1 | 16 | 65,536 | 0.0000305 | Recording |
| ADC | 0 to 3.3 V | 12 | 4,096 | 0.000806 V | Sensors |
| Neural tensor | −1 to +1 | 8 | 256 | 0.0078125 | Inference |
| Grayscale | 0 to 255 | 4 | 16 | 15.9375 | Displays |
Frequently asked questions
What is quantization error?
The reconstructed output minus the original input.
Is quantization error random?
Not always. Deterministic signals can create correlated error.
Why is clipping important?
Clipping is overload distortion and can dominate normal quantization noise.
What is one LSB?
The smallest nominal code step for the selected range and resolution.
Why use dither?
It reduces structured distortion by decorrelating error.
What is ENOB?
An equivalent ideal bit depth derived from measured performance.
Does one extra bit double accuracy?
It doubles levels and usually halves ideal step size.
Mid-rise versus mid-tread?
Mid-rise has a zero threshold. Mid-tread has a zero reconstruction level.
Can this replace calibration?
No. Critical systems require validated measurements and calibration.