Model Calibration Calculator

1. Analysis mode and data

Classification type

Model names

Binary mode supports several probability columns.

Class names

Used for multiclass probability columns.

Dataset

Binary format: actual, one or more model probabilities, optional weight, group, and ID columns.

Upload CSV

2. Calibration configuration

Binning method

Number of bins

Minimum samples per bin

Custom bin boundaries

Must begin at 0 and end at 1.

Decision threshold

Decimal precision

Confidence level

Confidence interval

Bootstrap samples

Random seed

Probability display

Include empty bins

3. Advanced analysis

Recalibration method

Group column

Weight column

Actual-label column

Observation ID column

Normalise multiclass probabilities

Use sample weights

4. Formulae used

ECE = Σᵦ (nᵦ / N) × |accuracyᵦ − confidenceᵦ|

MCE = maxᵦ |accuracyᵦ − confidenceᵦ|

Brier = (1 / N) × Σᵢ (pᵢ − yᵢ)²

Log loss = −(1 / N) × Σᵢ[yᵢln(pᵢ)+(1−yᵢ)ln(1−pᵢ)]

RMSCE = √(Σᵦ (nᵦ/N) × gapᵦ²)

O:E ratio = observed positive rate ÷ mean predicted probability

How to use

Choose binary or multiclass mode. Paste CSV data with labels and probability columns. Configure bins, confidence intervals, thresholds, and optional recalibration.

Run the analysis and inspect summary metrics. Review reliability diagrams, bin gaps, subgroup behaviour, and discrimination measures. Export the report for documentation.

Use validation or test predictions whenever possible. Fitting recalibration on evaluation data is optimistic. Separate calibration and evaluation datasets improve reliability.

Frequently asked questions

What is a calibrated model?

A calibrated model produces probabilities that match observed frequencies. Among predictions near 0.70, roughly 70% should be positive.

Is lower ECE always better?

Lower ECE usually indicates better calibration. Its value still depends on binning, sample size, and probability distribution.

Can a high-AUC model be poorly calibrated?

Yes. AUC measures ranking quality, while calibration measures probability accuracy. They assess different model properties.

How many bins should I use?

Ten bins are common. Use fewer bins for small datasets and more only when every bin remains sufficiently populated.

What does overconfidence mean?

The model is overconfident when predicted probabilities are more extreme than observed outcome frequencies support.

What does underconfidence mean?

The model is underconfident when predictions remain too close to 0.50 despite stronger observed outcome separation.

Which recalibration method should I choose?

Platt scaling is smooth and stable. Isotonic regression is flexible but needs more data. Temperature scaling is useful for neural networks.

Should calibration use training data?

No. Prefer held-out validation, cross-validation, or external predictions. Training-data calibration estimates are usually too optimistic.

Why analyse subgroups?

Overall calibration can hide poor performance within regions, classes, model versions, or demographic groups.

Run calculator

All calculations run inside your browser. The calculator does not transmit your dataset.

Input format

Binary: actual, Model A, Model B, weight, group, id.

Multiclass: actual, each class probability, weight, group, id.