Permutation Importance Plot Calculator

Calculated Results Test data · Accuracy

Displayed features

Most important

Age

0.0820

Mean importance

0.0333

Median importance

0.0170

Positive / negative

12 / 2

Total absolute

0.4740

Features reaching 80% importance

Features reaching 90% importance

Least important displayed feature

Customer ID

Baseline scores: Production: 0.9100 Challenger: 0.9250

Rank	Feature	Model	Mean	Median	Std. dev.	Std. error	95% CI	Min	Max	Repeats	CV
1	Age	Production	0.0820	0.0820	0.0110	0.0035	0.0752 to 0.0888	0.0820	0.0820	1	0.1341
1	Income	Challenger	0.0790	0.0790	0.0080	0.0025	0.0740 to 0.0840	0.0790	0.0790	1	0.1013
2	Age	Challenger	0.0730	0.0730	0.0100	0.0032	0.0668 to 0.0792	0.0730	0.0730	1	0.1370
2	Income	Production	0.0670	0.0670	0.0090	0.0028	0.0614 to 0.0726	0.0670	0.0670	1	0.1343
3	Account Balance	Challenger	0.0550	0.0550	0.0060	0.0019	0.0513 to 0.0587	0.0550	0.0550	1	0.1091
3	Account Balance	Production	0.0410	0.0410	0.0070	0.0022	0.0367 to 0.0453	0.0410	0.0410	1	0.1707
4	Tenure	Challenger	0.0180	0.0180	0.0040	0.0013	0.0155 to 0.0205	0.0180	0.0180	1	0.2222
4	Region_North	Production	0.0160	0.0160	0.0050	0.0016	0.0129 to 0.0191	0.0160	0.0160	1	0.3125
5	Region_South	Production	0.0120	0.0120	0.0040	0.0013	0.0095 to 0.0145	0.0120	0.0120	1	0.3333
5	Region_North	Challenger	0.0100	0.0100	0.0040	0.0013	0.0075 to 0.0125	0.0100	0.0100	1	0.4000
6	Tenure	Production	0.0090	0.0090	0.0030	0.0009	0.0071 to 0.0109	0.0090	0.0090	1	0.3333
6	Region_South	Challenger	0.0080	0.0080	0.0040	0.0013	0.0055 to 0.0105	0.0080	0.0080	1	0.5000
7	Customer ID	Challenger	-0.0010	-0.0010	0.0020	0.0006	-0.0022 to 0.0002	-0.0010	-0.0010	1	2.0000
7	Customer ID	Production	-0.0030	-0.0030	0.0020	0.0006	-0.0042 to -0.0018	-0.0030	-0.0030	1	0.6667

Interpretation: Positive values show performance loss after shuffling. Near-zero values suggest limited measured contribution. Negative values can indicate noise, redundancy, leakage, or sampling instability. Correlated features may divide importance and permutation importance does not prove causation.

Data Input

Input method

Upload CSV

Maximum file size: 2 MB.

Maximum tested rows

Use zero for every uploaded row.

Paste CSV data

Recognized columns include feature, model, importance, stddev, baseline, permuted, repeats, and group. Separate repeated values with semicolons.

Metric and Permutation Settings

Evaluation metric

Better metric direction

Repeated values represent

Default permutation repeats

Confidence level

Random seed

Sample fraction per repeat

Shuffle method

Evaluation dataset

Bootstrap confidence intervals

Preserve missing-value positions

Use stratified sampling

Feature Selection and Comparison

Search feature names

Maximum displayed features

Minimum absolute importance

Normalization

One-hot prefix delimiter

Models to compare

Challenger

Production

Use group column

Combine one-hot prefixes

Hide zero importance

Hide negative importance

Use absolute values

Chart Customization

Plot type

Sort order

Decimal places

Chart title

X-axis title

Y-axis title

Chart height

Show uncertainty bars

Show values

Show ranks

Show grid lines

Show legend

Show zero reference

Formula Used

For metrics where larger values are better, importance equals the baseline score minus the permuted score. A larger decrease means the model depends more strongly on that feature. Repeated shuffles estimate uncertainty and stability.

Importance(j) = Baseline score − Permuted score(j)

For error metrics where smaller values are better:
Importance(j) = Permuted error(j) − Baseline error

Standard error = Standard deviation ÷ √(number of repeats)
Confidence interval = Mean importance ± z × Standard error

How to Use

Paste CSV data or upload a file.
Select the metric direction and input method.
Choose feature grouping, filtering, and normalization.
Select a Plotly chart and display settings.
Calculate, inspect warnings, then export results.

Example Data Format

Feature	Model	Mean Importance	Standard Deviation
Age	Production	0.082	0.011
Income	Production	0.067	0.009
Account Balance	Production	0.041	0.007
Region	Production	0.014	0.005
Customer ID	Production	-0.003	0.002

Frequently Asked Questions

What does permutation importance measure?

It measures model performance loss after one feature is shuffled. Larger losses usually indicate stronger model dependence.

Why are some importance values negative?

A shuffled feature may improve the measured score by chance. Noise, redundancy, leakage, and sampling variation can cause this.

Should I use training or test data?

Use validation or test data when possible. Training data can overstate importance for an overfitted model.

How many permutation repeats are useful?

Ten repeats provide a practical start. More repeats improve stability but require more computation.

What happens with correlated features?

Correlated features can share or mask importance. Grouped permutation can provide a more realistic combined estimate.

Is permutation importance causal?

No. It explains model dependence, not whether a feature causes the predicted outcome.

What is the difference from impurity importance?

Permutation importance uses score changes on data. Impurity importance comes from tree split statistics and can favor high-cardinality variables.

How are error metrics handled?

Select lower as better. The calculator subtracts baseline error from permuted error so larger importance remains more influential.

Can I compare multiple models?

Yes. Include a model column, select models, and use the grouped comparison plot.