Formula Used
Validation size ≈ N ÷ K Training size = N − validation size Model fits = K × repetitions Search fits = combinations × K × repetitions × models Standard error = sample standard deviation ÷ √fold count Confidence interval = mean ± critical value × standard error
How to Use the Calculator
- Enter the dataset size and feature count.
- Select the machine learning task.
- Choose the appropriate validation method.
- Set K, repetitions, and randomisation options.
- Add class, group, or time-series settings.
- Enter training and validation fold scores.
- Configure search cost and nested validation.
- Submit the form and review warnings.
- Copy, print, or export the results.
Worked Example
A dataset contains 1,000 observations. Five-fold validation creates five validation segments. Each segment contains about 200 observations.
Each model trains on about 800 observations. It validates on the remaining 200 observations. Every observation becomes validation data once.
With three repetitions, fifteen model fits are required. Score variation then reveals stability. Large gaps may indicate overfitting.
| Input | Example value | Meaning |
|---|---|---|
| Samples | 1,000 | Total observations |
| K | 5 | Five validation folds |
| Repetitions | 3 | Three full cross-validation cycles |
| Total fits | 15 | Five folds times three repetitions |
| Average validation size | 200 | One fifth of the dataset |
Choosing a Validation Strategy
| Method | Best use | Main caution |
|---|---|---|
| Standard K-Fold | Independent, balanced observations | Class proportions may vary |
| Stratified K-Fold | Classification with uneven classes | Each class needs enough samples |
| Group K-Fold | Patients, users, sites, or subjects | Groups must never cross folds |
| Repeated K-Fold | Score stability analysis | Runtime grows quickly |
| Time Series Split | Chronological observations | Never leak future information |
| Nested Cross-Validation | Unbiased model selection | Computationally expensive |
Common Mistakes
- Preprocessing the complete dataset before splitting.
- Using standard folds for highly imbalanced classes.
- Allowing one group to appear in multiple folds.
- Shuffling observations in a chronological dataset.
- Choosing more folds than minority-class samples.
- Reporting only the best fold score.
- Ignoring score variance and confidence intervals.
- Tuning hyperparameters on the outer test folds.
- Comparing models with inconsistent fold assignments.
Frequently Asked Questions
What does K mean?
K is the number of folds. Each fold becomes validation data once. The remaining folds become training data.
Should I use five or ten folds?
Five folds often balance speed and reliability. Ten folds may help smaller datasets. Compare stability before choosing.
When should I use stratification?
Use stratification for classification problems. It preserves class proportions in each fold. This improves minority representation.
What is repeated cross-validation?
Repeated validation runs several fold assignments. It measures score stability more thoroughly. It also increases computation.
Why use Group K-Fold?
Related observations must stay together. Group splitting prevents identity leakage. This is common with patients or users.
Can time-series data use normal K-Fold?
Usually, it should not. Future observations can leak into training. Use chronological windows instead.
What does a large score deviation mean?
Large deviation indicates unstable performance. The model may depend on specific samples. Repeated validation can investigate this.
What is nested cross-validation?
Nested validation separates tuning from evaluation. Inner folds select parameters. Outer folds estimate final performance.
Does cross-validation replace a test set?
Not always. A final untouched test set remains valuable. It provides one independent performance estimate.