Coverage Results
Results use the current preprocessing and vocabulary settings.
Target and Recommendation
Interpretation
Frequency Ranking Table
Search tokens and inspect coverage contribution.
| Rank | Token | Frequency | Relative % | Cumulative | Token coverage | Type coverage | Document frequency | Status |
|---|
Coverage Curve Data
| Vocabulary size | Covered tokens | Uncovered tokens | Coverage % | OOV % | Marginal gain | Frequency cutoff | Retained types |
|---|
Formula Used
Token Coverage (%) = Covered Token Count ÷ Total Token Count × 100 Type Coverage (%) = Included Unique Terms ÷ Total Unique Terms × 100 OOV Rate (%) = OOV Token Count ÷ Total Token Count × 100 Marginal Gain = Current Coverage − Previous Coverage
How to Use
Paste a corpus or load an example preset. Select preprocessing and vocabulary controls. Press calculate to generate the coverage analysis.
Use the target percentage to estimate vocabulary size. Compare training, validation, and test curves. Export tables, vocabulary, charts, or a PDF.
Worked Example
| Rank | Token | Frequency | Cumulative tokens | Coverage |
|---|---|---|---|---|
| 1 | data | 40 | 40 | 40% |
| 2 | model | 30 | 70 | 70% |
| 3 | learning | 20 | 90 | 90% |
| 4 | rare | 10 | 100 | 100% |
Frequently Asked Questions
What is vocabulary coverage?
Vocabulary coverage measures represented token occurrences. Frequent words usually dominate early ranks. Rare words extend the curve slowly.
What is token coverage?
Token coverage weights repeated occurrences. Common terms contribute more than rare terms. It suits model input planning.
What is type coverage?
Type coverage measures unique vocabulary inclusion. Every unique term has equal weight. It rises linearly by rank.
What does OOV mean?
OOV means out of vocabulary. These tokens are not directly represented. They may map to an unknown token.
Why lowercase text?
Lowercasing merges capitalization variants. It reduces vocabulary size quickly. Proper names may lose distinctions.
Why remove stop words?
Stop words are extremely frequent. Removing them changes coverage substantially. Use this option carefully.
What is the elbow point?
The elbow marks diminishing returns. Later words add little coverage. It offers a practical vocabulary estimate.
Why compare validation coverage?
Validation text tests generalization. Lower coverage suggests distribution shift. It can expose domain mismatch.
How are duplicates handled?
Duplicate tokens are merged by default. Their frequencies are added together. This prevents fragmented rankings.
Can this replace a production tokenizer?
This calculator provides exploratory analysis. Production tokenizers may behave differently. Validate decisions with deployed tooling.