Formula used
Vocabulary size counts distinct tokens after selected processing. Filtering changes the deployable model vocabulary. Reserved tokens are added when enabled.
How to use the calculator
- Paste training text, upload a corpus, or enter frequencies.
- Select tokenization and preprocessing rules matching your pipeline.
- Set frequency thresholds, top-N limits, and special tokens.
- Optionally paste test text for overlap and OOV analysis.
- Enter Heaps’ Law and embedding assumptions.
- Calculate, inspect charts, then export the required data.
Worked example
Suppose a corpus contains ten token occurrences and six unique tokens. A minimum frequency of two retains three tokens covering seven occurrences. Adding five special tokens creates an effective vocabulary of eight.
| Token | Frequency | Minimum frequency | Status |
|---|---|---|---|
| model | 3 | 2 | Included |
| data | 2 | 2 | Included |
| token | 2 | 2 | Included |
| rare | 1 | 2 | Excluded |
Interpretation guide
A larger vocabulary preserves more lexical detail. It also increases embedding and output-layer memory. Frequency pruning can reduce cost while maintaining strong coverage.
Type-token ratio usually falls as corpora grow. Compare TTR only across similarly sized samples. OOV rate directly measures unseen test-token occurrences.
Word tokenizers remain interpretable but create unknown words. Subword methods usually reduce OOV problems. Character vocabularies are tiny but produce longer sequences.
Common preprocessing mistakes
- Building training and test vocabularies independently.
- Applying different case or punctuation rules across splits.
- Removing meaningful symbols from technical or source-code corpora.
- Using aggressive stemming without checking semantic collisions.
- Ignoring reserved tokens when estimating model parameters.
- Choosing top-N vocabulary without measuring occurrence coverage.
- Comparing TTR values from very different corpus sizes.
Frequently asked questions
What is vocabulary size in machine learning?
It is the number of distinct tokens available to a model. The count depends on tokenization and preprocessing. Reserved model tokens may increase the final size.
Why does vocabulary size matter?
Vocabulary size affects coverage, memory, and computational cost. Larger vocabularies increase embedding and softmax parameters. Smaller vocabularies may create more unknown tokens.
What is an OOV token?
An out-of-vocabulary token is absent from training vocabulary. Word-level models often map it to UNK. Subword tokenizers usually reduce this problem.
Should punctuation be removed?
That choice depends on the task. Punctuation helps sentiment, syntax, and generation models. It may be unnecessary for simpler topic analysis.
What minimum frequency should I use?
There is no universal threshold. Start with two or three for moderate corpora. Verify that retained-token coverage remains acceptable.
What does type-token ratio indicate?
TTR estimates lexical diversity within a sample. Higher values indicate relatively more distinct tokens. Corpus length strongly influences the result.
What are reserved tokens?
Reserved tokens support padding, unknown words, sequence boundaries, and masking. They occupy vocabulary IDs like ordinary tokens. Include them in model memory estimates.
Does this calculator create subword vocabularies?
It analyzes words, characters, and configurable n-grams. It does not train BPE or WordPiece merges. Those algorithms require iterative merge optimisation.
How accurate is Heaps’ Law?
Heaps’ Law provides an empirical growth estimate. Accuracy depends on suitable K and beta values. Fit them using observed corpus growth points.
Can I compare training and test text?
Yes, paste test text into the comparison field. The calculator reports overlap, coverage, and OOV values. Use identical preprocessing for both corpora.