Language Model Perplexity Analysis
Model comparison
| Rank | Model | Dataset | Tokens | Perplexity | Difference | Improvement |
|---|
Visual analysis
Calculation history
No calculations saved yet.
Merged calculation report
Run a calculation to create a report.
Formula used
Perplexity = exp[-(1/N) × Σ ln(pᵢ)]
Perplexity = exp(NLL / N)
Perplexity = eᴴ, 2ᴴ, or 10ᴴ according to the selected logarithm base.
How to use the calculator
- Select the calculation mode matching your available data.
- Choose the logarithm base used by your values.
- Enter probabilities, losses, or corpus-level rows.
- Adjust smoothing, clipping, exclusions, and precision.
- Calculate, review warnings, inspect charts, and export results.
Example data
| Mode | Example input | Meaning |
|---|---|---|
| Probabilities | 0.25, 0.50, 0.10, 0.40 | Observed token probabilities. |
| Log probabilities | -1.3863, -0.6931, -2.3026 | Natural-log token probabilities. |
| NLL | 24.75 over 20 tokens | Total negative log-likelihood. |
| Corpus | Sentence A,12,-18.4 | Name, tokens, total log-likelihood. |
Interpretation notes
Lower perplexity usually means stronger predictive confidence. Compare results only when datasets and tokenization match. Vocabulary differences can distort direct comparisons.
Perplexity is the inverse geometric mean probability. It represents effective average next-token uncertainty. Domain difficulty strongly affects the score.
Frequently asked questions
What does perplexity measure?
It measures how surprised a language model is by evaluation data.
Is lower perplexity always better?
Usually yes, but only for comparable datasets, vocabularies, and tokenization.
Can perplexity be below one?
Proper probability-based perplexity is normally at least one.
Why are zero probabilities problematic?
Their logarithm is undefined and creates infinite perplexity.
What is corpus-level perplexity?
It combines total log-likelihood across all valid corpus tokens.
What is macro-average perplexity?
It averages sequence perplexities without weighting by token count.
What is micro-average perplexity?
It uses combined log-likelihood and combined token count.
Should padding tokens be included?
Usually not, unless the evaluation method explicitly scores them.
How does cross-entropy relate to perplexity?
Perplexity is the exponential of average cross-entropy.