Tokenisation Challenge Game

Round 1 · Mark token boundaries

Click the gaps where one token ends and the next begins.

Click a boundary marker to toggle it. Correct boundaries depend on the selected strategy.

Ready. Configure a strategy and begin a challenge.

Round scoreboard

Score

Accuracy

Streak

Lives

Round

1/10

Time

Session progress0%

Current tokenizer

Achievements

Token output and explanation

Inspect token IDs, positions, lengths, and the rule behind each split.

Vocabulary laboratory

Create custom vocabularies for subword and unknown-token experiments.

Vocabulary, one token per line

Performance analytics

Plotly charts update after every checked challenge.

Learning guide

What tokenisation does

Tokenisation converts raw text into units a program can count, index, compare, or send into a language model. The chosen rules determine every boundary.

Word and sentence boundaries

Whitespace is simple but fragile. Rule-based tokenizers handle punctuation, contractions, abbreviations, dates, decimals, email addresses, and sentence-ending marks more carefully.

Subword tokenisation

BPE, WordPiece, and unigram methods split unfamiliar words into reusable pieces. This reduces unknown words while keeping vocabulary size manageable.

Vocabulary size and coverage

Larger vocabularies can shorten token sequences but need more storage. Smaller vocabularies create longer sequences and more subword fragments.

Multilingual and Unicode text

Some languages do not use spaces consistently. Emoji, combining marks, right-to-left scripts, and mixed-language text require Unicode-aware processing.

Context length and model cost

Language models count tokens rather than visible words. Different tokenizers can change prompt length, processing cost, and how much content fits inside context limits.