Token Count Calculator for Machine Learning

Token Analysis Results Calculated locally

Input Tokens0

Estimated Output Tokens0

Total Tokens0

Remaining Tokens0

Context Limit0

Context Usage0%

Estimated Request Cost£0.000000

Generated Chunks0

Context-window usage 0 of 0 tokens

Detailed Measurements

Role Distribution

Warnings and Optimisation Notes

1. Text and Conversation Input

Prompt, document, code, JSON, Markdown, or HTML

One prompt or record per line

CSV files are read locally. Choose the text column after loading.

CSV text column

Upload a text dataset

Supported: TXT, CSV, JSON, Markdown, logs, and HTML.

Active input source

Privacy mode

Text is analysed in your browser. This file does not send prompt content to a server.

2. Model, Tokenizer, and Context Settings

Model preset

Tokenizer family

Approximation mode

Language

Content type

Context-window size

Reserved output tokens

Expected output tokens

Custom characters per token

Custom words per token

Manual special-token overhead

Tokens per chat message

Include special tokens

Include whitespace effects

Case-sensitive frequency analysis

3. Cost and Usage Estimation

Currency symbol

Input price per million tokens

Output price per million tokens

Cached input price per million

Cached input share (%)

Batch discount (%)

Requests per day

Active days per month

Comparison model total rate

4. Chunking, Embeddings, and Fine-Tuning

Maximum tokens per chunk

Overlap tokens

Embedding price per million tokens

Number of similar documents

Fine-tuning epochs

Optimisation target tokens

Preserve paragraphs

Preserve sentences

Preserve headings

Token Visualisation and Detailed Token Table

Highlighted Token Pieces

Token Details

#	Estimated ID	Token text	Decoded value	Character position	Length	Whitespace	Special	Frequency

Chunked Output

Batch Dataset Analysis

Record	Preview	Words	Tokens	Limit Status

Prompt Optimisation Report

Formula Used

Estimated Tokens = Adjusted Character Count ÷ Characters Per Token
Remaining Tokens = Context Window − Input Tokens − Reserved Output Tokens
Input Cost = Input Tokens ÷ 1,000,000 × Input Rate
Output Cost = Output Tokens ÷ 1,000,000 × Output Rate
Training Tokens = Dataset Tokens × Number of Epochs
Embedding Tokens = Document Tokens × Number of Documents

This calculator uses configurable estimation rules. Exact model tokenizers can produce slightly different values. Use conservative mode for safer context planning.

How to Use

Choose single text, chat messages, or batch records.
Paste content or upload a supported local file.
Select a tokenizer family and model context preset.
Enter output, pricing, chunking, and dataset assumptions.
Press Calculate Tokens to generate all reports.
Review warnings before sending large model requests.
Export the summary, tables, chunks, or printable report.

Example Data

Example	Characters	Approximate tokens	Recommended use
Short instruction	400	About 100	Prompt testing
Technical article	20,000	About 5,000	Summarisation
Source-code file	20,000	About 6,500	Code review
Knowledge document	80,000	About 20,000	Chunked retrieval

Frequently Asked Questions

What is a token?

A token is a model-readable text unit. It may represent a word, word fragment, symbol, space, or punctuation mark.

Are tokens equal to words?

No. One word may become one or several tokens. Punctuation and spaces can also affect the count.

Why do model counts differ?

Models use different vocabularies and tokenisation algorithms. The same prompt can therefore produce different token counts.

Does code use more tokens?

Code often includes short identifiers and punctuation. Those patterns can raise tokens per word compared with ordinary prose.

How does language affect token count?

Languages with different scripts may use different token densities. Mixed scripts and emojis can increase byte-level tokenisation.

What is context-window usage?

It is the share of available model context occupied by input, formatting overhead, and reserved output capacity.

Why reserve output tokens?

Reserved capacity prevents the input from consuming the entire context window. The model then has space to answer.

How should documents be chunked?

Use chunks below the embedding or model limit. Add modest overlap while preserving sentences and paragraph boundaries.

Is the cost estimate exact?

No. It depends on pricing, cache rules, output length, and exact tokenisation. Enter current provider rates manually.

Is uploaded text private?

This calculator reads files locally in the browser. It does not submit prompt content through the PHP page.