What Counts as a Token (and Why AI Cares)

Language models do not read words — they read tokens, the fixed chunks of text their vocabularies are built from. Every price, every context window, every "maximum length" in AI tooling is quoted in tokens, and the gap between what you think you sent and what the model actually counted is where budgets and context windows quietly disappear.

A token is not a word

Modern models build text with byte-pair encoding (BPE): a statistical process that starts from raw characters and repeatedly merges the pair of symbols that occurs most often, until a vocabulary of useful chunks emerges. Common sequences win their own token — "the" is a token of its own, and the space before it usually rides along, so " the" mid-sentence and "The" at the start of a line are different tokens. Rare sequences never earn a merge, so uncommon words get decomposed: "unbelievable" becomes something like "un" + "believ" + "able". Numbers, URLs, and code tend to split even more aggressively, because their characters repeat in irregular patterns. The practical consequence: token count is not word count and not character count either — it sits between them and drifts with the text.

The two rules of thumb

Two approximations dominate, and real tokenizers land near both for ordinary English:

~4 characters per token        1000 chars ≈ 250 tokens
~0.75 words per token          1000 words ≈ 1333 tokens

They disagree on every text, which is why careful estimators compute both and take the larger — the safer answer when you are budgeting against a limit. Take the sentence "The quick brown fox jumps over the lazy dog.": it is 44 characters and 9 words. The character rule gives ceil(44 / 4) = 11 tokens; the word rule gives ceil(9 / 0.75) = 12. The estimate is 12, the higher of the two. A 100-word email with 500 characters lands the other way: 125 by characters, 134 by words, so 134 is reported. For plain English the two estimates agree closely — and that agreement is exactly why "4 characters per token" became folklore.

Where the estimate breaks

The average holds only for the kind of text it was measured on. Dense English prose tracks the 4-chars-per-token line; poetry and dialogue with short lines waste tokens on whitespace and punctuation; code is worse, because every brace, quote, and operator tends to cost its own token. Non-Latin scripts drift further: a Chinese character is usually a token of its own (or more), so the same "1000 words" translated can triple in tokens. Long unbroken strings — a 60-character base64 blob, say — split into many single-character tokens, which is why pasting encoded data into a prompt eats context far faster than its length suggests. When accuracy matters, run the real tokenizer for the model you are calling; treat estimates as a safety margin, not a quote.

Context windows are token budgets

A model's context window is quoted in tokens, and everything in the conversation counts against it: the system prompt, every message you send, the full transcript of the exchange, and the answer being generated. The windows differ enormously by model — roughly 8,192 tokens for early GPT-4, about 16,385 for GPT-3.5-turbo, 128,000 for GPT-4-turbo and GPT-4o, and 200,000 for Claude 3 models — but the accounting is identical everywhere. That is why a long chat that worked yesterday starts refusing today: the transcript grew until system prompt + history + your question + room for the answer no longer fit. Estimators report how much of the window a text consumes and how many tokens remain, which turns "it feels long" into a number you can plan around.

Why a character counter still matters

Because tokens are chunks of bytes, character counting and token counting are related — and characters are what you can see and edit. When a form says "500 characters max", trimming characters reliably trims tokens; the reverse is not true, since two texts with the same word count can differ in tokens by 30% or more. And tokens are made from bytes, not letters: an emoji or an accented character can be several bytes, several tokens, and anywhere from one to four characters. If you want the full picture of how text becomes bytes before it becomes tokens, How Text Encoding Works covers the byte layer underneath.

Try it yourself

Paste any text into the Token Counter — it estimates tokens by both rules, takes the higher, and shows what percent of a given model's context window your text would consume, plus the tokens remaining. To trim toward a budget, the Word Counter shows words, sentences, and reading time as you edit, and the Character Counter tracks the character-level limits that forms and social platforms actually enforce.