Tokens: the Lego bricks of text

Models don't read letters or words. They read tokens.

A large language model (LLM) can't read text the way you do. First, the text is chopped into pieces called tokens. A token might be a whole word ( cat), part of a word (un, believ, able), or a bit of punctuation.

As a rule of thumb for English, one token is about 4 characters, or roughly three quarters of a word. Models are priced, limited and measured in tokens, not words.

Fun fact: The splitter above is a toy with simple rules. Real tokenizers learn their pieces from huge amounts of text, so common words get one token and rare words get several.

Quick check

Roughly how many tokens is a 750-word English essay?

1. About 100
2. About 1,000
3. About 10,000
4. Exactly 750

Answer: About 1,000. A word is about 1.3 tokens on average, so 750 words is close to 1,000 tokens. The exact number depends on the tokenizer and the text.