Token
A token is the smallest chunk of text a language model reads and writes, usually a common word or a fragment of a longer one. Models do not see letters or words directly. A tokeniser splits your text into tokens, each token maps to an integer id, and that sequence of ids is what the model actually processes. See the glossary for related terms and /llms for the wider picture.
Tokenisers differ between model families, so the same paragraph produces a different token count depending on which model reads it. Common English words often become a single token, while rare words, names, code punctuation and non-Latin scripts split into several.
Why it matters for local AI
Tokens are the unit everything else on your machine is measured in. The context window is a token count. The KV cache grows per token, so memory use tracks tokens rather than characters. Generation speed is quoted per token. When you compare two setups, you are comparing how fast and how cheaply each one moves tokens.
This is also why a file that looks small can overflow a window. Source code, JSON, tables and logs tokenise badly compared to prose, sometimes by a wide margin, so a modest file of structured text can cost several times what the same byte count of plain English would.
Each token id is converted into an embedding vector before the model does any real work, which is where parameters first enter the picture.
Rule of thumb
- Treat a token as somewhat less than a word for ordinary English prose, and assume markup, code and non-English text cost noticeably more per character.
- When you need an exact count, run the text through the tokeniser shipped with the model you are using rather than estimating. Estimates are fine for planning, not for deciding whether something fits.