Term
Token
The atomic unit an LLM reads and emits. A token typically corresponds to ~3–4 characters of English text or a single short word; for code, tokenisation depends on the model's tokeniser. Pricing, context windows, and rate limits are all denominated in tokens.
Background
Tokens are produced by a tokeniser (BPE, SentencePiece) that splits text into a learned vocabulary. "Hello world" might be 2 tokens; a code variable like `userIdHash` might be 3-4. English averages ~0.75 words per token; code is closer to ~0.5. Tokens are the universal currency of LLM operations — context-window size, latency, and API cost all scale with token count. Optimising prompts often means reducing tokens, which is why concise instructions and compact data formats matter.