Fundamentals of Generative AI

A token is a sub-word unit (a chunk of characters) that an LLM reads and produces. Models process text as sequences of tokens rather than whole words or individual characters.

On average about one token corresponds to roughly 3-4 characters, and a token is often shorter than a word — common rules of thumb put it near 0.75 words per token. Long or rare words split into multiple tokens.

Token count drives both cost (token-based pricing) and how much fits in the context window. Two texts of equal character length can use different numbers of tokens depending on vocabulary.

Tokenization is the process of breaking input text into tokens that the model can process, and mapping each token to an integer ID. The reverse step turns generated token IDs back into text.

Chunking is splitting documents into smaller pieces (chunks) before embedding and indexing them, so relevant passages can be retrieved and fit into the prompt.

Large chunks preserve context but may dilute relevance and waste tokens; small chunks are precise but can lose surrounding context and split key information across pieces.