Tokens and Context Windows Without the Jargon
If you have used an AI chat tool, you may have seen mentions of tokens, limits or “context”. These terms describe how the system reads and remembers text, and understanding them explains several odd behaviors.
What is a token?
A model does not read letters or whole words the way you do. It breaks text into chunks called tokens. A common short word might be a single token; a long or rare word may be split into several pieces; punctuation and spaces can count too. Different systems split text differently, so there is no universal conversion, but a token is often smaller than a word.
Why use chunks? They offer a practical middle ground. Letters alone would make sequences very long, while whole words would require an enormous vocabulary and handle unfamiliar words poorly.
What is a context window?
The context window is the amount of text, measured in tokens, that the model can consider at one time. It includes your instructions, the conversation so far, any documents you paste and the response being written. Think of it as a desk: only what fits on the desk is in view. Material beyond that is not visible to the model.
Consequences you may notice
- Forgetting early details. In a long conversation, older messages may fall out of the window, so the assistant seems to lose track of what you said first.
- Trouble with very long documents. If a file exceeds the window, it must be split or summarized, and details can be lost.
- Cut-off answers. Responses have limits, and a long reply may stop before it finishes.
- Uneven attention. Even within the window, a model may use some parts of the input better than others.
Memory is not the same as context
Some products add features that save notes between conversations. That is a separate mechanism layered on top. Within a single exchange, what the model knows about you is mostly what appears in the window.
Tips for working within limits
- Put key instructions clearly and restate them if the conversation is long.
- Paste only the relevant sections of a document.
- Ask for summaries at checkpoints and start a fresh session with that summary.
- Break large tasks into stages.
Why costs and speed link to tokens
Processing more tokens takes more computation, so longer inputs and outputs generally take more time and resources. Providers often describe usage in these terms, though details vary by service.
An everyday comparison
Imagine reading a long report through a small cardboard frame that you slide along the page. Whatever sits inside the frame is clear; what lies outside has to be remembered or is lost. A larger frame shows more at once, but moving it and reading everything inside takes longer. A context window behaves similarly: it limits what can be considered together, and handling more of it costs more effort.
This also explains why summarizing earlier parts of a long discussion and carrying the summary forward can work better than hoping the model remembers every detail.
The simple picture is this: text is cut into tokens, and the model can only look at a limited number of them at once. Keep that in mind and many puzzling behaviors become predictable, from forgotten instructions to unfinished replies.