Context window
The context window is the maximum amount of text, measured in tokens, that a model can hold in view at once when generating its next output. It covers everything the model is working from: the system prompt, the conversation so far, any documents you pasted in, and the reply being written. When the total passes the limit, something has to be dropped or summarised away. See the glossary for the rest of the vocabulary, and /llms for how this plays out in practice.
The window is a hard architectural ceiling on what the model can attend to, not a soft preference. It is also a budget you spend: every token of retrieved context is a token unavailable for the answer.
Why it matters for local AI
On your own machine the context window is a memory decision, not just a quality one. As the model reads and writes tokens it builds a KV cache, a per-token store of intermediate state that lets it avoid recomputing the whole sequence on each step. That cache grows linearly with the number of tokens in the window and sits in VRAM alongside the model weights.
Two consequences follow. First, a long window can cost as much memory as a meaningfully larger model would, so raising the context length may force you down a size class or a quantization level. Second, the cost is paid as you fill the window, so a session that starts comfortably can run out of memory deep into a long document. Apps let you cap the window explicitly for exactly this reason.
Speed suffers too. Attention has to read the whole cache for every new token, so generation slows as the conversation grows, and that read is limited by memory bandwidth.
Rule of thumb
- Set the window to the longest input you actually use, not the largest the model advertises. Unused context still reserves memory in most apps.
- If a model just fits in memory at a short context and you want a long one, step down a size or a quantization level rather than hoping the cache fits.