KV cache
The KV cache is the block of memory holding the key and value tensors for every token the model has already processed, kept so that generating the next token does not require recomputing the entire sequence. It is the least visible of the memory costs on a local machine and the one that surprises people most, which is why it belongs in the glossary next to the weights themselves.
Without it, each new token would rerun attention over the whole conversation. With it, the model reuses that state and only computes the new position.
Why it matters for local AI
The cache lives in the same memory as the weights, so it competes directly with them for VRAM. Unlike the weights, it is not a fixed cost: it grows linearly with how much of the context window you have filled.
That produces a failure mode people report constantly. A model loads fine, answers a few questions, then runs out of memory partway through a long session or a large document. Nothing changed except the length of the conversation.
Size depends on the model's architecture as much as on your settings. Models that share key and value heads across query heads have much smaller caches than older designs at the same context length, which is why two similarly sized models can behave very differently at long context.
Most engines let you cap the context length and quantize the cache itself, trading a little quality for a lot of memory.
Rule of thumb
Set the context length you actually need, not the maximum the model advertises. The engine allocates cache for the configured window, so asking for a huge context reserves memory you may never use.
Budget for weights plus cache together. If a quantization fits your card with nothing to spare, it does not fit, because long contexts will take the rest.