Attention
Attention is the mechanism that lets a model weigh every earlier token in its context when deciding what to produce next. For each new position, the model compares that position against all the ones before it, scores how relevant each is, and blends the ones that matter into its working state. It is the core operation of the transformer architecture behind current local models. The glossary covers the surrounding terms, and /llms covers running these models.
Everything a model does with long-range structure, matching a pronoun to a name twenty lines back, keeping to a format you specified at the top of a prompt, comes out of this weighing step.
Why it matters for local AI
Attention is the reason long contexts cost memory on your own hardware. To avoid recomputing the comparison for every earlier position on every new token, the engine running the model keeps the intermediate keys and values in a KV cache. That cache scales with the length of the context window and lives in VRAM or unified memory next to the weights.
It is also why generation slows down as a session grows. Producing each token means reading the whole cache back, so the work per token rises with conversation length, and the limiting factor is memory bandwidth rather than raw compute.
Model families reduce this cost in different ways, mainly by having several query heads share one set of keys and values. The details vary, but the practical effect is the same: two models of similar size can have very different cache appetites at the same context length, which is worth checking before you assume one fits where the other did.
Rule of thumb
- If throughput falls off steadily during a long chat rather than all at once, suspect cache growth from attention before you suspect thermal throttling.
- When you compare two models for long-context work, compare their cache cost per token of context, not only their parameter counts.