VRAM
VRAM, short for video RAM, is the dedicated memory soldered onto a graphics card, and on a discrete GPU it is the only memory the chip can read model weights from at full speed. It is the single number that decides most of what you can run locally, which is why it shows up on nearly every page in the glossary and across /hardware.
A model has to be loaded into VRAM before the GPU can use it. If the weights plus working memory do not fit, you either shrink the model or push part of it into system RAM, where it runs far slower.
Why it matters for local AI
VRAM capacity is a wall, not a slope. Up to the limit, a GPU runs at full speed. Past the limit, layers spill to system RAM across the PCIe bus and throughput drops by an order of magnitude, even on an otherwise fast card. A cheaper card with more VRAM often beats a faster card with less, because the faster card cannot hold the model at all.
Three things compete for the same pool:
- The model weights, which scale with parameters and with the quantization you chose.
- The KV cache, which grows with the context window you actually use.
- Framework overhead, plus whatever your desktop compositor is already holding.
Capacity is not the whole story. Once a model fits, generation speed is governed by memory bandwidth instead.
Rule of thumb
Budget for the model to fit with headroom to spare, not exactly. Weights alone are the floor, and long contexts can add a surprising amount on top, so a model that fits with nothing left over will run out of memory partway through a long conversation.
If a model will not fit, drop to a smaller quantization before you drop to a smaller model. A heavily quantized large model usually beats a full precision small one.