Parameters
Parameters are the learned numbers inside a model, the weights adjusted during training that encode everything it knows. Their count is the usual shorthand for model size, which is why names carry a number of billions: that number is the parameter count. The glossary explains the rest of the vocabulary, and /llms/models covers specific model families.
Parameter count is a rough proxy for capability within a family, and a poor one across families. A newer, better-trained small model often beats an older larger one. What the count does predict reliably is memory use.
Why it matters for local AI
Every parameter has to be stored somewhere you can read it quickly, so the parameter count sets the floor on how much VRAM or unified memory a model needs before you add a KV cache or anything else.
What varies is the number of bits per parameter. At full precision each weight takes a couple of bytes. Quantization stores them more coarsely, cutting the footprint roughly in proportion to the bit width, at some cost in output quality. This is the main lever you have: the same model can be a comfortable fit or an impossible one depending on the quantization level you pick, and formats like GGUF exist to package those variants.
Parameter count also drives speed. Generating a token means reading essentially all the active weights from memory, so a bigger model is slower on the same hardware for reasons of memory bandwidth, not compute. Mixture-of-experts models split the difference by activating only part of their parameters per token while still needing all of them resident.
Adapters such as LoRA add a small number of extra parameters on top of a frozen base, which is why fine-tuning that way is far cheaper than training the whole model.
Rule of thumb
- Estimate memory as parameter count times bytes per parameter at your chosen quantization, then leave clear headroom for the cache and the rest of the system.
- If a model does not fit, try a lower quantization of the same model before dropping to a smaller one. A quantized larger model is often the better trade, down to a point.