Quantization for local LLMs
Quantization is the single technique that makes local language models practical on consumer hardware, which is why it gets a page of its own under local LLMs. The glossary entry for quantization defines the term if you want it first; this page covers the parts that matter when you are choosing a file to download.
Why the weights tolerate it
A trained model is a very large pile of numbers, and the striking thing is how little precision most of those numbers need. The behaviour of the network comes from the pattern across millions or billions of parameters, not from the exact value of any one of them. Rounding each weight to a coarser grid perturbs every number slightly, and those perturbations largely cancel out across a layer instead of compounding.
The tolerance is uneven, though. Some layers, particularly those handling embeddings and the final output projection, degrade noticeably when compressed hard, so good schemes leave those at higher precision and compress the bulk of the network more aggressively. Modern methods also store a scaling factor per small block of weights rather than one per tensor, which stops outliers dragging a whole block off target.
Redundancy scales with size, too. A large model has enough slack to absorb heavy compression; the same setting can visibly damage a small one.
The precision ladder
Models are usually released at 16 bits per weight, the practical reference point for full quality. From there the common rungs step down through 8, 6, 5 and 4 bits, with more experimental levels below that.
- 8 bits. Effectively indistinguishable from the released weights. Chosen when you have memory to spare and want no argument about quality.
- 6 and 5 bits. Losses are hard to notice in normal use. A good place to sit if the next rung down makes you nervous.
- 4 bits. The workhorse, where most people run most models, because it roughly quarters the memory requirement against the 16-bit release while keeping the model recognisably itself.
- Below 4 bits. Real degradation shows, usually as weaker reasoning and sloppier instruction following rather than obvious gibberish. Worth it only when the alternative is not running the model at all.
The useful rule is that a larger model at a lower precision usually beats a smaller model at a higher one, as long as you stay at 4 bits or above.
Formats you will meet
The compression method and the container file are separate things, and names from both get used interchangeably.
GGUF is the format you will see most often for local use. It packs the weights, the tokenizer and the metadata into one file a runtime can load directly, and it supports a wide range of quantization levels. Filenames carry the level as a suffix, which is how you tell variants apart.
Other formats exist, generally aimed at GPU-only inference with their own tooling. They can be faster on the hardware they target, but they are less portable. Starting out, the single-file approach will cause you the fewest problems. Picking a quantization walks through reading the suffixes for a specific machine.
Context length changes the arithmetic
A quantized model file tells you how much memory the weights need, not how much the model needs while running.
Every token in the conversation leaves entries in the KV cache, and that cache grows with the context window you allow. On a long context the cache can rival the weights for memory use, which catches people out: the model loads fine, then runs out of room partway into a long document.
So budget for the cache as well as the weights, as will this model fit in memory sets out. And if you are short, you have two dials rather than one. Dropping a quantization rung frees memory permanently; reducing the context window frees it only when you do not need the length. Most engines can also quantize the cache itself, which buys back headroom at some cost to quality on long inputs.