Quantization
Quantization is the process of storing a model's weights at lower numerical precision than they were trained at, so the model takes less memory and less time to read. It is the single most useful trick in local AI, and the reason large models run on ordinary hardware at all, so it is worth reading alongside the rest of the glossary.
A trained model usually holds each weight as a 16 bit number. A quantized copy stores the same weights in fewer bits, with a scheme that records enough scaling information to reconstruct an approximation at inference time.
Why it matters for local AI
Memory is the binding constraint on a home machine. Quantization attacks it directly: fewer bits per weight means a smaller file, less VRAM used, and fewer bytes read per token, which also raises generation speed because throughput is limited by memory bandwidth.
The cost is accuracy. Lower precision means the reconstructed weights drift from the originals, and the model gets slightly worse at the things it was marginal at already: long reasoning chains, code, precise instruction following, less common languages. The damage is gradual at first and then steep.
Most local models are distributed as quantized files in the GGUF format, with several precision levels per model. Modern schemes do not quantize everything equally, keeping the layers that are most sensitive at higher precision.
Rule of thumb
Prefer a bigger model at a smaller quantization over a smaller model at full precision, as long as the bigger one still fits with headroom. More parameters at reduced precision generally beat fewer parameters at full precision.
There is a floor. Mid range quantizations lose very little you would notice in normal use, while the most aggressive ones degrade output in ways that are obvious. Go as small as you must to fit, and no smaller.
Test on your own prompts. Quality loss shows up unevenly, and the only benchmark that matters is your actual workload.