The same simple leaf shape repeated across a pale panel, rendered in mosaic tiles that grow coarser from left to right

Picking a quantization

By the end of this guide you will be able to look at a list of a dozen files with cryptic suffixes, understand what separates them, and pick one. It is for anyone who has chosen a model and now has to choose which version of it to download. It belongs with the other guides.

The one sentence version: higher precision costs more memory and preserves more of the original model, lower precision fits on smaller hardware and degrades gently for a while before it degrades badly.

What this assumes

  • What quantization is doing. The model's parameters were trained at a high precision. Quantizing stores them with fewer bits each, which shrinks the file and the memory it occupies at the cost of some fidelity.
  • A memory budget. Will this model fit in memory? works one out, and your card's VRAM or your machine's unified memory figure is in the hardware section.
  • Which file formats your app accepts. GGUF is the format most local-first runtimes expect, and it is the one whose file names carry the quantization label. Other engines use other formats, and the runtimes index covers which is which.

Steps

  1. Read the ladder as a spectrum, not a menu. Arrange the available files by size. The largest is closest to the original weights. The smallest is the most compressed. Everything in between trades one for the other. The labels differ between formats and schemes, but the ordering by file size is always reliable.

  2. Rule out everything that does not fit. Apply your memory budget, remembering that the weights are not the whole cost. Whatever is left after the KV cache and runtime overhead is what you have to spend on precision. This usually eliminates most of the list in one pass.

  3. Take the largest that fits with headroom. Among what remains, more precision is better, with one caveat: leave room to grow the context window later. If the biggest survivor only fits with a short context, drop a rung.

  4. Understand where the cliff is. Going down the ladder, quality falls slowly at first. The model still writes well, still follows instructions, and you would struggle to tell the difference in casual use. Past a certain point the fall gets steep: the model starts making arithmetic errors, losing the thread over long passages, or producing subtly wrong code that looks fine. The exact position of that cliff varies by model and by family, which is why nobody can give you a single rule. Assume the bottom two or three rungs of any ladder are for people who have no other option.

  5. Branch on hardware class. On a card with a large VRAM pool, the interesting question is usually "how large a model can I run at decent precision?", and the answer is often a bigger model at a middling quantization rather than a smaller model at a high one. On a unified-memory laptop, the pool is bigger but the memory bandwidth is the limit, and a smaller file also means less data moved per token, so a lower rung buys speed as well as space. On a card with a modest pool, you are simply taking the largest thing that fits.

  6. Prefer size class over precision when you have to choose. Given the same memory budget, a larger model at a moderate quantization usually beats a smaller model at near-original precision. This holds until you reach the bottom of the ladder, where the larger model's damage outweighs its extra capability.

  7. Test on your own work, not on a demo. Take a handful of tasks you actually care about and run them at two adjacent rungs. If you cannot tell the difference, take the smaller one and spend the memory on context instead. If you can, you have found your floor.

  8. Write down what you chose. Six weeks later, when a model behaves oddly, the first useful question is which file you are actually running. See organising model files.

Common mistakes

  • Downloading the smallest file because it downloads fastest, then judging the model by it.
  • Assuming a label means the same thing across two different formats or two different families.
  • Spending your entire memory budget on precision and leaving nothing for context.
  • Believing quality degrades in a straight line. It does not, and the last stretch is much worse than the first.
  • Comparing quantizations on a single question. The damage shows up in long tasks, arithmetic and code, not in a greeting.

Related