GGUF
GGUF is a single-file format for language model weights that packs the tensors, the tokenizer and the metadata a runtime needs into one file. It is the format most local LLM tools expect, so a model you download for home use is very often a .gguf. See the glossary for the surrounding vocabulary.
The format came out of the llama.cpp project and replaced its earlier formats. Its design goals are mundane and useful: one file rather than a directory of shards and config, memory-mappable so the engine can page weights in without a separate load step, and self-describing so a loader can read the architecture, the context window it was trained for and the chat template without extra files.
Why it matters for local AI
Two things make GGUF the practical default for running models on your own machine.
First, quantization is baked into the file. The same model is published as a family of GGUF files, each at a different precision, and the choice is made at download time rather than at load time. The quantization level is in the filename and the metadata, so picking a file is picking how much VRAM or system RAM the model will need.
Second, the engines converge on it. The llama.cpp family and the tools built on top of it all load GGUF, which means a file you download works across several of the apps without conversion.
The cost is that GGUF is a runtime format, not a training format. Adapters and fine-tuning work happens against the original weights, and conversion to GGUF is the last step.
Rule of thumb
- Match the file to your memory, not to your ambition. A model that fits entirely in VRAM at a lower precision will normally beat a higher-precision file that spills into system RAM.
- The very smallest quantizations of a small model are usually a worse deal than a mid quantization of the next model down in size.