A long low shelf of plain unmarked cloth-bound volumes in muted colours

Glossary of Local AI Terms

Local AI has a vocabulary problem. Model cards, forum threads and spec sheets all assume you already know what a quantized 7B model with a long context does to your graphics card. This glossary explains the terms that actually decide what you can run at home, in plain language, with the numbers left where they belong, on the hardware and model pages.

These are working definitions for someone choosing a GPU or a first model, not formal ones for researchers. Where a term has a precise mathematical meaning and a practical one, you get the practical one, plus a note about where the difference matters.

Hardware and memory

What fits on your machine, and how fast it moves once it fits.

  • VRAM. The memory on your graphics card, and the single hardest limit on which models you can load at all.
  • Memory bandwidth. How fast that memory can be read. It sets your generation speed once a model fits.
  • Unified memory. One pool of memory shared by CPU and GPU, the approach Apple silicon takes, with different tradeoffs from a discrete card.
  • KV cache. The per-conversation memory a model keeps while generating. It grows as the conversation gets longer, which is why long chats run out of room.
  • Quantization. Storing a model's weights at lower precision so it fits in less memory, at some cost in quality.

Models and inference

What a model is made of, and what happens when you run one.

  • Token. The chunk of text a model reads and writes. Roughly a word fragment, and the unit everything else is counted in.
  • Context window. How many tokens a model can consider at once. Its working memory for a single request.
  • Parameters. The learned weights that make up a model. The 7B or 70B in a model's name.
  • Attention. The mechanism that lets a model weigh which earlier tokens matter. It is why long contexts get expensive.
  • Inference. Running a trained model to get output, as opposed to training it. Inference is what you do on your own hardware.

Training and formats

Changing a model, and the file formats you will meet when downloading one.

  • Fine-tuning. Further training an existing model on your own data to change its behaviour or style.
  • LoRA. A small add-on trained instead of the full model, cheap to make and easy to swap. Common in image generation.
  • Embedding. Text or images turned into a list of numbers you can search by meaning. The basis of local retrieval.
  • Diffusion. The process behind most local image and video models, which start from noise and refine it into a result.
  • GGUF. The quantized model file format used by llama.cpp and the tools built on it.

How to use this

Pages across the site link here on the first mention of a term, so you can get a definition without losing your place. Each term page then links back out: to the hardware that constrains it, the model pages where it shows up, and the apps that expose it as a setting you have to choose.

If you are starting from zero, read VRAM, quantization and parameters first. Between them those three answer most of "will this model run on my machine?". Then pick an area and go deeper: local LLMs, image generation, video or audio.