Inference
Inference is running a trained model to produce output: you give it a prompt, it returns text, an image or audio. It is the opposite end of the process from training, which is where the parameters were learned. Almost everything covered on this site and in the glossary is about inference, because that is the part you can realistically do on your own hardware. /llms covers doing it with language models.
The distinction matters because the two have completely different hardware profiles. Training adjusts weights over enormous datasets and needs sustained compute, plus memory for gradients and optimiser state. Inference only reads the weights, which makes it far cheaper, and puts different pressure on your machine.
Why it matters for local AI
Local inference is usually memory bound, not compute bound. Producing each token requires reading the active weights out of memory and into the chip. That read, not the arithmetic, is what takes the time, so throughput tracks memory bandwidth much more closely than it tracks raw compute.
This explains behaviour that otherwise looks odd. A card with modest compute but fast memory can outrun a nominally stronger one. Adding a faster GPU to the same setup may change very little if the model still has to spill out of VRAM. And when weights do not fit and get read across the system bus instead, speed collapses, because the bus is far slower than the memory on the card.
Prompt processing is the exception. Reading a long prompt happens in parallel across positions and is genuinely compute heavy, which is why a long prompt can take a while before the first token appears while the rest of the reply then flows steadily.
Rule of thumb
- Fit the whole model and its KV cache in fast memory. That single decision matters more than any other tuning you can do, which is what makes quantization worth the quality trade.
- When choosing hardware for inference, compare memory capacity first and memory bandwidth second. Compute is rarely the binding constraint.