Memory bandwidth
Memory bandwidth is how much data a processor can read from its memory per second, and for local text generation it is the main thing that decides how fast tokens come out. It sits alongside capacity as one of the two memory numbers worth learning, and the rest of the pair is covered in the glossary under VRAM.
Generating one token requires reading essentially every active weight in the model. The arithmetic is cheap by comparison, so the hardware spends most of its time waiting on memory rather than computing.
Why it matters for local AI
Once a model fits in memory, adding compute barely helps. This is why two cards with similar raw compute can generate at very different speeds, and why a laptop with lots of unified memory can load a huge model and still feel slow: it has the capacity but not the bandwidth.
It also explains a pattern people find confusing:
- Prompt processing, where the machine reads your input in parallel, is compute bound and usually fast.
- Token generation, one token at a time, is bandwidth bound and usually the bottleneck.
- Running on CPU and system RAM is slow mostly because that memory is several times slower than the memory attached to a GPU, not because the CPU cannot do the math.
Memory type is the honest signal here. Vendors publish the memory technology on a card, and newer generations of graphics memory are faster than older ones at the same width.
Rule of thumb
Bandwidth sets your ceiling on generation speed; capacity sets whether you get to run at all. Fit first, then optimise for bandwidth.
Smaller weights move faster. A more aggressive quantization reduces the bytes read per token, so it speeds up generation as well as saving space. This is the main reason quantized models feel quicker, not just smaller.