A pegboard wall of hand tools hung in neat rows in a workshop

Hardware for Local AI

One question comes first: what can I run on this machine? Every other hardware question is downstream of it. Whether you came here for local LLMs, image generation, video generation, or speech and music, the same checklist decides what is possible: how much memory your accelerator has, how fast that memory is, and how well the apps you want to run support the chip you own.

You probably do not need to buy anything to start. But memory capacity is a hard ceiling: a model either fits or it does not, and patience will not make an oversized model run well.

Memory matters more than compute

Most buying advice is written for gaming or rendering, where raw compute wins. Local inference works differently. Generating a token or a diffusion step means reading the model's weights out of memory and doing relatively little arithmetic with each one. That makes the job memory bound.

Two numbers follow from this. The first is VRAM capacity, or unified memory capacity on systems that share one pool between CPU and GPU. The weights have to live somewhere. If they spill into system RAM or onto disk, throughput collapses, often by an order of magnitude or worse. The second is memory bandwidth, which is how fast the chip can stream those weights. For text generation, bandwidth is the single best predictor of tokens per second on a given model. A card with modest compute and fast memory will usually beat a card with the reverse.

Two things shrink the memory problem. Quantization stores weights at lower precision, which cuts the footprint substantially at some cost to quality, and it is the main reason large models run on consumer hardware at all. The other is being realistic about the KV cache, the per conversation memory that grows with context length. A model that fits comfortably at short context can run out of room at long context, so budget headroom on top of the weights rather than filling the card to the brim.

Consumer NVIDIA

NVIDIA is the default, and not because the silicon is magic. CUDA is where almost every runtime, quantization format and optimization lands first, and it lands there working. If you want the shortest distance between downloading something and having it run, this is it.

Within the consumer line, pick on VRAM first and generation second. A previous generation card with more memory is usually the better local AI purchase, because it can hold models the smaller card simply cannot. Used cards from older high end tiers are a common value pick for this reason. Multi GPU setups can pool memory for larger language models, but add power, cooling and configuration work, so treat that as a second step.

Apple Silicon

Apple's approach is unified memory, one pool shared by CPU and GPU. In practice that means a well specced Mac can load models far larger than a typical desktop graphics card holds, without any multi GPU complexity. Support is good: the major local LLM engines ship Metal backends, and image tooling works too, though usually a step behind CUDA on features and speed.

The tradeoffs are real. Memory is fixed at purchase and cannot be upgraded, so buy more than you think you need. Bandwidth varies a lot across chip tiers within one generation, and the higher tiers are where the useful bandwidth lives. Training and fine tuning support is thinner than on NVIDIA.

AMD

AMD cards offer good memory capacity for the money, and support through ROCm and Vulkan backends has improved steadily. Popular LLM apps work. The caveat is coverage rather than capability: newer techniques, less common quantization formats and some image and video tooling arrive later, and setup can involve more version matching. If you enjoy troubleshooting, the value is there. If you would rather spend your time on models than on drivers, weigh that honestly.

Small boards and edge devices

Single board computers and mini PCs run small models usefully. Expect modest language models, speech recognition, text to speech and embedding work rather than large chat models or video generation. NVIDIA's Jetson line, mini PCs with integrated graphics that borrow system memory, and Raspberry Pi class boards for the smallest models all have a place. This is the right tier for always on jobs: a home voice assistant, a transcription service, a local search index.

How to choose

Start with what you own. Run something on it, see where it falls short, and let that define the upgrade.

If the budget is tight, buy a used card with the most VRAM you can find from a recent enough generation, and expect to use quantized models. If the budget is moderate, a current mid to high tier consumer NVIDIA card handles most language, image and audio work without fuss. If you want to run large language models above all else and dislike tinkering, a high memory Mac gets you there in one purchase. If you want image and video generation, favor NVIDIA, because that is where the tooling is.

Match the hardware to the kind of work you actually care about, text, image, video or audio, not to the spec sheet.

Related