Warm amber light glowing through the mesh side of a dark computer tower under a desk in an unlit room

RTX 4090 for local AI

The RTX 4090 is NVIDIA's consumer flagship from the Ada Lovelace generation, and for a long stretch it was the default answer to "which single card should I buy for local AI". It suits people who want one desktop card that handles text, image and video work without constant compromise. For how it compares with the rest of the field, see the GPU index and the wider hardware page.

Memory

The RTX 4090 ships with 24 GB of GDDR6X memory on a wide bus, which puts it at the top of the consumer tier for both capacity and memory bandwidth. Capacity is the hard limit on what loads at all; bandwidth is what makes it feel quick once loaded.

What fits in 24 GB of VRAM is always approximate, because it depends on quantization, on how long a context window you ask for, and on how much room the KV cache needs as a conversation grows. As a rough guide:

  • Small and mid-size text models, up to roughly 30B parameters at 4-bit, sit comfortably in 24 GB with generous context.
  • Models in the 70B class are reachable only at aggressive quantization and short context, and they will feel cramped.
  • Image models, including large diffusion transformers, generally fit with headroom for LoRA stacks and high resolutions.

Treat those bands as starting points and verify against the specific model page you care about.

What it runs well

  • Text (/llms): comfortable. Most popular open-weight models run at usable speed with room for long context, and light fine-tuning is practical on a single card.
  • Image (/image): comfortable. High resolutions, batch generation and multiple adapters all fit.
  • Video (/video): workable. Short clips and moderate resolutions are fine; longer sequences and higher frame counts start pressing against 24 GB.
  • Audio (/audio): comfortable. Speech, music and voice models are far less memory-hungry than video, so the card is rarely the bottleneck here.

Runtime support

The 4090 sits squarely inside the CUDA ecosystem, which is the path of least resistance for local AI. Nearly every engine, extension and quantization format is tested against NVIDIA hardware first, so you spend less time debugging and more time running inference.

  • llama.cpp for direct control, GGUF models and fine-grained offload between GPU and system RAM.
  • Ollama when you want models pulled and served with minimal setup.
  • LM Studio for a desktop GUI with a built-in model browser.

Image, video and audio tooling follows the same pattern: CUDA builds are the reference, and everything else is a port.

Trade-offs

The 4090 is a large, power-hungry card. Plan for a high-wattage power supply with a modern 12V high-power connector, good case airflow, and physical clearance. Most partner designs are three slots or more and long enough to conflict with front radiators or drive cages in mid-tower cases. Check the card length against your chassis before ordering.

Unlike the 3090 before it, the 4090 has no NVLink connector, so multi-GPU setups fall back to splitting models across the PCIe bus rather than pooling memory directly.

Against its siblings: the RTX 3090 offers the same 24 GB for less outlay and less speed, which matters if capacity is your only concern (3090 vs 4090). The RTX 5090 raises both capacity and bandwidth at the cost of more power and heat (4090 vs 5090). If you would rather trade raw speed for a quiet, low-power machine with a large pooled memory space, compare against the M3 Max.

Related