llama.cpp
llama.cpp is an inference engine: a library written in C and C++, along with the command line tools and local server built on top of it. It is the layer most other local LLM software sits on, which is why it shows up in local LLM runtimes first. It suits people who are comfortable in a terminal, want to see exactly how memory is being spent, or need to run a model on hardware the friendlier tools do not reach. If you would rather not think about any of that, start with one of the other two programs covered under local LLMs.
How it runs models
llama.cpp works with GGUF files. A GGUF file holds the weights, the tokenizer and the metadata for one model at one quantization level, so the file you download is the only thing you need. Larger quantizations keep more of the original precision and take more memory. Smaller ones fit on modest cards at some cost in quality.
The project builds against several compute backends:
- CUDA for NVIDIA cards, which is the best supported path. See /hardware/gpus and the pages for the RTX 4090, RTX 5090 and RTX 3090.
- Metal on Apple silicon, where the unified memory pool means a machine like the M3 Max can hold models that would not fit in a discrete card's VRAM.
- ROCm or Vulkan for AMD cards such as the 7900 XTX. Vulkan is the more portable of the two and reaches hardware ROCm does not.
- CPU, which is always available and always slower, but will run a model on a machine with no usable GPU at all.
You can also split a model between GPU and system memory, keeping as many layers as fit on the card and running the rest on the CPU. That is how llama.cpp runs models larger than the GPU alone could hold.
Fit
Choose llama.cpp when you want control over placement, memory and sampling, when you are working on hardware that needs an unusual backend, or when you are building something on top of an inference library rather than using a finished app.
Choose Ollama instead when you want models fetched and managed for you and a local API running in the background without setup. Choose LM Studio when you want a window to click around in and a browsable catalogue of models.
Working with it
The workflow starts with the model file. You find a GGUF build of the model you want, pick a quantization that leaves room in memory for both the weights and the KV cache, and download it. The LLM models pages cover which models are worth your disk space.
From there you point the tooling at the file and set a context window length. Context is not free. The cache grows with the length you ask for, and asking for the model's maximum on a card that cannot hold it is the most common reason a load fails.
llama.cpp also ships a local server. Start it, and editors, chat front ends and your own scripts can send requests to it over HTTP on your own machine, with nothing leaving it.