Local LLM runtimes
A runtime is the program that loads a model file from disk and generates text with it. The model pages under /llms/models cover the weights themselves. This section, part of local LLMs, covers the software that runs them.
What a runtime does
The weights on your drive are inert. A runtime reads them into memory, holds the conversation state, and does the arithmetic that turns your prompt into one token after another. Everything people mean by local inference happens inside this program.
A runtime is also where the practical limits live. It decides which file formats it can open, which hardware it can use for the maths, how much memory the KV cache is allowed to take, and whether other software on your machine can talk to it.
Why the choice matters
Four things separate one runtime from another.
- Hardware backends. A runtime has to be built against the compute stack your machine has, usually CUDA on NVIDIA cards, Metal on Apple silicon, ROCm or Vulkan on AMD cards, or plain CPU. Broad backend support is what lets the same model run on very different machines. See /hardware for which machines run what.
- Model formats. Most local setups revolve around GGUF files, which bundle the weights and their quantization into a single file. Other runtimes prefer other formats, and each one can only open what it was built to open.
- Ease of use. Some programs hand you a library and a terminal. Others hand you a window with a download button. Both end up running the same model, with very different amounts of setup in between.
- A local API. If you want a chat front end, an editor plugin or your own script talking to the model, the program needs to expose an HTTP endpoint on your machine. That API is what turns a runtime into infrastructure rather than a single-purpose tool.
None of these makes one runtime correct. They make each one a better or worse match for how you work and what you own.
The runtimes covered here
- llama.cpp is the inference library and command line tooling that most of the local ecosystem is built around. Maximum control, minimum hand holding.
- Ollama is a model manager with a local API. It pulls models by name, keeps them in one place, and runs a background service other tools can call.
- LM Studio is a desktop app. Browse models, download them, chat in a window, and switch on a local server when you need one.
You can run all three on the same machine. They share model files more often than not, so trying a second one is cheap.