Ollama
Ollama is a model manager wrapped around a local inference engine. You ask for a model by name, it fetches it, and it runs as a background service that other software on your machine can call. Among the local LLM runtimes it is the one that gets out of the way fastest, which makes it a good default for anyone new to local LLMs who wants a working model today rather than a tuning project.
It suits developers wiring a local model into something else, and anyone who would rather manage models the way they manage container images than by hunting for files.
How it runs models
Ollama runs GGUF models. Most of the time you never see the file. You pull a model by name and Ollama stores it in its own directory, tracking which quantization you have. Names in its catalogue usually carry a size and a quantization tag, and there is a default for each model so you can skip the decision entirely. You can also point it at a GGUF file you already downloaded, which matters if you keep a shared model directory.
On hardware it takes the usual local-inference paths and picks one for you:
- CUDA on NVIDIA cards. See /hardware/gpus and the pages for the RTX 4090, RTX 5090 and RTX 3090.
- Metal on Apple silicon, where unified memory sets the ceiling instead of a fixed card. See the M3 Max.
- ROCm or Vulkan on AMD cards such as the 7900 XTX.
- CPU, used for whatever does not fit on the GPU, or for everything if there is no usable GPU.
The automatic placement is the point and occasionally the problem. It is convenient when a model fits and quietly slow when it does not, because the overflow lands on the CPU rather than failing loudly.
Fit
Choose Ollama when you want a local API running all the time with minimal setup, when several tools on your machine should share one model service, or when you want to swap models by name in a script.
Choose llama.cpp instead when you need direct control over memory placement, backends or sampling, or when you are building on the library itself. Choose LM Studio when you want a graphical app to browse models and chat in, rather than a service with no face.
Working with it
The workflow is short. Pull a model by name, check that the quantization you pulled fits your memory alongside its KV cache, and talk to it. The LLM models pages cover which families are worth pulling.
Two settings are worth setting deliberately rather than leaving to defaults. The first is the context window length, since the default is often shorter than the model supports and longer contexts cost memory. The second is how long a model stays resident after your last request, which decides whether the next one answers immediately or waits for a reload.
Because the service exposes an HTTP API locally, editors, chat front ends, agent frameworks and your own scripts can all point at the same running model. That shared endpoint, rather than raw speed, is usually the reason people keep Ollama installed.