Getting started with local LLMs

Running a language model on your own machine takes four decisions, and this page walks through them in order. It is part of local LLMs, which links to the model and software pages referenced below. Nothing here needs a terminal command. The goal is to get you to the point where you know which files to download and what to expect when you load them.

Start with the machine you have

The first question is not which model is best, it is which models will fit. A model has to be held in memory while it runs, and the amount of memory you have sets a hard ceiling on what you can load. On a discrete GPU that means VRAM. On an Apple Silicon Mac it means unified memory shared with everything else the system is doing.

The hardware section covers what specific cards and chips can handle. If you want the arithmetic rather than a lookup table, will this model fit in memory shows how to estimate the requirement from a model's parameter count before you download anything.

You do not need a GPU at all to begin. A model can run on CPU using system RAM. It will be slower, but it works, and it is a reasonable way to find out whether local models suit you before spending money.

Pick a runtime

A runtime is the program that loads the model file and does the inference. Your choice mostly affects how much setup you do and what interface you get. The runtimes index covers the options in detail, but the short version is:

llama.cpp is the engine much of the ecosystem is built on. It runs almost anywhere, including CPU only, and gives you the most control.
Ollama wraps that engine in a model library and a local server. It is the shortest path from install to first reply.
LM Studio is a desktop application with a chat window and a model browser, which suits you if you would rather not touch a config file.

Any of the three will get you a working chat. Switching later is cheap, because they largely read the same model files.

Pick a model family

Model families differ in tone, in how well they follow instructions, and in which languages and tasks they were trained for. The models index introduces the main open families, including Llama, Mistral and Qwen.

Choose a family first, then a size within it that your memory budget allows. Smaller models in a recent family often beat larger models from an older one, so do not assume the biggest number you can fit is the best answer. If you want a structured way to narrow the field, choosing your first local model walks through it.

Pick a quantization

Models are published at full precision and then compressed into smaller variants. This is quantization, and it is what makes a model that would never fit on your machine fit comfortably. Each is usually offered at several levels, trading size against output quality, and the middle of the range is almost always the right starting point. Picking a quantization explains how to read the labels, and the quantization page goes deeper into why the technique works at all.

What the first run feels like

Expect a download measured in gigabytes, then a pause while the model loads into memory. After that, replies appear one token at a time rather than all at once.

Two things surprise most people. First, the model has no memory between conversations unless the app gives it one. Second, there is a limit to how much text it can consider at once, called the context window, and filling it uses extra memory on top of the model itself.