Choosing your first local model
By the end of this guide you will have picked one model to download first, and you will know why you picked it rather than one of the hundreds of others. It is written for anyone who has decided to run a model locally but has not yet chosen one. It sits alongside the other guides.
The short version: your hardware sets the ceiling, your task sets the shape, and the first model you run should be small enough that you get it working before you start optimising.
What this assumes
Three things worth having to hand.
- A rough idea of what your machine can hold. That means the VRAM on your graphics card, or on an Apple Silicon machine the unified memory shared between CPU and GPU. Hardware and the GPU pages have the figures if you are not sure what your machine is capable of.
- A way to run the model. Picking the app first helps, because the engine inside it decides what file formats you can use. The runtimes index covers the common options.
- A sense of the vocabulary. Parameters describe a model's size, quantization describes how compressed its weights are, and inference is the act of running it. You do not need to understand these deeply yet, but you will see all three on every download page.
Steps
-
Write down the job. "Summarise my notes", "help me write code", "answer questions about a pile of documents", "talk to me". Each of these rewards a different model. A general chat model is a reasonable default if the job is fuzzy, but if you already know it is code, choose a model trained for code.
-
Find your memory budget. Take the memory you have and subtract what the rest of the system needs. On a discrete card, the operating system and your desktop already hold some of the VRAM. On a unified-memory laptop, the model competes with everything else you have open. Treat the number you are left with as a hard ceiling, not a target. Our will this model fit in memory? guide walks through the arithmetic.
-
Pick a size class that fits comfortably. Models come in families, and a family usually ships several parameter counts. On a card with a large VRAM pool you can consider the mid and large members of a family. On a laptop, or on a card with a modest pool, start at the small end. The point of the first run is to see tokens appear, not to run the biggest thing you can technically load.
-
Pick a family with a track record. Llama, Mistral and Qwen are all widely used, widely documented and well supported across apps. Widely used matters more than you think for a first model: when something goes wrong, someone has already hit the same wall and written about it.
-
Prefer the instruction-tuned variant. Families usually publish a base model and one or more tuned variants. The base model continues text. The instruction-tuned one answers you. For a first model you want the second.
-
Choose a quantization one notch below the biggest that fits. Leaving headroom means the model still works when you raise the context window later. Our picking a quantization guide explains the ladder.
-
Read the model card before you download. It tells you the licence, the intended use, the context length and whether the weights are what you think they are. See reading a model card.
-
Run it, then change one thing. Get a boring short conversation working first. Then raise the context length, or step up a size class, or swap quantization, one change at a time, so you know which change caused which result.
Common mistakes
- Choosing the largest model your memory can technically hold, then finding it slows to a crawl as soon as the conversation gets long.
- Downloading the base model instead of the instruction-tuned variant and concluding the model is broken.
- Picking an obscure fine-tune first, so that every problem you hit is one nobody has documented.
- Changing model, quantization and settings all at once, which leaves you with no idea what helped.
- Ignoring the licence until after you have built something on top of the model.