Fine-tuning local LLMs
Fine-tuning is where you stop using a model as shipped and start changing its weights. It is the most demanding thing you can do locally, and the thing most often reached for when something simpler would work. This page on local LLMs covers when it earns its keep and what it takes to run.
Prompting, retrieval, then training
There are three ways to make a model behave the way you want, and they cost wildly different amounts of effort.
Prompting changes the instructions you send. It is free, instant and reversible, and it handles more than people expect. A carefully written system prompt with a few worked examples covers most formatting and tone requirements.
Retrieval changes what the model sees. You keep your documents outside the model, search them at question time, and paste the relevant passages into the context window. This is the right answer when the problem is knowledge rather than behaviour. Training a model on your internal wiki is a poor way to teach it facts: they go stale and you cannot tell what it absorbed. Retrieval keeps the source of truth editable.
Fine-tuning changes the model. Reach for it when you want a consistent style across thousands of calls, when you need a narrow task done reliably by a small model, or when your domain has vocabulary the base model keeps getting wrong. See the glossary entry for the definition.
The honest test is whether you have tried the first two properly. If prompting fails completely, training on a few hundred examples rarely rescues it.
Full fine-tuning versus adapters
Full fine-tuning updates every weight in the model. It needs memory for the weights, the gradients and the optimiser state, which together run to several times what it costs to simply run the model. For anything but small models that puts it out of reach on a single consumer card.
Adapters changed that. Rather than updating the original weights, you freeze them and train a small set of new ones alongside. LoRA is the method almost everyone uses: it inserts low-rank matrices into the model's layers, trains only those, and leaves the base untouched. The resulting file is small enough to share and swap, so you can keep several adapters for one base model and load whichever the task needs.
Quantized variants go further, keeping the frozen base compressed while training the adapter at higher precision. Between adapters and quantization, most hobbyist fine-tuning now happens on hardware that could not have hosted full training at all.
What it costs in memory
Training always needs more memory than inference on the same model, and the gap is what to plan around. Running a model needs room for the weights plus the KV cache. Training needs room for the weights, the activations for each example in the batch, and the gradients and optimiser state for every parameter being updated. Adapters shrink the last part dramatically, but activations still scale with batch size and sequence length.
So if a model only just fits for inference on your machine, it will not fit for training. The GPU pages cover what specific cards hold, and the enthusiast build guide targets the capacity that makes local training comfortable rather than a fight. Shorter sequences and smaller batches are the first dials to turn when you are close.
Data and evaluation
Fine-tuning is mostly a data problem. A few hundred to a few thousand examples that genuinely represent the task beat a much larger set scraped together carelessly. Consistency matters more than volume: if your examples disagree about format or tone, the model learns the disagreement. Match the format to the use, edge cases included.
Hold back a slice of your data before you start and never train on it, or you cannot tell improvement from memorisation. Evaluate on the thing you actually want rather than the training loss: run realistic prompts through the base model and the tuned model and compare outputs side by side. Watch for the common failure where the model nails your narrow task and loses general capability it used to have.
Keep base models and adapters apart, with notes on which data produced which adapter. Organising model files covers a layout that survives a few rounds of this.