Local LLM agents
An agent is a model in a loop with tools. That is the whole idea, and everything else about agents on local LLMs follows from it. The model is asked to do something, it decides on an action, something outside the model carries that action out, the result is fed back in, and the loop runs again until the task is done.
The loop
An agent has four moving parts.
- A goal, given once as an instruction in plain language.
- Tools the model may call: read a file, run a search, query a database, execute code.
- A loop that runs the chosen action and appends the result to the conversation.
- A stopping condition: the model declaring it is finished, or a limit on steps or time.
The model itself does nothing but produce text; it never touches a file or a network. The scaffolding reads its output, recognises a request for a tool, performs it, and writes the outcome back into the context. An agent's quality comes as much from that scaffolding and the tools you expose as from the model. You can build a working one with a loop, a few functions, and a prompt describing them.
Why local matters here
Agents make the case for local models better than chat does. A conversation is a handful of calls; an agent loop can be dozens, and that changes three things.
Privacy. A useful agent is an agent with access. It reads your files, your code, your notes, and with a hosted model every read becomes an upload. Locally the data never leaves the machine, which is often the difference between an agent you can point at real work and one you can only demo. Keeping everything offline covers checking that.
Cost per step. Metered per-token pricing punishes loops, because every step resends the accumulated context. Locally the marginal cost is electricity and time, which makes it reasonable to let an agent retry, explore a dead end and back out.
Latency and control. No network round trip per step, no rate limits, no model changing under you between runs. That matters once you have tuned prompts against a model's quirks.
What to expect from small models
Local agents work, with caveats. Small models are decent at single tool calls and much weaker at long chains. They lose the thread after several steps, repeat actions they have already taken, or declare success without having done the work. Keep loops short and tasks narrow. An agent that does one well-defined job with three tools is far more reliable than a general assistant with twenty.
They also follow tool descriptions literally, so vague descriptions produce wrong calls. Write them as documentation for someone who will never ask a clarifying question, with an example of correct use.
Error recovery is the weakest area. When a tool returns an error, a capable model reads it and adjusts; a small one often retries the identical call. Handle that in the loop: detect repeats, cap retries, and return short, specific error messages rather than stack traces.
Larger models are better at all of this, which makes agents one of the clearest reasons to run the biggest model your hardware allows. Will this model fit in memory and the hardware pages find that ceiling.
Context and structured output
Two technical constraints shape every local agent.
The first is the context window. Each iteration adds the action and its result to the conversation, and tool output is verbose, so a few file reads can fill a window. Long contexts also cost memory, because the KV cache grows with them, and they slow generation down. Trim aggressively: summarise old steps, truncate tool output, drop results the agent has finished with.
The second is structured output. The loop has to parse the model's intent reliably, so the model must emit something machine-readable, typically JSON naming a tool and its arguments. Small models drift here, adding commentary around the JSON or inventing fields. Most runtimes support constrained decoding, which forces output to match a schema during generation rather than validating it afterwards. Use it. The runtimes index notes which engines offer it.