An open laptop glowing faintly on a small table by a window, seen from across a dim room at dusk

Running models on a laptop

By the end of this guide you will be able to pick models and settings that run well on a laptop, and you will understand why the same model that flies on a desktop can crawl on a portable machine. It is for people who do not have a tower and do not want one, and for people who want a workable setup while travelling. It is one of the how-tos on the guides page.

What this assumes

Laptops differ from desktops in three ways that matter for inference: memory is capped at whatever you bought, cooling is limited, and power is limited further still when you are unplugged.

Useful to know first:

  • How much memory your machine can give a model. On a discrete laptop GPU that is the VRAM figure. On an Apple silicon machine it is a share of unified memory, minus whatever the rest of the system is using.
  • That memory bandwidth in laptops is usually lower than in desktop cards, so generation speed is lower even when a model fits comfortably.
  • What quantization does, since it is the main lever you have. See picking a quantization.
  • That a longer context window costs memory through the KV cache, separately from the model weights.
  • The general capability picture in the hardware section, and the M3 Max page if you are on Apple silicon.

Steps

  1. Find your real memory ceiling. Open your usual working set of applications first, then look at how much memory is actually free. That figure, not the number on the spec sheet, is what a model has to fit inside.
  2. Choose a model a size class below what the ceiling suggests. Leaving headroom keeps the machine responsive and stops the system swapping when something else wakes up. Choosing your first local model walks through the size classes.
  3. Pick a quantization that fits with room to spare rather than the largest one that technically loads. On a constrained machine the smaller file is usually the better experience, because it also reads faster from memory.
  4. Set a modest context window to start with. Ask for the length you need for real work, not the maximum the model advertises, and raise it only if you have memory left over.
  5. Run a sustained test, not a short one. Generate continuously for several minutes and watch what happens. Laptops boost hard at first and then settle, so the first few seconds tell you nothing about how the machine behaves under load.
  6. Fix the thermal side before blaming the model. Keep the machine on a hard surface, keep the vents clear, and expect the fans to run. If it throttles badly, step down a size class rather than fighting it.
  7. Decide how you want it to behave on battery. Many machines cut power limits sharply when unplugged, so either accept slower generation or keep heavy work for when you are plugged in.
  8. Match the workload to the machine. Text generation is the most laptop friendly. Image generation is heavier but workable at smaller output sizes, see image getting started. Video generation is the least laptop friendly, so check video before planning around it.

Common mistakes

  • Reading the spec sheet memory figure as available memory. The operating system, your browser and your editor all take their share first.
  • Loading the biggest model that fits and calling the setup done. The machine will be usable for one task and unusable for everything else at the same time.
  • Treating a short burst of fast generation as the steady state. Sustained load is where a laptop reveals its limits.
  • Requesting a long context by default. It quietly consumes memory you would rather spend on a better model.
  • Trying to do video work on a thin laptop and concluding that local AI does not work. The workload was wrong, not the idea.

Related