A shallow wooden crate filled with smooth river stones stacked to just below its rim, leaving a band of empty space

Will this model fit in memory?

By the end of this guide you will be able to look at a model on a download page and form a reasonable estimate of whether it will run on your machine, before you spend an hour downloading it. It is for anyone who has hardware in front of them and a model in mind. It is one of the guides.

There is no single number to memorise. What you need is a mental model of the three things competing for the same pool of memory.

What this assumes

  • How much memory you actually have available. On a discrete card that is the VRAM figure, minus whatever your desktop and other applications are already holding. On Apple Silicon it is the unified memory pool, which the system will only let a process claim part of. The hardware section and the individual GPU pages list capacities.
  • The model's parameter count and its quantization. Both appear in the file name of most distributions.
  • Roughly how long your conversations or documents will be, in tokens. This drives the second component and is the one people forget.

Steps

  1. Start with the weights. The weights are the bulk of the model and their size is set by two things: how many parameters there are, and how many bits each parameter is stored in. Twice the parameters at the same precision is roughly twice the size. The same parameters at half the precision is roughly half the size. You do not need a formula, because the file you are about to download has its size printed next to it, and that is the most honest estimate of the weights you will get.

  2. Check the weights fit with room to spare. If the file size is already close to your available memory, stop there. The weights are the floor, not the total. Something has to sit on top of them.

  3. Add the KV cache. During generation the runtime stores the model's intermediate state for every token in the conversation. That is the KV cache, and it grows roughly in proportion to how many tokens are in play. A short question costs almost nothing. A long document, or a chat you have been adding to all afternoon, can cost a large fraction of what the weights cost. This is why a model that loaded fine can fall over an hour later.

  4. Decide your real context length. The context window a model supports and the context window you configure are different things. Apps generally let you cap it. Setting a smaller cap is the most effective lever you have for making a model fit, and for everyday use you will rarely notice.

  5. Add runtime overhead. The runtime itself needs working memory: buffers for the current computation, whatever the graphics driver reserves, and on some setups a second copy of part of the model during loading. Treat this as a fixed slice off the top rather than something you can compute, and leave a margin for it.

  6. Branch on your hardware class. On a card with a large VRAM pool, you want everything resident on the card, so the sum of weights, cache and overhead must stay under the card's capacity. If it does not, you either drop a size class or lower the precision. On a unified-memory laptop the pool is larger but shared, so the failure mode is different: the model loads, then the machine starts swapping and everything gets slow. Leave more headroom there than the raw capacity suggests. On a machine where the model does not fit on the GPU at all, some engines will place part of it in system RAM. It will run. It will run at the speed of the slowest part.

  7. Verify by watching, not by trusting. Load the model, feed it a long document, and watch memory use climb. The number at load time is the optimistic case. The number after a long session is the one that matters.

Common mistakes

  • Comparing the download size against total memory and stopping there, with no allowance for cache or overhead.
  • Configuring the maximum context the model supports because it is offered, rather than the context you need.
  • Forgetting that the display, the browser and the compositor are already holding VRAM.
  • Assuming that because a model loaded, it will stay loaded through a long session.
  • Treating partial offload to system memory as a fit. It is a fallback, and it costs far more speed than the memory saving is worth.

Related