How much RAM does a local LLM need?
With the model on the GPU, about 2.2 GB of system RAM, if the caches are sized for one user. With llama-server's defaults it grows to about 9 GB in a long chat.
Measured: a 30-turn chat
Same model, same questions, RX 9070 XT, 32 GB RAM. System RAM used by the server after the 30th turn:
llama.cpp defaultslocalllm
Speed was the same in both runs (35.2 tok/s for Qwen3.8, ~85 for gemma-4).
Where the RAM goes
- Prompt cache. llama-server keeps up to 8 GiB of earlier prompts in RAM by default
(
--cache-ram 8192). Useful for many users; for one user, 0.5-2 GB is enough. - Context checkpoints. 32 per slot by default. Each is a copy of part of the model state.
- Memory-mapped weights. With mmap the whole file can stay in the working set. Reading weights straight into
VRAM (
--load-mode none) keeps them out of RAM. - Embeddings and host buffers. Usually 0.3-0.5 GB each.
The settings localllm uses
| Installed RAM | --cache-ram | --ctx-checkpoints |
|---|---|---|
| up to 16 GB | 512 MiB | 2 |
| up to 32 GB | 1024 MiB | 4 |
| up to 64 GB | 2048 MiB | 8 |
| more | llama.cpp defaults | |
And if the model doesn't fit the GPU?
Then RAM holds part of the model. For a MoE model like gemma-4-26B-A4B, keeping some layers' experts in RAM still gives 45 / 36 / 31 tok/s with 8 / 13 / 18 layers off the GPU (for 12 / 10 / 8 GB cards). A dense model with layers in RAM is much slower, under 5 tok/s.
localllm doctor prints the estimate for your PC: how much RAM the picked model uses and how much is
left for other apps.