From pip install to a running model

1. It looks at your PC

llama.cpp lists the GPUs it can use (Vulkan on Windows and Linux, Metal on macOS). localllm takes the one with the most memory, skipping integrated graphics, reads installed and free RAM, and looks up your card's memory bandwidth to estimate speed. It also reads your system language. Run localllm doctor to see all of it without downloading a model.

2. It picks a model

From the catalog it keeps the models that fit on the card with room for an 8k-token conversation. A MoE model that doesn't fit whole can keep some experts in system RAM. Among those, it takes the best measured score in your language, and on a near tie (under 2 points) the faster one. You can override it: localllm --model KEY.

3. It downloads and checks the files

llama.cpp comes from build b11476, the one the numbers were measured on. The model comes from a pinned commit on Hugging Face. Each file is checked against a SHA-256 stored in the code. Downloads resume if they stop. Everything goes to ~/.localllm/. The first time, before the multi-GB model download, it shows what your PC can run and asks.

4. It starts the model

llama-server starts on a private port with the settings from the catalog: flash attention, an 8-bit KV cache, one slot, caches sized to your RAM, MTP drafting where it helps. In front of it, a small gateway serves the chat page and the four APIs on port 8080. Your browser opens the chat page.

$ localllm
[localllm] loading gemma4-26b-a4b-qat on AMD Radeon RX 9070 XT ...
[localllm] ready. chat: http://127.0.0.1:8080   API: http://127.0.0.1:8080/v1 (OpenAI), ...

5. You use it

• Chat in the browser, or localllm chat in the terminal (/save, /think, Ctrl+C stops an answer).
• Point an app at http://127.0.0.1:8080/v1 with any API key.
• Run localllm tune once to measure the speed settings on your PC.
• Want saved chats? localllm --ui. Long documents or an agent? --profile max.

What you don't have to do

• Pick a quantization, or find out a 2-bit file lost 13 points in your language.
• Work out whether a model fits with your context length.
• Find the llama.cpp flags that matter, or notice that the prompt cache is eating your RAM.