What localllm does, and what it doesn't

Model choice from measurements

Each model in the catalog was run through the same tests on the same PC: Global-MMLU-Lite (400 questions translated into many languages), INCLUDE (real exams from 44 countries), ThaiExam, MGSM math word problems and FLORES translation. localllm takes the scores in your language, keeps the models that fit your card with an 8k-token context, and picks the best. Gaps under 2 points count as a tie, because the benchmark margin is about ±4, and a tie goes to the faster model.

The catalog is small on purpose: three files today. A model is added only after it is measured. See every score.

Per-language results

Quantization does not cost every language the same. On Qwen3.8-27B, going from ~3.5 bits to 2 bits cost English, Chinese and Spanish 8-9 points, and Hindi, Arabic and Thai 13. localllm keeps the 2-bit file only for cards too small for anything else and says so in doctor.

Memory: less RAM, same speed

With the model fully on the GPU, localllm uses about 2.2-2.4 GB of system RAM in a 30-turn chat, where llama-server's defaults reach 9.2-9.3 GB. Weights are read straight into VRAM instead of being memory-mapped, which also keeps the working set small when a MoE model has to keep some experts in RAM.

When a model spills out of VRAM into shared memory on Windows, answers get several times slower with no message from the driver. localllm checks for it after loading and tells you.

Speed settings that were measured

• Multi-token prediction for Qwen3.8 (drafting 2 tokens: about 40% faster; 3 was slower).
• The Vulkan host-visible memory fix for AMD and Intel cards without Resizable BAR, applied per model: 1.64x faster on Qwen3.8-27B, 7-9% slower on gemma-4, so it is not on for everything.
• localllm tune measures these on your PC in a few minutes and keeps a setting only if it is at least 1.1x faster.

One endpoint, four APIs

http://127.0.0.1:8080 answers OpenAI (/v1/chat/completions, /v1/responses, embeddings), Anthropic (/v1/messages), Ollama (/api/chat) and Gemini (generateContent) requests, including tool calls, streaming and images with --vision. Existing SDKs only need a new base URL. localllm serve --host 0.0.0.0 shares it on your network behind an API key.

Two profiles: lean or max

lean (the default) keeps an 8k-token context and caches sized so system RAM stays around 2.2 GB. max (--profile max) spends what the PC has: a context that fills the free VRAM (about 78k tokens on a 16 GB card) and a larger prompt cache, for long documents and coding agents. localllm doctor shows both for your PC. Same model, same answers; only memory changes.

A chat page with saved chats

localllm --ui opens our own chat page instead of llama-server's: chats saved as plain files on your disk, a model picker when you run several models, the speed of each answer, and live VRAM and RAM with a warning when the model spills out of the GPU. The page loads nothing from the internet. Shared on your network, it asks for the key printed in the terminal.

Agents and notes: localllm connect

Writes the config for Hermes Agent, OpenCode, Claude Code, Codex CLI, Aider or Copilot for Obsidian so they use the local model, backs up the old file, and --undo puts it back. For Obsidian it also tells you what would still go to a cloud service.

See what running locally saves

localllm savings counts the tokens this PC answered this month, how much of each prompt came from the cache, and, at the cloud prices you enter, what that would have cost. No prices are built in: they change, and a guess would be worse than none.

Optional extras

• Routing between local models (--models auto): each message goes to the model with the best score for its language and task. Off by default: on a 16 GB card a swap costs 4-9 seconds.
• Your own cloud key (localllm route): off by default. When you turn it on, a rule decides when a request goes to the cloud, and every answer says where it came from.
• Benchmarks (localllm eval): score any OpenAI-compatible server in your language, including a long-context test that hides numbers deep in 8k-64k-token documents (--suites longctx) and news summaries in 24 languages (--suites summarize).
• Energy per answer (localllm tune): the GPU's own power reading while it answers, shown as joules per token in doctor. AMD and NVIDIA; --prefer-energy keeps settings that save more than 15% energy for less than 5% speed.
• Safe upgrades: llama.cpp is pinned to the build the numbers were measured on. A new build is smoke-tested on our PC first, and doctor says when yours differs.

What it doesn't do (yet)

• No desktop app. The interface is a browser page, the terminal and the API.
• No numbers yet from the long-context test for the catalog models; they are queued.
• No models for cards under ~10 GB in the catalog yet. Measuring them is issue #38.
• Only Windows with an AMD card is tested by us. Everything else depends on community reports.