What people use it for
Seven situations where it helps, with the commands. If yours isn't here, open an issue.
A private assistant on your own PC
The problem. You don't want every question to go to a cloud service, and you don't want to spend an evening on setup.
With localllm. pip install make-localllm-easier, then localllm. The chat page opens when the model is ready. After the first download it works without internet.
Result. A model that answers at 50-90 tok/s on a 16 GB card, with your data staying on the disk.
Working in Thai, Japanese, Hindi, Arabic or another language
The problem. Most model comparisons are in English. A model that is fine in English can be clearly weaker in your language, and quantization makes the gap larger.
With localllm. localllm keeps scores per language and picks for yours. localllm eval --langs en,ja measures any model on your hardware, in a few minutes per language.
Result. On our test PC, Chinese knowledge questions go to Qwen3.8 (+5.5 points), and Thai math to gemma-4 (89.6 vs 87.2).
A local API for development
The problem. You want an endpoint for prototypes, tests or internal tools without per-token bills.
With localllm. localllm serve gives you OpenAI, Anthropic, Ollama and Gemini formats on one port, with tool calling. Set the base URL in the SDK you already use.
Result. Code written against the OpenAI or Anthropic SDK runs locally unchanged.
A coding agent on a local model
The problem. Agent tools like Hermes, OpenCode or Aider can run up large API bills, and you would like to try a local model for part of the work.
With localllm. localllm serve --profile max, then localllm connect hermes (or opencode, claude-code, codex, aider). --undo restores your old config.
Result. The agent talks to your GPU. How well each agent works with these models is still being measured (#96).
Your notes in Obsidian, kept on your PC
The problem. AI plugins for notes usually send your text to a cloud service, and it is easy to miss which setting does.
With localllm. localllm connect obsidian --path YOUR_VAULT makes the local model the default in Copilot for Obsidian and lists anything that would still reach the cloud, such as a cloud embedding model for search.
Result. Chat about your notes with the model on your GPU; --undo puts the plugin's old settings back.
An AMD or Intel graphics card
The problem. Guides and tools often assume NVIDIA and CUDA. On Windows without Resizable BAR, llama.cpp on Vulkan can be up to 1.7x slower.
With localllm. localllm uses the Vulkan builds of llama.cpp and applies the host-visible memory fix for models where it measured faster. localllm tune checks it on your PC.
Result. Our own test PC is an RX 9070 XT on Windows 11, so this is the setup we know best.
A PC that is also used for other things
The problem. Local models can take most of your RAM and keep the GPU busy, which gets in the way of games and other apps.
With localllm. Caches are sized to your RAM (about 2.2 GB in a long chat). With routing on, models unload after 15 minutes without a message and load again on the next one.
Result. The rest of your RAM stays free, and the GPU is free when you are not chatting.
doctor tells you what would fit, but no small model has been measured for the catalog yet
(#38).Found yours?
$ pip install make-localllm-easier $ localllm