Install and run it
Two commands. The first run downloads llama.cpp (~30 MB) and one model (8-14 GB), once.
Install
- Have Python 3.9 or newer (
python --version). - Install:
pip install make-localllm-easier - Run
localllm. The chat page opens when the model is loaded.
$ pip install make-localllm-easier $ localllm
The first run shows what your PC can run and asks before the 8-14 GB model download.
localllm doctor shows the same without downloading a model.
No Python?
From version 0.7.1, every release has a single localllm file for Windows,
macOS and Linux, and an AppImage. Download it and run it from a terminal. winget and Homebrew packages follow.
Then
localllm chat # chat in the terminal localllm doctor # what this GPU can run, speed, RAM localllm list # every measured model and its scores localllm serve # API only, no browser localllm tune # measure the fastest settings once localllm eval # score a running server in your language localllm report # share your results (anonymous) localllm connect hermes # point an agent at it localllm --ui # chat page with saved chats localllm savings # tokens answered locally this month
Using it from an app: base URL http://127.0.0.1:8080/v1, any API key.
API docs and examples.
Common problems
pip is not found
Install Python 3.9 or newer from python.org (on Windows, tick "Add python.exe to PATH"), then use python -m pip install make-localllm-easier.
localllm is not found after installing
pip put it in a folder that is not on your PATH. python -m localllm works the same.
"no measured model fits this GPU yet"
The catalog's smallest model needs about 10 GB of VRAM. Run localllm doctor to see what sizes fit; smaller models are tracked in #38.
It is slower than the numbers here
Run localllm tune. On Windows, if doctor warns that the model spilled into system RAM, close other GPU-heavy apps. On AMD/Intel, check Resizable BAR in your BIOS; see this guide.
The first answer takes a long time
The first run of a new llama.cpp build compiles GPU shaders once (about 15 seconds). Later starts are fast.
Where are the files?
~/.localllm/: models, llama.cpp, tune results, saved chats. Delete the folder to remove everything. Use LOCALLLM_MODELS to point at a folder of GGUF files you already have.
Windows or macOS warns about the downloaded file
The single-file builds are not code-signed yet. Windows: More info > Run anyway. macOS: right-click > Open once. Or install with pip, which has no warning.
Something else? Open an issue with the output of localllm doctor.
Help the next person
If it ran on your PC, localllm tune, localllm eval and localllm report take
10-30 minutes and add your GPU to the leaderboard. The report has no names, paths or keys,
and you read it before you submit it.