Run a coding agent on a local model (Hermes, OpenCode, Aider, Claude Code)
Three commands point an agent at the model on your GPU, with a context large enough for agent prompts. What works, what to expect, and how to go back.
Set it up
localllm serve --profile max # a context that fills the GPU: agents send long prompts localllm connect hermes # or opencode, claude-code, codex, aider, cline hermes # use the agent as usual
connect changes only the base URL and the model name in the agent's own config file, keeps a backup,
and localllm connect hermes --undo restores it. On a 16 GB card --profile max gives about
78k tokens of context with gemma-4-26B-A4B; Hermes asks for 64k.
What to expect
- Speed: about 90 tok/s decode on an RX 9070 XT; long prompts take longer the first time and are re-used from the prompt cache after that.
- Quality: fine for small edits, explanations, tests and scripts. Large multi-file refactors are where a frontier cloud model is still clearly better.
- Agent-by-agent results on our models are still being measured (#96).
Check the prompt cache and the savings
localllm savings shows the share of prompt tokens that came from the cache. Agents resend the same
instructions on every call, so this should be high; if it is low, the agent's prompt changes between calls and every
call is slower than it needs to be.