Local AI or cloud AI: when to use which
A local model on your GPU is private, free per token and works offline; a frontier cloud model knows more and reasons better. Most people need both. Here is how to split the work, with our measurements.
What a local model is good at today
On a 16 GB card, the best models we measured score 82.2% on translated world-knowledge questions in English and 96.8% on grade-school math, at 90 tok/s. That is enough for everyday writing, summaries, translation drafts, explaining code, and questions about your own documents.
- Private by construction. Your prompts and answers never leave the PC; useful for client documents, health or finance notes, code under NDA.
- No per-token bill. The cost is the electricity and the GPU you already have.
- Works offline, on a plane or behind a strict firewall.
- Predictable. The model doesn't change under you; you pick when to upgrade.
Where the cloud still wins
- Hard multi-step reasoning, large refactors across many files, and questions that need the newest knowledge.
- Very long contexts: a 16 GB card holds about 78k tokens next to the model (
--profile max); cloud models take far more. - Lower-resource languages at small sizes: quantization costs Thai, Hindi and Arabic more than English (13 vs 8 points at 2 bits on Qwen3.8-27B), so check the scores in your language.
Using both
localllm route can send a request to your own cloud API key when a rule says so (a prompt too long for
the local model, a language below the quality floor). It is off by default, the cost and destination are shown, and
every answer is labelled with where it came from. Keys stay in your OS keychain.
localllm savings shows how many tokens your PC answered this month and, at the prices you enter, what
they would have cost in the cloud.
One thing localllm will not do: pass a Claude or ChatGPT subscription login on to another tool. Use an API key; the providers' terms don't allow the other way and it risks your account.