Which local LLM should I run on a 16 GB GPU?
Short answer: a ~27B dense model at about 3.5 bits, or a 26B MoE at 4 bits. Measured on an RX 9070 XT, with scores per language.
The short answer
On a 16 GB card, the two models we measured as best are gemma-4-26B-A4B QAT (4-bit, 13.3 GB) and Qwen3.8-27B at ~3.5 bits (12.2 GB). Both fit with a long context: about 78k tokens for gemma-4.
| Qwen3.8-27B Q3 | gemma-4-26B-A4B QAT | |
|---|---|---|
| File size | 12.2 GB | 13.3 GB |
| Decode speed (RX 9070 XT) | ~50 tok/s | ~90 tok/s |
| English knowledge | 81.5% | 82.2% |
| Chinese knowledge | 76.2% | 73.5% |
| Math, English / Thai | 94.4 / 87.2 | 96.8 / 89.6 |
| Thai local exams | 67.1% | 65.7% |
For most languages they are within 2 points of each other, which is inside the benchmark margin, so gemma-4 wins on speed. Qwen3.8 is clearly ahead on Chinese, Spanish and Japanese knowledge questions.
Why not a bigger model at 2 bits?
Because it costs more than it gives. Qwen3.8-27B at 2 bits (7.8 GB) scored 8-13 points lower than at ~3.5 bits, and the lower-resource languages paid the most: Hindi, Arabic and Thai lost 13. A 177B MoE squeezed to 1.6 bits scored below the 27B at 3 bits.
What about 70B?
A dense 70B model needs about 42 GB at 4-5 bits. On a 16 GB card it only fits at ~1.6 bits, where quality is gone. MoE models with experts in system RAM are the realistic route to bigger models; we are measuring them (research notes).
Check your card
localllm doctor lists which model sizes fit your GPU, at which quantization, how fast they should run,
and the best measured model for your language.