Running AI models on your own machine — no cloud, no API keys, no monthly bill — used to mean a workstation full of GPUs. Not anymore. Modern open models like Gemma 4 12B run comfortably on a normal laptop. This guide explains the why and the how so you can pick a model and start in minutes.
Why run AI locally?
- Privacy — your prompts and files never leave your computer.
- Offline — it works on a plane, in a workshop, anywhere.
- No cost per token — once it's downloaded, it's free to use as much as you like.
- Control — pick the exact model and settings; nothing changes under you overnight.
The trade-off: a local model is usually smaller than a frontier cloud model, so it's a little less capable — but for chat, coding help, summarising, and tinkering, today's small models are genuinely good.
The two numbers that matter: parameters and quantization
Parameters (the "B" in 12B = billions) roughly tracks how capable a model is. Bigger is smarter but heavier.
Quantization shrinks a model so it fits in less memory by storing its weights at lower precision. You'll see tags like Q4_K_M or Q8_0:
Q8— almost identical to the full model, largest file.Q4_K_M— the sweet spot for most people: ~half the size, minimal quality loss.Q2/Q3— smallest, noticeably lower quality; only if you're tight on memory.
A rough rule of thumb for how much memory a model needs:
| Model size | Q4 file (approx) | Comfortable on |
|---|---|---|
| 1–3B | 1–2 GB | any laptop |
| 7–8B | 4–5 GB | 8 GB RAM / GPU |
| 12–14B | 7–9 GB | 16 GB RAM / GPU |
| 27–30B | 16–20 GB | 24 GB+ GPU |
Rule of thumb: pick a model whose Q4 file is comfortably smaller than your free RAM (or VRAM if you have a GPU). Leave a few GB of headroom for the context window.
VRAM vs RAM (do I need a GPU?)
- With a GPU, the model loads into VRAM and runs fast. VRAM is the limit.
- Without a GPU, it runs on the CPU using normal RAM — slower, but totally usable for smaller models. Apple Silicon Macs are a special case: their unified memory acts like fast VRAM, so a 16 GB Mac runs 12B models nicely.
How to actually run one
Two friendly starting points — pick whichever you like:
- Ollama — a clean command line + local API. Great for developers and scripting. One command to download and chat.
- LM Studio — a polished desktop app. Browse models in a store-like UI, chat in a window, and flip on a local API server. Great if you prefer a GUI.
Both download models for you and handle the technical bits. Start there, grab a small model (try a 7–8B first, then Gemma 4 12B if you have 16 GB), and you're running AI locally.