Free Tool
Local AI Hardware Calculator
Find out exactly how much VRAM and RAM you need to run any open-source LLM locally.
2K4K8K16K32K64K
12345678
Model VRAM
— GB
KV cache
— GB
Total VRAM needed
— GB
System RAM
— GB
Recommended hardware
What is quantization?
Quantization reduces model weight precision from 16-bit floats to 4–8 bits. Q5_K_M is the sweet spot: ~38% smaller than FP16 with barely noticeable quality loss.
VRAM vs RAM
If your GPU doesn't have enough VRAM, the model offloads layers to system RAM — inference slows dramatically. Apple Silicon uses unified memory, so RAM and VRAM are the same pool.
Context & KV cache
Every token in the context window occupies VRAM in the KV cache. Longer contexts or more concurrent users multiply this cost linearly.
Inference frameworks
Ollama
One-command setup. Best for local single-user use on Mac, Linux, Windows. Supports GPU offloading automatically.
llama.cpp
Bare-metal C++ inference. Maximum performance, runs on CPU-only too. Great for ChibyooShell SSH sessions on remote servers.
vLLM
Production serving with PagedAttention. Best for multi-user setups and large models on NVIDIA GPUs.
LM Studio
Desktop GUI with local OpenAI-compatible API. No terminal needed. Ideal for beginners.
FAQ
With Q4 quantization, a 70B model needs ~40 GB VRAM. That requires an NVIDIA A100 80GB, two RTX 3090/4090 in NVLink, or an Apple M2 Ultra (192 GB unified memory). For home use, a 13B or 34B model at Q5 is more practical.
When the entire model fits in VRAM, CPU is nearly irrelevant — the GPU handles all computation. CPU matters only for tokenization and when layers are offloaded to RAM due to insufficient VRAM.
Yes — Apple Silicon's unified memory architecture means RAM and VRAM share the same pool. An M2 Pro with 32 GB can run 13B–27B models smoothly. The M2 Ultra with 192 GB can handle 70B+ models. Energy efficiency is also excellent.
Q5_K_M is the recommended default — it reduces model size by ~38% compared to FP16 with minimal quality loss. Q4_K_M is a good choice if you're tight on VRAM. Avoid Q2 for any task requiring coherent reasoning.
The KV cache grows linearly with context. A 7B model at 4K context uses ~1 GB for KV cache; at 32K context that becomes ~8 GB. For very long contexts, consider reducing model size to keep total VRAM in budget.
Managing remote servers where you run your AI? ChibyooShell gives you SSH, monitoring, Docker control, and SFTP — all from your iPhone.
Download ChibyooShell