The short answer: VRAM depends on model size, quantization, and context
There is no single VRAM number that works for every local LLM. The practical requirement is the sum of model weights, KV cache, runtime overhead, and safety margin. A small Q4 model can run on modest hardware, while a larger Q8 model with long context can require much more memory. This is why two users with the same GPU can get different results if one wants short chat and the other wants long-context coding or RAG.
As a practical rule, 6GB is entry level, 8GB is usable for small models, 12GB is a more comfortable desktop baseline, 16GB gives better headroom, 24GB is a strong consumer tier, and 48GB or more opens the door to larger models and higher quantization. Apple Silicon unified memory is a separate category because CPU and GPU share the same memory pool, but it still needs room for macOS, apps, and runtime overhead.
Model weights usually dominate the memory budget
The largest part of the memory budget is usually the model weights. FP16 weights are large, while quantized GGUF files reduce size by storing weights with fewer bits. A 7B model in Q4 can fit into much less memory than the same model in FP16 or Q8. This is the reason quantization is central to local LLM recommendations: it changes what is realistically runnable on consumer GPUs.
However, file size alone is not enough. Some models have different architectures, MoE active parameter behavior, tokenizer settings, and context defaults. A model file that appears to fit on paper may still be tight once the runtime reserves memory. The recommendation should include a conservative margin instead of treating the file size as the entire requirement.
KV cache grows with context length
KV cache is the memory used to store attention information for tokens already processed. It grows with context length, batch settings, layer count, hidden size, and implementation details. This is why a model that works at 4K context can become unstable or slow at 16K, 32K, or higher context. Long context is not free just because the model card advertises it.
For users, this means the right VRAM target depends on workload. Simple chat can leave context moderate. Coding across many files, summarizing long documents, and RAG can push memory requirements higher. If long context is the priority, a smaller or more strongly quantized model may be a better fit than a larger model that consumes nearly all VRAM with weights alone.
Realistic GPU tiers for local LLMs
At 6GB, expect small models and conservative context. At 8GB, 3B to 7B Q4 models become realistic, but headroom is limited. At 12GB, many 7B and some 14B quantized models become more practical. At 16GB, the same models can use better quantization or more context. At 24GB, strong 14B and some larger quantized models become realistic for daily use. At 48GB and above, larger local experiments become much easier.
These tiers are not promises. Speed depends on memory bandwidth, GPU backend, CPU, RAM, drivers, and inference tool. NVIDIA CUDA support is usually the broadest in local inference tools. AMD can work well in supported stacks but may need more attention to backend compatibility. Apple Silicon can use unified memory effectively, but the usable model budget is still lower than the headline RAM amount because the system shares that memory.
When partial offload helps, and when it hurts
If a model does not fully fit in VRAM, some tools can offload part of the model to CPU memory. This can make a model load, but it often reduces speed because data has to move between GPU and system memory. On discrete GPUs, PCIe and CPU memory bandwidth can become bottlenecks. On unified memory Macs, the penalty is different, but memory bandwidth and system pressure still matter.
Partial offload should be presented honestly. It can be useful for experimentation, occasional long answers, or testing a larger model, but it should not be the default recommendation for a normal user who expects interactive speed. Local LLM should prefer full-GPU recommendations when possible and label partial offload separately when the model is technically runnable but less comfortable.
FAQ
Is 8GB VRAM enough for a local LLM? Yes, for small and carefully quantized models. It is not enough for every model, and long context will reduce the usable range.
Is 12GB VRAM enough? It is a reasonable starting point for many 7B models and some 14B quantized models, especially with balanced settings.
Does more VRAM always mean better answers? Not always. More VRAM expands the candidate set, but model quality, tuning, quantization, and use case still determine the best recommendation.
How can I estimate my own setup? Enter your VRAM, RAM, operating system, and use case into Local LLM. The tool applies model size, quantization, KV cache, and margin rules to current model data.