How much VRAM do you need to run a local LLM?

To run a GGUF model you need enough memory for the weights, the KV cache and a little runtime overhead. This guide shows how to work out the number yourself and what fits on common GPUs.

The formula

Memory = file size + KV cache + runtime overhead.

The file size is the GGUF file you download, so it is known in advance. The runtime overhead is the working memory llama.cpp and the GPU driver need, usually somewhere around 0.5 GB. The KV cache depends on the model and on how much context you use.

How big is the KV cache?

For each token in the context, the model stores a key and a value for every layer. In an fp16 cache that is:

bytes per token = layers × KV heads × (key size + value size) × 2

A typical 8B model has 32 layers, 8 KV heads and a head size of 128, so it uses 32 × 8 × 256 × 2 = 131,072 bytes per token. At an 8K context that is about 1 GB. At 32K it is about 4.3 GB, and at 128K it is about 17 GB. Our model pages show each model’s own KV cost per token, read from its GGUF header.

What fits in common GPU sizes

With a 4-bit quant such as Q4_K_M and an 8K context, as a rough guide for recent models:

GPU memoryModels that fit at 4-bitBrowse
4 GBSmall models up to about 3B parametersModels for 4 GB
6 to 8 GB7B and 8B modelsModels for 8 GB
12 GBUp to about 14B modelsModels for 12 GB
16 GB14B models with room for contextModels for 16 GB
24 GBModels of 24B to 34B parametersModels for 24 GB
48 GB70B modelsModels for 48 GB

When the model does not fit

You have a few options, roughly in order of how much they cost you. Use a smaller quant, which trades a little quality for memory. Use a shorter context window, which shrinks the KV cache. Store the KV cache in lower precision if your runtime supports it. Or let llama.cpp keep some layers on the CPU: it works, but each layer moved off the GPU slows generation down.

Mixture-of-experts models

A mixture-of-experts model uses only a few experts for each token, which makes it faster than a dense model of the same total size. All of the experts still have to be in memory, though, so the file size is that of the full model. The GGUF file size on each model page already includes every expert.

Apple Silicon and shared memory

On a Mac the GPU and CPU share one pool of memory, so the total RAM is what counts. The system only lets the GPU use part of it, so leave several gigabytes for the operating system and your other apps.

Check any model on this site with its run check: choose your VRAM and context, and the table shows which quants fit. Start from all models or models that fit in 8 GB.

Frequently asked questions

How much VRAM do I need for a 7B or 8B model?

About 6 to 7 GB at Q4_K_M with an 8K context window. The file itself is roughly 4.4 to 4.9 GB, and the KV cache and runtime take the rest.

What happens if the model does not fit in VRAM?

llama.cpp can keep some layers on the GPU and run the rest on the CPU. It works, but it is slower, and the more layers on the CPU, the slower it gets. A smaller quant or a shorter context window is usually the better first step.

Does context length change how much memory I need?

Yes. The KV cache grows in proportion to the number of tokens in the context window. For a typical 8B model it is about 1 GB at 8K tokens and about 17 GB at 128K tokens, so very long contexts can need more memory than the model itself.

Do mixture-of-experts models need less memory?

Not for the weights. A mixture-of-experts model only uses a few experts per token, which makes it faster, but all experts must still be in memory, so the file size is that of the full model.