GGUF quantization explained
A quantized model stores its weights with fewer bits so that it fits in less memory. GGUF files name the method in the file name, such as Q4_K_M or IQ3_XS. This guide explains how to read those names and which one to choose.
Reading a quant name
Take Q4_K_M. The Q4 means roughly 4 bits per weight on average. The K says it is a k-quant, a block-wise scheme that stores weights in small groups with their own scale, and that uses a mix of precisions across the model. The last letter, S, M or L, is small, medium or large: larger variants keep more of the sensitive tensors at higher precision, so they are bigger and a little more accurate.
Older names ending in _0 or _1, such as Q4_0 or Q8_0, use a simpler scheme. Q8_0 is still popular because 8 bits is accurate enough to be close to lossless. Names starting with IQ are i-quants, formats built for very low bit counts that are usually made with an importance matrix. F16, BF16 and F32 are not quantized: they are the original 16-bit or 32-bit weights.
The common quants at a glance
| Quant | Bits per weight | Size vs 16-bit | What to expect |
|---|---|---|---|
| Q2_K | ~3 | ~19% | Use it only when memory is very tight. Expect a clear drop in quality, which is usually worse on small models and easier to live with on large ones. |
| Q3_K_M | ~3.9 | ~24% | A reasonable choice when a 4-bit file will not fit, mostly on larger models. Quality loss is noticeable but often acceptable. |
| IQ3_XS | ~3.3 | ~21% | A reasonable choice when a 4-bit file will not fit, mostly on larger models. Quality loss is noticeable but often acceptable. |
| IQ4_XS | ~4.3 | ~27% | The usual sweet spot between file size and quality, and the quant most people start with. |
| Q4_K_S | ~4.6 | ~29% | The usual sweet spot between file size and quality, and the quant most people start with. |
| Q4_K_M | ~4.9 | ~31% | The usual sweet spot between file size and quality, and the quant most people start with. |
| Q5_K_M | ~5.7 | ~36% | Closer to the original model than 4-bit, for a moderate increase in size. A good pick when you have memory to spare. |
| Q6_K | ~6.6 | ~41% | Very close to the original quality for most uses, at a clearly larger size than 4-bit. |
| Q8_0 | ~8.5 | ~53% | Close to lossless. It costs roughly twice the memory of a 4-bit file, so it suits cases where you have plenty of VRAM. |
| F16 | ~16 | ~100% | The unquantized model. It gives original quality and needs the most memory, so it is mainly for conversion or reference. |
Bit widths are approximate averages and vary a little from model to model.
Which quant should you choose?
Pick the largest quant that fits in your memory with room left for context. Quality rises quickly from 2 to 4 bits and then flattens, which is why Q4_K_M is the common default. If you have memory to spare, Q5_K_M or Q6_K are closer to the original, and Q8_0 is close to lossless.
Size matters too. Large models tolerate low-bit quantization better than small ones, so a 70B model at 3 bits is often a better trade than a 7B model at 3 bits. If a small model is going to run on a phone or a laptop, stay at 4 bits or higher.
Does quantization make the model faster?
Often. Text generation spends most of its time reading the weights from memory, so a smaller file can generate faster on hardware with limited memory bandwidth. But some formats need extra computation to decode, so the smallest quant is not always the fastest. If speed matters, test two quants on your own hardware.
Weights, KV cache and context
The quant only changes the weights. The KV cache, the memory the model uses to remember the conversation, is separate and grows with your context length. Runtimes such as llama.cpp can store the KV cache in lower precision to save memory. See how much VRAM you need for the numbers.
Browse GGUF models by quant
Frequently asked questions
Is Q4_K_M good enough?
For most people, yes. It is the most widely used GGUF quant because it cuts the file to roughly a third of the 16-bit size while keeping quality high enough for chat, summaries and general use. If you notice errors in tasks such as code or maths, try Q5_K_M or Q6_K.
What is the difference between Q4_K_M and Q4_K_S?
Both are 4-bit k-quants. The M (medium) variant keeps a few important tensors at higher precision, so it is slightly larger and slightly better than S (small). Q4_K_M is the more common choice.
What is an imatrix, and what do IQ quants do?
An importance matrix (imatrix) is a measurement, made on sample text, of which weights matter most. Quantizers can use it to spend bits where they help most. IQ quants are formats designed for very low bit counts, and they are usually made with an imatrix. They can be smaller for the same quality, but some are slower to run on certain hardware.
Does a smaller quant run faster?
Often, yes. Generating text mostly depends on reading the weights from memory, so a smaller file can generate faster on memory-bandwidth-limited hardware. Some formats cost extra compute to decode, so the fastest quant is not always the smallest one.