Small GGUF models
91171 GGUF models with roughly 4B to 9B parameters. Sizes run from 3.51B to 9.5B parameters. The longest trained context is 4194304K tokens.
What counts as a small model?
On this site small means roughly 4B to 9B parameters. A Q4_K_M file is usually 2 to 6 GB, which fits an 8 GB GPU with room for context. 91,171 models fit this class. The most downloaded are Ornith-1.5-9B, locate anything.cpp and Ornith 1.0 9B.
Their Q4_K_M files run 0.0 to 42.5 GB. The families you will see most are Llama (17914), Qwen (13245) and Mistral (6032). The size class sorts models by parameter count, so a model with more experts or a longer context can need more memory than its class suggests. The model pages show the real file sizes.
Frequently asked questions
How much VRAM does a small GGUF model need?
A Q4_K_M file is usually 2 to 6 GB, which fits an 8 GB GPU with room for context.
Which small GGUF model is the most popular?
By Hugging Face downloads, Ornith-1.5-9B.