GGUF models by LiteLLMs

73 GGUF models quantized by LiteLLMs on Hugging Face. Sizes run from 272M to 141B parameters. The longest trained context is 1024K tokens.

Most downloadedNewestLargestLongest contextSmallest
MI
Mixtral 8x22B v0.1
Mistral · LiteLLMs
Params141B
Context64K
Q4_K_M85.6 GB
GGUF
LL
Llama 3 Open Ko 8B
Llama · LiteLLMs
Params8B
Context8K
Q4_K_M4.9 GB
GGUF
LL
Meta Llama 3 70B Instruct
Llama · LiteLLMs
Params70.6B
Context8K
Q4_K_M42.5 GB
GGUF
LL
Llama-3-8b-64k-PoSE
Llama · LiteLLMs
Params8B
Context128K
Q4_K_M4.9 GB
GGUF
LL
Llama 3 Smaug 8B
Llama · LiteLLMs
Params8.03B
Context8K
Q4_K_M4.9 GB
GGUF
LL
French-Alpaca-Llama3-8B-Instruct-v1.0
Llama · LiteLLMs
Params8B
Context8K
Q4_K_M4.9 GB
GGUF
LL
Llama-3-8B-Instruct-Gradient-1048k
Llama · LiteLLMs
Params8B
Context1024K
Q4_K_M4.9 GB
GGUF
LL
Hermes-2-Pro-Llama-3-8B
Llama · LiteLLMs
Params8.03B
Context8K
Q4_K_M4.9 GB
Tool use
MI
Mistral-7B-Instruct-v0.3
Mistral · LiteLLMs
Params7.25B
Context32K
Q4_K_M4.4 GB
GGUF
LL
Meta-Llama-3-13B-Instruct
Llama · LiteLLMs
Params13.9B
Context8K
Q4_K_M8.4 GB
GGUF
MI
Mistral 7B Merge 14 v0.1
Mistral · LiteLLMs
Params7.24B
Context32K
Q4_K_M4.4 GB
GGUF
LL
Llama3 8B Chinese Chat
Llama · LiteLLMs
Params8.03B
Context8K
Q4_K_M4.9 GB
GGUF
LL
Hermes-2-Theta-Llama-3-8B
Llama · LiteLLMs
Params8.03B
Context8K
Q4_K_M4.9 GB
Tool use
LL
Llama-3-Open-Ko-8B-Instruct-preview
Llama · LiteLLMs
Params8.03B
Context8K
Q4_K_M4.9 GB
GGUF
LL
Meta Llama 3 8B Instruct
Llama · LiteLLMs
Params8.03B
Context8K
Q4_K_M4.9 GB
GGUF
WI
WizardLM 2 8x22B
Wizardlm · LiteLLMs
Params141B
Context64K
Q4_K_M85.6 GB
GGUF
DP
OGNO 7B
Dpo · LiteLLMs
Params7.24B
Context32K
Q4_K_M4.4 GB
GGUF
LL
Llama 3 8B Web
Llama · LiteLLMs
Params8.03B
Context8K
Q4_K_M4.9 GB
GGUF
LL
dolphin 2.9 llama3 8b
Llama · LiteLLMs
Params8.03B
Context8K
Q4_K_M4.9 GB
CodingTool use
OP
OpenELM-1_1B-Instruct
Openelm · LiteLLMs
Params1.08B
Context2K
Q4_K_M0.7 GB
GGUF
LL
Llama3-OpenBioLLM-70B
Llama · LiteLLMs
Params70.6B
Context8K
Q4_K_M42.5 GB
GGUF
CA
CarbonVillain en 10.7B v3
Carbonvillain · LiteLLMs
Params10.7B
Context4K
Q4_K_M6.5 GB
GGUF
LL
Meta-Llama-3-8B
Llama · LiteLLMs
Params8.03B
Context8K
Q4_K_M4.9 GB
GGUF
CO
c4ai-command-r-plus
Command R · LiteLLMs
Params104B
Context128K
Q4_K_M62.8 GB
GGUF
GE
gemma 7b
Gemma · LiteLLMs
Params8.54B
Context8K
Q4_K_M5.3 GB
GGUF
LL
Llama3 OpenBioLLM 8B
Llama · LiteLLMs
Params8.03B
Context8K
Q4_K_M4.9 GB
GGUF
OP
OpenELM-450M-Instruct
Openelm · LiteLLMs
Params457M
Context2K
Q4_K_M0.3 GB
GGUF
ST
starcoder2-15b-instruct-v0.1
StarCoder · LiteLLMs
Params16B
Context16K
Q4_K_M9.9 GB
Coding
LL
Llama-3-MAAL-8B-Instruct-v0.1
Llama · LiteLLMs
Params8.03B
Context8K
Q4_K_M4.9 GB
GGUF
PH
Phi 3 mini 128k instruct
Phi · LiteLLMs
Params3.82B
Context128K
Q4_K_M2.4 GB
Coding

About LiteLLMs on Hugging Face

LiteLLMs has published 73 GGUF models that ggufmodels.com tracks, across families such as Llama, Mistral and Openelm. The most downloaded are Mixtral 8x22B v0.1, Llama 3 Open Ko 8B and Meta Llama 3 70B Instruct.

The quants LiteLLMs publishes most often are Q5_K_S (73), Q5_K_M (73), Q5_0 (73) and Q4_K_S (73). Their Q4_K_M files range from 0.2 to 85.6 GB. Parameter counts run from 272M to 141B, and the longest trained context window is 1024K tokens.

Frequently asked questions

What GGUF models has LiteLLMs published?

73 models are listed here, mainly from the Llama, Mistral and Openelm families. The most downloaded is Mixtral 8x22B v0.1.

Which quants does LiteLLMs publish?

The most common are Q5_K_S (73 models), Q5_K_M (73 models), Q5_0 (73 models) and Q4_K_S (73 models).

Keep browsing