MI

Voxtral Small 24B

MistralGGUF by handy-computerUpdated 2026-10-16
Download on Hugging Face
Parameters24.3B
Context—
Layers—
Architecturevoxtral
Smallest quant14.3 GB
Quick answer

Voxtral Small 24B needs about 16.5 GB of memory at Q4_K_M (a 14.3 GB file) with an 8K context window, so it fits on a 24 GB GPU. This repository has 6 quants, from 14.3 GB (Q4_K_M) to 48.5 GB (F16).

24‑billion‑parameter audio‑LLM for transcription, translation and audio understanding

Voxtral Small 24B is a 24.3 B parameter model built on the Mistral architecture. It adds a Whisper‑large‑v3 audio encoder and a 40‑layer decoder for speech tasks. The model supports 32k‑token context and multilingual audio input.

Good at

  • Speech transcription and translation
  • Audio‑based Q&A and summarization
  • Multilingual voice function calling

Limits

  • No vision support
  • Requires 8k context for memory estimates
  • License is Apache‑2.0

Best for

  • Transcribing long audio files up to 30 min
  • Generating summaries from spoken content
  • Building voice‑activated workflows

Who it’s for. Developers needing an offline, open‑source audio‑LLM for transcription, translation, or voice‑controlled applications. The model is suitable for server or edge deployments that can accommodate its memory footprint.

Quants & memory

QuantFileEst. memoryvs VRAMResultDownload
Q4_K_M14.3 GB 16.5 GB
Too big Get file →
Q5_K_M17.1 GB 19.7 GB
Too big Get file →
Q6_K19.9 GB 22.8 GB
Too big Get file →
Q8_025.8 GB 29.4 GB
Too big Get file →
BF1648.5 GB 54.9 GB
Too big Get file →
F1648.5 GB 54.9 GB
Too big Get file →

For 8 GB VRAM use Q4_K_M (14.3 GB file, ~16.5 GB memory at 8k context). For 12 GB VRAM use Q5_K_M (17.14 GB file, ~19.7 GB memory). For 16 GB VRAM use Q6_K (19.94 GB file, ~22.8 GB memory).

What GPU do you need for Voxtral Small 24B?

These figures add the file size, an fp16 KV cache at an 8K context window and about 0.5 GB for the runtime. They are estimates, so leave a little headroom, and expect to use less context or a smaller quant if you are right at the limit.

  • 4 GB VRAM: no quant fits at an 8K context.
  • 6 GB VRAM: no quant fits at an 8K context.
  • 8 GB VRAM: no quant fits at an 8K context.
  • 12 GB VRAM: no quant fits at an 8K context.
  • 16 GB VRAM: no quant fits at an 8K context.
  • 24 GB VRAM: the best quant that fits is Q6_K (19.9 GB file, about 22.8 GB in use).
  • 32 GB VRAM: the best quant that fits is Q8_0 (25.8 GB file, about 29.4 GB in use).
  • 48 GB VRAM: the best quant that fits is Q8_0 (25.8 GB file, about 29.4 GB in use).

Which Voxtral Small 24B quant should you choose?

Q4_K_M is the usual starting point for Voxtral Small 24B: it balances file size and quality well. Here it is a 14.3 GB file.

The Q8_0 file is 25.8 GB, about 1.8 times the size of Q4_K_M. It keeps the weights closer to the original, which only matters if you have the memory to spare.

New to quant names? Read GGUF quantization explained.

Specifications

Architecture from the GGUF header

Architecturevoxtral
Parameters24.3B

Source

Quantized byhandy-computer
FamilyMistral
Licenseapache-2.0
Downloads (HF)140,997
Last updated2026-10-16

Frequently asked questions about Voxtral Small 24B

How much VRAM does Voxtral Small 24B need?

About 16.5 GB at Q4_K_M with an 8K context window. That is the 14.3 GB file, plus the KV cache and roughly 0.5 GB of runtime overhead. Smaller quants need less and longer context needs more.

Can I run Voxtral Small 24B on an 8 GB GPU?

Not at an 8K context. The smallest file needs about 16.5 GB, so you would need a larger GPU or CPU offloading.

What is the file size of Voxtral Small 24B GGUF?

It depends on the quant: Q4_K_M is 14.3 GB, Q5_K_M is 17.1 GB, Q6_K is 19.9 GB, Q8_0 is 25.8 GB, BF16 is 48.5 GB, F16 is 48.5 GB.

Where can I download Voxtral Small 24B GGUF?

The files are on Hugging Face in the handy-computer/Voxtral-Small-24B-2507-gguf repository. The download buttons on this page open the file on Hugging Face. ggufmodels.com does not host any model files.

What license does Voxtral Small 24B use?

Hugging Face lists the license as apache-2.0. Licenses can differ between the quantized files and the original model, so check the repository and the base model before commercial use.

Can I run Voxtral Small 24B on 8 GB VRAM?

Use the Q4_K_M quantized file; it requires about 16.5 GB of VRAM at 8k context, so 8 GB is insufficient.

Does the model support image input?

No, Voxtral Small 24B does not support vision input.

What license applies to this model?

The model is released under the Apache‑2.0 license.

Other GGUFs of Voxtral-Small-24B-2507

Similar models

Browse all →

More from handy-computer

All by handy-computer →

More Mistral models

All Mistral →

Other medium models

All medium models →