IQ2 NL IQ3 NL Test Models needs about 7.8 GB of memory at Q4_K (a 6.5 GB file) with an 8K context window, so it fits on a 8 GB GPU. This repository has 17 quants, from 0.0 GB to 17.8 GB (F16).
IQ2 NL IQ3 NL Test Models – Quantized variants of NVIDIA‑Nemotron‑Nano‑9B‑v2
IQ2 NL IQ3 NL Test Models are experimental quantizations of NVIDIA‑Nemotron‑Nano‑9B‑v2. They use the nemotron_h hybrid Mamba‑2 / attention architecture and support 1,048,576 context tokens. The models are released under the MIT license.
Good at
- Hybrid Mamba‑2 / attention architecture
- Large context support
- Experimental IQ2_NL and IQ3_NL quantization
Limits
- No vision support
- Requires llama.cpp fork for IQ2_NL/IQ3_NL
- Large memory for higher‑bit quants
Best for
- Chat and reasoning tasks
- Testing low‑bit quantization
- Research on non‑linear block types
Who it’s for. Developers and researchers interested in evaluating new quantization types for the Nemotron family. The models are suitable for those who can run llama.cpp with the required fork and are comfortable with experimental releases.
Quants & memory
| Quant | File | Est. memory | vs VRAM | Result | |
|---|---|---|---|---|---|
| — | 0.0 GB | 0.5 GB | Fits | Get file → | |
| IQ2_NL | 3.1 GB | 4.0 GB | Fits | Get file → | |
| Q2_0 | 3.2 GB | 4.1 GB | Fits | Get file → | |
| IQ1_M | 3.4 GB | 4.3 GB | Fits | Get file → | |
| IQ3_NL | 4.1 GB | 5.1 GB | Fits | Get file → | |
| IQ1_M | 4.9 GB | 5.9 GB | Fits | Get file → | |
| Q2_K | 5.0 GB | 6.1 GB | Fits | Get file → | |
| Q3_K | 5.4 GB | 6.5 GB | Fits | Get file → | |
| Q4_K | 6.5 GB | 7.8 GB | Tight | Get file → | |
| Q4_K | 6.5 GB | 7.8 GB | Tight | Get file → | |
| Q5_K | 7.1 GB | 8.4 GB | Too big | Get file → | |
| Q5_K | 7.1 GB | 8.4 GB | Too big | Get file → | |
| Q6_K | 9.1 GB | 10.7 GB | Too big | Get file → | |
| Q6_K | 9.1 GB | 10.7 GB | Too big | Get file → | |
| Q8_0 | 9.5 GB | 11.1 GB | Too big | Get file → | |
| Q8_0 | 9.5 GB | 11.1 GB | Too big | Get file → | |
| F16 | 17.8 GB | 20.4 GB | Too big | Get file → |
For 8 GB VRAM, use IQ2_NL (file size 3.12 GB, memory 4 GB at 8 k context). For 12 GB VRAM, Q4_K (file size 6.53 GB, memory 7.8 GB). For 20 GB VRAM, F16 (file size 17.79 GB, memory 20.4 GB). The estimates are approximate.
What GPU do you need for IQ2 NL IQ3 NL Test Models?
These figures add the file size, an fp16 KV cache at an 8K context window and about 0.5 GB for the runtime. They are estimates, so leave a little headroom, and expect to use less context or a smaller quant if you are right at the limit.
- 4 GB VRAM: the best quant that fits is IQ2_NL (3.1 GB file, about 4.0 GB in use).
- 6 GB VRAM: the best quant that fits is IQ1_M (4.9 GB file, about 5.9 GB in use).
- 8 GB VRAM: the best quant that fits is Q4_K (6.5 GB file, about 7.8 GB in use).
- 12 GB VRAM: the best quant that fits is Q8_0 (9.5 GB file, about 11.1 GB in use).
- 16 GB VRAM: the best quant that fits is Q8_0 (9.5 GB file, about 11.1 GB in use).
- 24 GB VRAM: the best quant that fits is F16 (17.8 GB file, about 20.4 GB in use).
Which IQ2 NL IQ3 NL Test Models quant should you choose?
Q4_K (6.5 GB) is the middle option in this repository. The usual sweet spot between file size and quality, and the quant most people start with.
If memory is tight, the smallest file is the first one in the table at 0.0 GB, with the quality trade-offs that come with fewer bits.
New to quant names? Read GGUF quantization explained.
Specifications
Architecture from the GGUF header
Source
Frequently asked questions about IQ2 NL IQ3 NL Test Models
How much VRAM does IQ2 NL IQ3 NL Test Models need?
About 7.8 GB at Q4_K with an 8K context window. That is the 6.5 GB file, plus the KV cache and roughly 0.5 GB of runtime overhead. Smaller quants need less and longer context needs more.
Can I run IQ2 NL IQ3 NL Test Models on an 8 GB GPU?
Yes, with Q4_K: the 6.5 GB file needs about 7.8 GB at 8K context. Larger quants in this repository do not fit in 8 GB.
What is the file size of IQ2 NL IQ3 NL Test Models GGUF?
It depends on the quant: file is 0.0 GB, IQ2_NL is 3.1 GB, Q2_0 is 3.2 GB, IQ1_M is 3.4 GB, IQ3_NL is 4.1 GB, IQ1_M is 4.9 GB and 11 more.
What context length does IQ2 NL IQ3 NL Test Models support?
The GGUF header lists a trained context length of 1,048,576 tokens (1024K).
Where can I download IQ2 NL IQ3 NL Test Models GGUF?
The files are on Hugging Face in the eaddario/IQ2_NL-IQ3_NL-Test-Models repository. The download buttons on this page open the file on Hugging Face. ggufmodels.com does not host any model files.
What license does IQ2 NL IQ3 NL Test Models use?
Hugging Face lists the license as mit. Licenses can differ between the quantized files and the original model, so check the repository and the base model before commercial use.
Can I run IQ2_NL on 8 GB VRAM?
Yes. The IQ2_NL quant is 3.12 GB and requires about 4 GB of VRAM at 8 k context.
Do these models support vision?
No. The models are purely language models with no vision capability.
What license applies to these models?
They are released under the MIT license.