LL

codellama CodeLlama 13b Python hf

LlamaGGUF by RichardErkhovUpdated 2024-07-23
Download on Hugging Face
Parameters13B
Context—
Layers—
Architecture—
Smallest quant0.0 GB
Quick answer

codellama CodeLlama 13b Python hf needs about 0.5 GB of memory at Q4_K_M (a 0.0 GB file) with an 8K context window, so it fits on a 4 GB GPU. This repository has 22 quants, from 0.0 GB (Q5_K_M) to 7.4 GB (IQ4_NL).

13B CodeLlama Python model for code generation

CodeLlama 13b Python hf is a 13‑billion‑parameter LLM focused on Python code synthesis. It is a fine‑tuned variant of the Code Llama family, designed for code completion and Python‑specific tasks.

Good at

  • Python code completion
  • Code understanding
  • Code generation

Limits

  • No vision support
  • No instruction or chat mode
  • Limited to text generation

Best for

  • Automated code completion
  • Python script generation
  • Debugging assistance

Who it’s for. Developers and researchers needing a dedicated Python code model. The model is suitable for integration into IDEs, code assistants, or automated code generation pipelines.

Quants & memory

QuantFileEst. memoryvs VRAMResultDownload
Q5_K_M0.0 GB 0.5 GB
Fits Get file →
Q8_00.0 GB 0.5 GB
Fits Get file →
Q4_K_M0.0 GB 0.5 GB
Fits Get file →
Q6_K0.1 GB 0.6 GB
Fits Get file →
Q5_10.1 GB 0.6 GB
Fits Get file →
Q4_K0.2 GB 0.7 GB
Fits Get file →
Q5_K0.3 GB 0.8 GB
Fits Get file →
Q4_K_S1.0 GB 1.7 GB
Fits Get file →
Q5_01.5 GB 2.2 GB
Fits Get file →
Q5_K_S1.9 GB 2.6 GB
Fits Get file →
Q4_03.0 GB 3.9 GB
Fits Get file →
IQ3_M3.6 GB 4.6 GB
Fits Get file →
Q4_14.3 GB 5.3 GB
Fits Get file →
Q2_K4.9 GB 5.9 GB
Fits Get file →
IQ3_XS5.4 GB 6.5 GB
Fits Get file →
IQ3_S5.7 GB 6.8 GB
Fits Get file →
Q3_K_S5.7 GB 6.8 GB
Fits Get file →
Q3_K6.3 GB 7.6 GB
Tight Get file →
Q3_K_M6.3 GB 7.6 GB
Tight Get file →
Q3_K_L6.9 GB 8.3 GB
Too big Get file →
IQ4_XS7.0 GB 8.4 GB
Too big Get file →
IQ4_NL7.4 GB 8.8 GB
Too big Get file →

For 8 GB VRAM, Q5_K_M or Q4_K_M provide 0.5 GB memory usage. For 16 GB VRAM, Q6_K or Q5_1 use 0.6 GB. For 32 GB VRAM, Q4_K or Q5_K use 0.7–0.8 GB. Larger quants such as Q5_0 or Q5_K_S require 2.2–2.6 GB and are suitable for 24 GB+ VRAM.

What GPU do you need for codellama CodeLlama 13b Python hf?

These figures add the file size, an fp16 KV cache at an 8K context window and about 0.5 GB for the runtime. They are estimates, so leave a little headroom, and expect to use less context or a smaller quant if you are right at the limit.

  • 4 GB VRAM: the best quant that fits is Q4_0 (3.0 GB file, about 3.9 GB in use).
  • 6 GB VRAM: the best quant that fits is Q2_K (4.9 GB file, about 5.9 GB in use).
  • 8 GB VRAM: the best quant that fits is Q3_K (6.3 GB file, about 7.6 GB in use).
  • 12 GB VRAM: the best quant that fits is IQ4_NL (7.4 GB file, about 8.8 GB in use).

Which codellama CodeLlama 13b Python hf quant should you choose?

Q4_K_M is the usual starting point for codellama CodeLlama 13b Python hf: it balances file size and quality well. Here it is a 0.0 GB file.

The Q8_0 file is 0.0 GB, about 0.5 times the size of Q4_K_M. It keeps the weights closer to the original, which only matters if you have the memory to spare.

If memory is tight, the smallest file is Q5_K_M at 0.0 GB, with the quality trade-offs that come with fewer bits. Closer to the original model than 4-bit, for a moderate increase in size. A good pick when you have memory to spare.

New to quant names? Read GGUF quantization explained.

Specifications

Architecture from the GGUF header

Parameters13B

Source

Quantized byRichardErkhov
FamilyLlama
Downloads (HF)1,572
Last updated2024-07-23

Frequently asked questions about codellama CodeLlama 13b Python hf

How much VRAM does codellama CodeLlama 13b Python hf need?

About 0.5 GB at Q4_K_M with an 8K context window. That is the 0.0 GB file, plus the KV cache and roughly 0.5 GB of runtime overhead. Smaller quants need less and longer context needs more.

Can I run codellama CodeLlama 13b Python hf on an 8 GB GPU?

Yes, with Q3_K: the 6.3 GB file needs about 7.6 GB at 8K context. Larger quants in this repository do not fit in 8 GB.

What is the file size of codellama CodeLlama 13b Python hf GGUF?

It depends on the quant: Q5_K_M is 0.0 GB, Q8_0 is 0.0 GB, Q4_K_M is 0.0 GB, Q6_K is 0.1 GB, Q5_1 is 0.1 GB, Q4_K is 0.2 GB and 16 more.

Where can I download codellama CodeLlama 13b Python hf GGUF?

The files are on Hugging Face in the RichardErkhov/codellama_-_CodeLlama-13b-Python-hf-gguf repository. The download buttons on this page open the file on Hugging Face. ggufmodels.com does not host any model files.

Can I run CodeLlama 13b Python on 8 GB VRAM?

Yes, using Q5_K_M or Q4_K_M quants which use about 0.5 GB memory at 8k context.

Does this model support image input?

No, it only accepts text input.

Is the model safe for instruction following?

No, the Python variant is not designed for instruction or chat modes.

Similar models

Browse all →

More from RichardErkhov

All by RichardErkhov →

More Llama models

All Llama →

Other medium models

All medium models →