LL

Llama 2 7b chat hf function calling v2

LlamaGGUF by TrelisUpdated 2023-11-24
Download on Hugging Face
Parameters7B
Context—
Layers—
Architecture—
Smallest quant0.0 GB
Quick answer

Llama 2 7b chat hf function calling v2 needs about 0.5 GB of memory at its main file (a 0.0 GB file) with an 8K context window, so it fits on a 4 GB GPU. This repository has 2 quants, from 0.0 GB to 4.1 GB (Q3_K_M).

Llama 2 7b chat with function calling v2

Llama 2 7b chat hf function calling v2 is a 7B parameter model that supports structured JSON function calls. It is optimized for function calling with minimal syntax. The model is available in GGUF format with Q3_K_M quantization.

Good at

  • Function calling with JSON output
  • Low memory footprint with Q3_K_M
  • Compact 7B parameter size

Limits

  • No vision support
  • Context length limited to 8k tokens
  • License terms not specified

Best for

  • API integration requiring function calls
  • Lightweight chatbot deployments
  • Rapid prototyping of function calling workflows

Who it’s for. Developers building chat interfaces that need reliable function calling. Researchers testing function calling mechanics in a 7B model.

Quants & memory

QuantFileEst. memoryvs VRAMResultDownload
—0.0 GB 0.5 GB
Fits Get file →
Q3_K_M4.1 GB 5.1 GB
Fits Get file →

For 8 GB VRAM, use the Q3_K_M quantized file (4.08 GB). It requires about 5.1 GB of VRAM at 8k context. The unknown quant file is 0 GB but memory estimate is 0.5 GB, suitable for very low‑budget setups.

What GPU do you need for Llama 2 7b chat hf function calling v2?

These figures add the file size, an fp16 KV cache at an 8K context window and about 0.5 GB for the runtime. They are estimates, so leave a little headroom, and expect to use less context or a smaller quant if you are right at the limit.

  • 4 GB VRAM: the best quant that fits is the main file (0.0 GB file, about 0.5 GB in use).
  • 6 GB VRAM: the best quant that fits is Q3_K_M (4.1 GB file, about 5.1 GB in use).

Specifications

Architecture from the GGUF header

Parameters7B

Source

Quantized byTrelis
FamilyLlama
Downloads (HF)3,165
Last updated2023-11-24

Frequently asked questions about Llama 2 7b chat hf function calling v2

How much VRAM does Llama 2 7b chat hf function calling v2 need?

About 0.5 GB at its main file with an 8K context window. That is the 0.0 GB file, plus the KV cache and roughly 0.5 GB of runtime overhead. Smaller quants need less and longer context needs more.

Can I run Llama 2 7b chat hf function calling v2 on an 8 GB GPU?

Yes, with Q3_K_M: the 4.1 GB file needs about 5.1 GB at 8K context. Larger quants in this repository do not fit in 8 GB.

What is the file size of Llama 2 7b chat hf function calling v2 GGUF?

It depends on the quant: file is 0.0 GB, Q3_K_M is 4.1 GB.

Where can I download Llama 2 7b chat hf function calling v2 GGUF?

The files are on Hugging Face in the Trelis/Llama-2-7b-chat-hf-function-calling-v2 repository. The download buttons on this page open the file on Hugging Face. ggufmodels.com does not host any model files.

Can I run this model on 8 GB VRAM?

Yes, the Q3_K_M quantized file uses about 5.1 GB of VRAM at 8k context, fitting within an 8 GB GPU.

Does this model support vision?

No, the model has no vision capabilities.

What is the context length?

The model is trained for 8k token context length.

Similar models

Browse all →

More from Trelis

All by Trelis →

More Llama models

All Llama →

Other small models

All small models →