LL

LLaVA-NeXT-Video-7B-DPO

LLaVAGGUF by ManishThotaUpdated 2024-06-29
Download on Hugging Face
Parameters6.74B
Context4K
Layers32
Architecturellama
Smallest quant7.2 GB
Quick answer

LLaVA-NeXT-Video-7B-DPO needs about 18.3 GB of memory at F16 (a 13.5 GB file) with an 8K context window, so it fits on a 24 GB GPU. This repository has 3 quants, from 7.2 GB (Q8_0) to 27.0 GB (F32).

LLaVA‑NEXT‑Video‑7B‑DPO: 7.74B parameter multimodal model with video vision and embedding

LLaVA‑NEXT‑Video‑7B‑DPO is a 6.74B parameter multimodal model that supports vision and embedding tasks. It is quantised for efficient inference and can process up to 4096 context tokens. The model is available in Q8_0, F16, and F32 GGUF formats.

Good at

  • Vision‑based embedding
  • Video context understanding
  • Multimodal inference

Limits

  • No explicit license information
  • No text‑only generation limits
  • Requires 8k context for full memory estimate

Best for

  • Video captioning
  • Multimodal search
  • Embedding generation

Who it’s for. Developers needing a lightweight vision‑enabled model for video analysis. The model is suitable for applications that require both visual and textual embeddings within a 7‑8B parameter budget.

Quants & memory

QuantFileEst. memoryvs VRAMResultDownload
Q8_07.2 GB 9.8 GB
Too big Get file →
F1613.5 GB 16.1 GB
Too big Get file →
F3227.0 GB 29.6 GB
Too big Get file →

For 8 GB VRAM, use the Q8_0 quantised file (7.16 GB, ~12 GB memory at 8k context). If more VRAM is available, F16 (13.48 GB, ~18.3 GB memory) offers higher precision. F32 provides the highest quality but needs ~31.7 GB memory at 8k context.

What GPU do you need for LLaVA-NeXT-Video-7B-DPO?

These figures add the file size, an fp16 KV cache at an 8K context window (worked out from this model’s own header) and about 0.5 GB for the runtime. They are estimates, so leave a little headroom, and expect to use less context or a smaller quant if you are right at the limit.

  • 4 GB VRAM: no quant fits at an 8K context.
  • 6 GB VRAM: no quant fits at an 8K context.
  • 8 GB VRAM: no quant fits at an 8K context.
  • 12 GB VRAM: the best quant that fits is Q8_0 (7.2 GB file, about 12.0 GB in use).
  • 16 GB VRAM: the best quant that fits is Q8_0 (7.2 GB file, about 12.0 GB in use).
  • 24 GB VRAM: the best quant that fits is F16 (13.5 GB file, about 18.3 GB in use).
  • 32 GB VRAM: the best quant that fits is F32 (27.0 GB file, about 31.7 GB in use).

Which LLaVA-NeXT-Video-7B-DPO quant should you choose?

F16 (13.5 GB) is the middle option in this repository. The unquantized model. It gives original quality and needs the most memory, so it is mainly for conversion or reference.

If memory is tight, the smallest file is Q8_0 at 7.2 GB, with the quality trade-offs that come with fewer bits. Close to lossless. It costs roughly twice the memory of a 4-bit file, so it suits cases where you have plenty of VRAM.

New to quant names? Read GGUF quantization explained.

Specifications

Architecture from the GGUF header

Architecturellama
Parameters6.74B
Layers32
Trained context4,096 tokens
KV cache (fp16)512 KB per token

Source

Quantized byManishThota
FamilyLLaVA
Downloads (HF)2,874
Last updated2024-06-29

Frequently asked questions about LLaVA-NeXT-Video-7B-DPO

How much VRAM does LLaVA-NeXT-Video-7B-DPO need?

About 18.3 GB at F16 with an 8K context window. That is the 13.5 GB file, plus the KV cache and roughly 0.5 GB of runtime overhead. Smaller quants need less and longer context needs more.

Can I run LLaVA-NeXT-Video-7B-DPO on an 8 GB GPU?

Not at an 8K context. The smallest file needs about 12.0 GB, so you would need a larger GPU or CPU offloading.

What is the file size of LLaVA-NeXT-Video-7B-DPO GGUF?

It depends on the quant: Q8_0 is 7.2 GB, F16 is 13.5 GB, F32 is 27.0 GB.

What context length does LLaVA-NeXT-Video-7B-DPO support?

The GGUF header lists a trained context length of 4,096 tokens (4K). The KV cache grows with the context you use: about 4.3 GB at 8K and 2.1 GB at 4K.

Where can I download LLaVA-NeXT-Video-7B-DPO GGUF?

The files are on Hugging Face in the ManishThota/Ollama_Quantised_LLaVA-NeXT-Video-7B-DPO repository. The download buttons on this page open the file on Hugging Face. ggufmodels.com does not host any model files.

Can I run LLaVA‑NEXT‑Video‑7B‑DPO on 8 GB VRAM?

Yes, the Q8_0 quantised file is 7.16 GB and requires about 12 GB of VRAM at 8k context, which fits within an 8 GB VRAM budget with some headroom.

What is the maximum context length for this model?

The model supports up to 4096 context tokens.

Does the model support vision tasks?

Yes, it has detected vision abilities and can process visual inputs.

Similar models

Browse all →

More from ManishThota

All by ManishThota →

More LLaVA models

All LLaVA →

Other small models

All small models →