RE

ReluLLaMA 70B PowerInfer

RelullamaGGUF by TiinyUpdated 2024-02-02
Download on Hugging Face
Parameters70B
Context—
Layers—
Architecture—
Smallest quant42.2 GB
Quick answer

ReluLLaMA 70B PowerInfer needs about 47.7 GB of memory at its main file (a 42.2 GB file) with an 8K context window, so it fits on a 48 GB GPU.

ReluLLaMA 70B PowerInfer – a large language model in GGUF format

ReluLLaMA 70B PowerInfer is a 70‑billion‑parameter model distributed as a GGUF file. It contains the LLM weights and predictor weights from SparseLLM/ReluLLaMA‑70B. The model is licensed under llama2.

Good at

  • Large context reasoning
  • High‑capacity language modeling
  • Predictor‑augmented inference

Limits

  • Requires large VRAM – 47.7 GB at 8 k context
  • No vision or multimodal support
  • Only one quant available (unknown)

Best for

  • Research requiring 70B parameter models
  • Large‑scale text generation
  • Benchmarking GGUF performance

Who it’s for. Researchers and developers who need a 70B parameter model in GGUF format and have sufficient GPU memory. The model is suitable for advanced language tasks where context size and parameter count are critical.

Quants & memory

QuantFileEst. memoryvs VRAMResultDownload
—42.2 GB 47.7 GB
Too big Get file →

The only available quant is listed as unknown. Its file size is 42.18 GB and it requires approximately 47.7 GB of VRAM at 8 k context. If you have a GPU with at least 48 GB of VRAM, this quant can be used. No other quants are provided.

What GPU do you need for ReluLLaMA 70B PowerInfer?

These figures add the file size, an fp16 KV cache at an 8K context window and about 0.5 GB for the runtime. They are estimates, so leave a little headroom, and expect to use less context or a smaller quant if you are right at the limit.

  • 4 GB VRAM: no quant fits at an 8K context.
  • 6 GB VRAM: no quant fits at an 8K context.
  • 8 GB VRAM: no quant fits at an 8K context.
  • 12 GB VRAM: no quant fits at an 8K context.
  • 16 GB VRAM: no quant fits at an 8K context.
  • 24 GB VRAM: no quant fits at an 8K context.
  • 32 GB VRAM: no quant fits at an 8K context.
  • 48 GB VRAM: the best quant that fits is the main file (42.2 GB file, about 47.7 GB in use).

Specifications

Architecture from the GGUF header

Parameters70B

Source

Quantized byTiiny
FamilyRelullama
Licensellama2
Downloads (HF)1,391
Last updated2024-02-02

Frequently asked questions about ReluLLaMA 70B PowerInfer

How much VRAM does ReluLLaMA 70B PowerInfer need?

About 47.7 GB at its main file with an 8K context window. That is the 42.2 GB file, plus the KV cache and roughly 0.5 GB of runtime overhead. Smaller quants need less and longer context needs more.

Can I run ReluLLaMA 70B PowerInfer on an 8 GB GPU?

Not at an 8K context. The smallest file needs about 47.7 GB, so you would need a larger GPU or CPU offloading.

What is the file size of ReluLLaMA 70B PowerInfer GGUF?

It depends on the quant: file is 42.2 GB.

Where can I download ReluLLaMA 70B PowerInfer GGUF?

The files are on Hugging Face in the Tiiny/ReluLLaMA-70B-PowerInfer-GGUF repository. The download buttons on this page open the file on Hugging Face. ggufmodels.com does not host any model files.

What license does ReluLLaMA 70B PowerInfer use?

Hugging Face lists the license as llama2. Licenses can differ between the quantized files and the original model, so check the repository and the base model before commercial use.

Can I run this model on a GPU with 8 GB VRAM?

No. The model requires about 47.7 GB of VRAM at 8 k context, far exceeding 8 GB.

What is the file size of the GGUF model?

The GGUF file is 42.18 GB.

Does this model support vision tasks?

No, it only contains language model weights and predictor weights.

Similar models

Browse all →

More from Tiiny

All by Tiiny →

More Relullama models

All Relullama →

Other large models

All large models →