ST

stories15M MOE

StoriesGGUF by ggml-orgUpdated 2024-08-05
Download on Hugging Face
Parameters8M
Context256
Layers6
Architecturellama
Smallest quant0.0 GB
Quick answer

stories15M MOE needs about 0.6 GB of memory at Q8_0 (a 0.0 GB file) with an 8K context window, so it fits on a 4 GB GPU. This repository has 3 quants, from 0.0 GB to 0.1 GB (F16).

Stories15M MOE – a 7.82 M‑parameter mixture of experts model

Stories15M MOE is a 7.82 M‑parameter llama architecture model. It contains four experts, each a copy of tinyllama‑15M‑stories. The model is intended for testing and not production use.

Good at

  • Small footprint
  • Mixture of experts architecture
  • Test‑ready for LoRA adapters

Limits

  • No vision support
  • Not for production
  • Limited language support

Best for

  • Bedtime story generation
  • LoRA fine‑tuning experiments
  • Low‑resource inference

Who it’s for. Developers and researchers who need a lightweight MOE model for experimentation or prototyping. It is suitable for those who want to test LoRA adapters or run small‑scale inference.

Quants & memory

QuantFileEst. memoryvs VRAMResultDownload
—0.0 GB 0.5 GB
Fits Get file →
Q8_00.0 GB 0.5 GB
Fits Get file →
F160.1 GB 0.6 GB
Fits Get file →

For 8 GB VRAM use the Q8_0 quant, 0.04 GB file, 0.6 GB memory at 8 k context. For 4 GB VRAM use the unknown quant, 0.02 GB file, 0.6 GB memory. For higher precision use F16, 0.07 GB file, 0.6 GB memory.

What GPU do you need for stories15M MOE?

These figures add the file size, an fp16 KV cache at an 8K context window (worked out from this model’s own header) and about 0.5 GB for the runtime. They are estimates, so leave a little headroom, and expect to use less context or a smaller quant if you are right at the limit.

  • 4 GB VRAM: the best quant that fits is F16 (0.1 GB file, about 0.6 GB in use).

Which stories15M MOE quant should you choose?

Q8_0 (0.0 GB) is the middle option in this repository. Close to lossless. It costs roughly twice the memory of a 4-bit file, so it suits cases where you have plenty of VRAM.

If memory is tight, the smallest file is the first one in the table at 0.0 GB, with the quality trade-offs that come with fewer bits.

New to quant names? Read GGUF quantization explained.

Specifications

Architecture from the GGUF header

Architecturellama
Parameters8M (approximate, mixture of experts)
Layers6
Trained context256 tokens
KV cache (fp16)7 KB per token

Source

Quantized byggml-org
FamilyStories
Licensemit
Downloads (HF)61,071
Last updated2024-08-05

Frequently asked questions about stories15M MOE

How much VRAM does stories15M MOE need?

About 0.6 GB at Q8_0 with an 8K context window. That is the 0.0 GB file, plus the KV cache and roughly 0.5 GB of runtime overhead. Smaller quants need less and longer context needs more.

Can I run stories15M MOE on an 8 GB GPU?

Yes, with F16: the 0.1 GB file needs about 0.6 GB at 8K context. Larger quants in this repository do not fit in 8 GB.

What is the file size of stories15M MOE GGUF?

It depends on the quant: file is 0.0 GB, Q8_0 is 0.0 GB, F16 is 0.1 GB.

What context length does stories15M MOE support?

The GGUF header lists a trained context length of 256 tokens (256). The KV cache grows with the context you use: about 0.1 GB at 8K and 0.0 GB at 256.

Where can I download stories15M MOE GGUF?

The files are on Hugging Face in the ggml-org/stories15M_MOE repository. The download buttons on this page open the file on Hugging Face. ggufmodels.com does not host any model files.

What license does stories15M MOE use?

Hugging Face lists the license as mit. Licenses can differ between the quantized files and the original model, so check the repository and the base model before commercial use.

Can I run this model on 4 GB VRAM?

Yes, the unknown quant file is 0.02 GB and requires about 0.6 GB memory at 8 k context.

Does it support vision tasks?

No, the model has no vision capabilities.

Is it suitable for production?

No, the model is intended for testing and experimentation only.

Similar models

Browse all →

More from ggml-org

All by ggml-org →

More Stories models

All Stories →

Other tiny models

All tiny models →