Mellum2.1 Thinking — GGUF

This repository contains GGUF builds of JetBrains/Mellum2.1-12B-A2.5B-Thinking in every quantization we publish, ready to run with llama.cpp, Ollama, LM Studio, and other GGUF-compatible runtimes. Each variant is one file; download the one you need.

Mellum2.1 Thinking is a Mixture-of-Experts reasoning model (64 experts, 8 activated per token, 131,072-token context) that emits its chain of thought inside <think>...</think> blocks before the final answer. For the full model description, evaluation results, and architecture details, see the original model card: JetBrains/Mellum2.1-12B-A2.5B-Thinking.

Available quantizations

File	Quantization	Description	Size	KLD vs BF16 ↓	Top-token match ↑
Mellum2.1-12B-A2.5B-Thinking-BF16.gguf	BF16	16-bit, no quantization (reference)	24.3 GB	—	—
Mellum2.1-12B-A2.5B-Thinking-Q8_0.gguf	Q8_0	8-bit, effectively lossless	12.9 GB	0.008	96.1%
Mellum2.1-12B-A2.5B-Thinking-Q6_K.gguf	Q6_K	6-bit k-quant, very high quality	10.9 GB	0.020	93.9%
Mellum2.1-12B-A2.5B-Thinking-Q4_K_M.gguf	Q4_K_M	4-bit k-quant, balanced (recommended)	8.1 GB	0.075	88.0%
Mellum2.1-12B-A2.5B-Thinking-MXFP4_MOE.gguf	MXFP4_MOE	MXFP4 4-bit on MoE experts, smallest	7.0 GB	0.115	85.6%

KL divergence and top-token agreement are measured against the BF16 logits on Wikitext-2 (n_ctx=512); lower KLD / higher agreement means closer to the unquantized model.

Download

hf download JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF Mellum2.1-12B-A2.5B-Thinking-Q4_K_M.gguf --local-dir .

Run with llama.cpp

# Pull and serve in one step (downloads the GGUF automatically; the suffix picks the quantization)
llama-server -hf JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF:Q4_K_M   --ctx-size 131072   --temp 0.6 --top-p 0.95 --top-k 20

# Or run a one-off prompt with a local file
llama-cli -m Mellum2.1-12B-A2.5B-Thinking-Q4_K_M.gguf   --ctx-size 131072   --temp 0.6 --top-p 0.95 --top-k 20   -p "Is 1024 a power of 2? Explain your reasoning."

The server exposes an OpenAI-compatible API on http://localhost:8080/v1:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="llama.cpp")

chat_response = client.chat.completions.create(
    model="JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF",
    messages=[
        {"role": "user", "content": "Is 1024 a power of 2? Explain your reasoning."},
    ],
    max_tokens=81920,
    temperature=0.6,
    top_p=0.95,
    extra_body={"top_k": 20},
)
print(chat_response.choices[0].message.content)

Run with Ollama

ollama run hf.co/JetBrains/Mellum2.1-12B-A2.5B-Thinking-GGUF:Q4_K_M

License

Released under the Apache 2.0 license.

---

For the full model card, evaluation results, and architecture details, refer to the original model: JetBrains/Mellum2.1-12B-A2.5B-Thinking.