mlx-community/VibeThinker-3B-OptiQ-4bit

A 4-bit mixed-precision MLX quant produced by mlx-optiq — the sensitivity-aware quantization toolkit for Apple Silicon. It edges stock uniform 4-bit on the six-metric Capability Score and stays markedly closer to the original bf16 weights (KL 0.58 vs 1.44).

A 4-bit mixed-precision MLX quant of mlx-community/VibeThinker-3B (a reasoning model fine-tuned from Qwen2.5-Coder-3B). Per-layer bit-widths come from a KL-divergence sensitivity pass on a six-domain calibration mix (prose · reasoning · code · agent · tool-call · constraint-bearing instructions). Sensitive layers go to 8-bit; robust ones stay at 4-bit.

Quantization details

Property	Value
Predominant precision	4-bit
Layers at 8-bit (sensitive)	141
Layers at 4-bit (robust)	111
Total quantized layers	252
Achieved bits per weight	5.12
Group size	64
Calibration mix	six-domain mix (40 samples × 6 domains)
Reference for sensitivity	bf16

We follow the same naming convention llama.cpp uses for Q4_K_M and similar mixed-precision quants: the "4-bit" label is for the predominant precision, not the weighted average. The few sensitive layers held at 8-bit make this build +0.5 GB larger than a stock uniform-4-bit quant (2.1 GB vs 1.6 GB) while recovering quality the uniform quant loses.

Usage

Load it with mlx-lm and use it as usual:

pip install mlx-lm

from mlx_lm import load, generate

model, tokenizer = load("mlx-community/VibeThinker-3B-OptiQ-4bit")
response = generate(
    model, tokenizer,
    prompt="Explain quantum computing in simple terms.",
    max_tokens=512,
)

This is a reasoning model — it emits a <think>…</think> block before its final answer. Give it room with max_tokens.

For more (mixed-precision KV-cache serving, sensitivity-aware LoRA fine-tuning, OpenAI + Anthropic-compatible inference server with structured/JSON output, hot-swap mounted adapters), install mlx-optiq:

pip install mlx-optiq

License

MIT (inherits from the base model).

VibeThinker-3B-OptiQ-4bit

Model card