Hugging Peers

flint-nvfp4-coldgptq

☆ Save
by FutureMLS-Lab text-generation transformers apache-2.0
1 downloads 22 likes on Hugging Face updated

Model card

flint-nvfp4-coldgptq

NVFP4 quantization of RecursiveSI/flint (revision 2a247be), Recursive's 39.8B-total / ~2.5B-active mixture-of-experts model for coding agents. The checkpoint is 23.5 GB instead of 79.6 GB, so one Blackwell GPU keeps most of its memory for KV cache.

What is quantized

  • The 12,032 routed experts of MoE layers 2–48 (gate_proj, up_proj, down_proj) are static W4A4 NVFP4: FP4 E2M1 weights in 16-element blocks with FP8 E4M3 block scales and FP32 global scales, in NVIDIA ModelOpt's per-expert layout (weight, weight_scale, weight_scale_2, input_scale).
  • Every other tensor is the unchanged BF16 source tensor: attention and its learned sinks, routers, shared experts, the dense layers 0–1, norms, embeddings and lm_head.

Evaluation

B300, SGLang 0.5.16 with flint_sglang 0.1.4 and the base repository's serve_b300.sh flags, using the model's default sampling (temperature 1.0, top-p 0.95, top-k 20). Accuracy is the mean over all samples; generation length counts completion tokens, reasoning included.

Benchmark Accuracy, BF16 → NVFP4 Generation length p50 / p90 (tokens), BF16 → NVFP4
GSM8K, 1,319 questions 96.1 → 96.0 451 / 2,796 → 459 / 3,004
GPQA Diamond, 198 questions × 3 samples 69.2 → 68.9 6,718 / 30,918 → 7,440 / 30,553
AIME 2026, 30 problems × 16 samples 84.4 → 83.8 7,927 / 32,768 → 8,294 / 32,768

GSM8K completions are capped at 16,384 tokens and GPQA and AIME at 32,768. The cap was reached by 8.8% / 7.7% of GPQA samples and 13.3% / 14.2% of AIME samples (BF16 / NVFP4), so AIME's p90 is the cap for both. In GPU memory the NVFP4 weights take 23.6 GB, against 74.2 GB for BF16.

Usage

NVFP4 needs a Blackwell GPU; this checkpoint was validated on B300. Install the Flint SGLang kit from the base repository (sglang/install.sh, or its Dockerfile), then point the kit's serve script at this checkpoint. SGLang reads the ModelOpt NVFP4 configuration from config.json, and the kit's default flags (FA4 attention, flashinfer_trtllm_routed MoE) work unchanged.

hf download FutureMLS-Lab/flint-nvfp4-coldgptq --local-dir flint-nvfp4-coldgptq
hf download RecursiveSI/flint --include "sglang/*" --local-dir flint-kit
MODEL=$PWD/flint-nvfp4-coldgptq bash flint-kit/sglang/serve_b300.sh

Use the chat template and sampling defaults shipped with the checkpoint, as described in the base model card.

License

Apache-2.0, the same license as the base model. See RecursiveSI/flint for the model's intended uses and limitations.