Model Overview

Description

All linear in DeepSeek-R1 are quantized to FP4 precision.

Evaluation

The table below summarizes our results on the MMLU benchmark.

To expedite testing, we randomly selected 25 samples from each subset of the MMLU dataset.

Our experiments show that the FP4 quantized model achieves accuracy on par with the BF16 version while delivering faster inference.

Quickstart

We recommend using the Chitu inference framework(https://github.com/thu-pacman/chitu) to run this model.

Here provides a simple command to show you how to run DeepSeek-R1-fp4.

torchrun \
    --nproc_per_node 8 \
    --master_port=22525 \
    -m chitu \
    serve.port=21002 \
    infer.cache_type=paged \
    infer.attn_type=flash_mla \
    infer.mla_absorb=absorb \
    models=DeepSeek-R1-fp4-all \
    models.ckpt_dir="your model path" \
    infer.tp_size=8 \
    infer.max_reqs=2 \
    infer.max_seq_len=2048 \
    request.max_new_tokens=512 \
    infer.use_cuda_graph=True

Contact

solution@qingcheng.ai