DeepSeek-V3.2-w8a8

说明：该模型是通过参考msmodelslim 量化而来，参考教程：DeepSeek 量化案例

基础模型

基础模型: 【deepseek-ai/DeepSeek-V3.2】

量化过程

source /usr/local/Ascend/ascend-toolkit/set_env.sh
source /usr/local/Ascend/nnal/atb/set_env.sh
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:False
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7


# 创建虚拟环境
conda create -y -n msit_env python=3.10
conda activate msit_env

# 安装依赖包
pip3 install attrs cython 'numpy>=1.19.2,<=1.24.0' decorator sympy cffi pyyaml pathlib2 psutil protobuf==3.20.0 scipy requests absl-py --user
pip install torch==7.1.0 torch_npu==2.7.1rc1
pip install transformers==4.48.2
git clone https://gitcode.com/Ascend/msit.git
cd ./msit/msmodelslim
bash install.sh 

# DeepSeek-V3.2(含MTP层) W8A8 混合量化
nohup msmodelslim quant \
 --model_path ./model/DeepSeek-V3.2 \
 --save_path ./model/DeepSeek-V3.2-w8a8 \
 --model_type DeepSeek-V3.2-Exp \
 --quant_type w8a8 \
 --device npu \
 --trust_remote_code True > quant_v3.2_w8a8.log 2>&1 &

tail -f quant_w8a8.log

使用 vLLM-Ascend 部署 DeepSeek-V3.2（双机版）

启动容器（两台机器同时执行）

export IMAGE=quay.nju.edu.cn/ascend/vllm-ascend:v0.11.0rc0-deepseek-v3.2-exp
export NAME=vllm-ascend

docker run --rm \
  --name $NAME \
  --net=host \
  --device /dev/davinci0 \
  --device /dev/davinci1 \
  --device /dev/davinci2 \
  --device /dev/davinci3 \
  --device /dev/davinci4 \
  --device /dev/davinci5 \
  --device /dev/davinci6 \
  --device /dev/davinci7 \
  --device /dev/davinci_manager \
  --device /dev/devmm_svm \
  --device /dev/hisi_hdc \
  -v /usr/local/dcmi:/usr/local/dcmi \
  -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
  -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
  -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
  -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
  -v /etc/ascend_install.info:/etc/ascend_install.info \
  -v /mnt/data/models:/mnt/data/models \
  -it $IMAGE bash

启动node 0

#!/bin/sh
# ip和网卡名称通过ifconfig命令获取
nic_name="xxxx"
local_ip="xxxx"

MODEL_PATH="你的模型存放路径"

export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=100
export HCCL_BUFFSIZE=1024
export HCCL_OP_EXPANSION_MODE="AIV"
export PYTORCH_NPU_ALLOC_CONF="expandable_segments:True"

vllm serve "${MODEL_PATH}" \
  --host 0.0.0.0 \
  --port 8888 \
  --data-parallel-size 2 \
  --data-parallel-size-local 1 \
  --data-parallel-address $local_ip \
  --data-parallel-rpc-port 13389 \
  --tensor-parallel-size 8 \
  --seed 1024 \
  --served-model-name DeepSeek-V3.2 \
  --enable-expert-parallel \
  --enable-auto-tool-choice \
  --tool-call-parser deepseek_v31 \
  --reasoning-parser deepseek_r1 \
  --max-num-seqs 16 \
  --max-model-len 32768 \
  --max-num-batched-tokens 32768 \
  --trust-remote-code \
  --quantization ascend \
  --no-enable-prefix-caching \
  --gpu-memory-utilization 0.95 \
  --additional-config '{"ascend_scheduler_config":{"enabled":true},"torchair_graph_config":{"enabled":true,"graph_batch_sizes":[16]}}'

启动node1

#!/bin/sh
# ip和网卡名称通过ifconfig命令获取
nic_name="xxxx"
local_ip="xxxx"
node0_ip="your node0_ip"

MODEL_PATH="你的模型存放路径"

export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=100
export HCCL_BUFFSIZE=1024
export HCCL_OP_EXPANSION_MODE="AIV"
export PYTORCH_NPU_ALLOC_CONF="expandable_segments:True"

vllm serve "${MODEL_PATH}" \
  --host 0.0.0.0 \
  --port 8888 \
  --headless \
  --data-parallel-size 2 \
  --data-parallel-size-local 1 \
  --data-parallel-start-rank 1 \
  --data-parallel-address $node0_ip \
  --data-parallel-rpc-port 13389 \
  --tensor-parallel-size 8 \
  --seed 1024 \
  --served-model-name DeepSeek-V3.2 \
  --max-num-seqs 16 \
  --max-model-len 32768 \
  --max-num-batched-tokens 32768 \
  --enable-expert-parallel \
  --enable-auto-tool-choice \
  --tool-call-parser deepseek_v31 \
  --reasoning-parser deepseek_r1 \
  --trust-remote-code \
  --quantization ascend \
  --no-enable-prefix-caching \
  --gpu-memory-utilization 0.95 \
  --additional-config '{"ascend_scheduler_config":{"enabled":true},"torchair_graph_config":{"enabled":true,"graph_batch_sizes":[16]}}'

测试

curl http://127.0.0.1:8888/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "DeepSeek-V3.2",
        "prompt": "你是谁?",
        "max_tokens": 16,
        "temperature": 0
      }'

【Logs】

2025-12-09
1. initial commit

【Model Files】

File Size	Last Updated
`694.47GB`	`2025-12-09`

【Model Download】

nohup modelscope download --model taoxiaoxin/DeepSeek-V3.2-w8a8 --local_dir your_local_path > dd.log 2>&1 &

tail -f dd.log

DeepSeek-V3.2: Efficient Reasoning & Agentic AI

Technical Report👁️

Introduction

We introduce DeepSeek-V3.2, a model that harmonizes high computational efficiency with superior reasoning and agent performance. Our approach is built upon three key technical breakthroughs:

DeepSeek Sparse Attention (DSA): We introduce DSA, an efficient attention mechanism that substantially reduces computational complexity while preserving model performance, specifically optimized for long-context scenarios.
Scalable Reinforcement Learning Framework: By implementing a robust RL protocol and scaling post-training compute, DeepSeek-V3.2 performs comparably to GPT-5. Notably, our high-compute variant, DeepSeek-V3.2-Speciale, surpasses GPT-5 and exhibits reasoning proficiency on par with Gemini-3.0-Pro.
- Achievement: 🥇 Gold-medal performance in the 2025 International Mathematical Olympiad (IMO) and International Olympiad in Informatics (IOI).
Large-Scale Agentic Task Synthesis Pipeline: To integrate reasoning into tool-use scenarios, we developed a novel synthesis pipeline that systematically generates training data at scale. This facilitates scalable agentic post-training, improving compliance and generalization in complex interactive environments.

We have also released the final submissions for IOI 2025, ICPC World Finals, IMO 2025 and CMO 2025, which were selected based on our designed pipeline. These materials are provided for the community to conduct secondary verification. The files can be accessed at assets/olympiad_cases.

Chat Template

DeepSeek-V3.2 introduces significant updates to its chat template compared to prior versions. The primary changes involve a revised format for tool calling and the introduction of a "thinking with tools" capability.

To assist the community in understanding and adapting to this new template, we have provided a dedicated encoding folder, which contains Python scripts and test cases demonstrating how to encode messages in OpenAI-compatible format into input strings for the model and how to parse the model's text output.

A brief example is illustrated below:

import transformers
# encoding/encoding_dsv32.py
from encoding_dsv32 import encode_messages, parse_message_from_completion_text

tokenizer = transformers.AutoTokenizer.from_pretrained("deepseek-ai/DeepSeek-V3.2")

messages = [
    {"role": "user", "content": "hello"},
    {"role": "assistant", "content": "Hello! I am DeepSeek.", "reasoning_content": "thinking..."},
    {"role": "user", "content": "1+1=?"}
]
encode_config = dict(thinking_mode="thinking", drop_thinking=True, add_default_bos_token=True)

# messages -> string
prompt = encode_messages(messages, **encode_config)
# Output: "<｜begin▁of▁sentence｜><｜User｜>hello<｜Assistant｜></think>Hello! I am DeepSeek.<｜end▁of▁sentence｜><｜User｜>1+1=?<｜Assistant｜><think>"

# string -> tokens
tokens = tokenizer.encode(prompt)
# Output: [0, 128803, 33310, 128804, 128799, 19923, 3, 342, 1030, 22651, 4374, 1465, 16, 1, 128803, 19, 13, 19, 127252, 128804, 128798]

Important Notes:

This release does not include a Jinja-format chat template. Please refer to the Python code mentioned above.
The output parsing function included in the code is designed to handle well-formatted strings only. It does not attempt to correct or recover from malformed output that the model might occasionally generate. It is not suitable for production use without robust error handling.
A new role named developer has been introduced in the chat template. This role is dedicated exclusively to search agent scenarios and is designated for no other tasks. The official API does not accept messages assigned to developer.

How to Run Locally

The model structure of DeepSeek-V3.2 and DeepSeek-V3.2-Speciale are the same as DeepSeek-V3.2-Exp. Please visit DeepSeek-V3.2-Exp repo for more information about running this model locally.

Usage Recommendations:

For local deployment, we recommend setting the sampling parameters to temperature = 1.0, top_p = 0.95.
Please note that the DeepSeek-V3.2-Speciale variant is designed exclusively for deep reasoning tasks and does not support the tool-calling functionality.

License

This repository and the model weights are licensed under the MIT License.

Citation

@misc{deepseekai2025deepseekv32,
      title={DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models}, 
      author={DeepSeek-AI},
      year={2025},
}

Contact

If you have any questions, please raise an issue or contact us at service@deepseek.com.

DeepSeek-V3.2-w8a8

Model card

DeepSeek-V3.2-w8a8

基础模型

量化过程

使用 vLLM-Ascend 部署 DeepSeek-V3.2（双机版）

启动容器（两台机器同时执行）

启动node 0

启动node1

测试

【Logs】

【Model Files】

【Model Download】

DeepSeek-V3.2: Efficient Reasoning & Agentic AI

Introduction

Chat Template

How to Run Locally

License

Citation

Contact