Hugging Peers

Hy3-SF-INT8

☆ Save
by siliconflow text-generation Pytorch Apache License 2.0
2.5K downloads 1 likes on ModelScope updated

Model card

Hy3-SF-INT8

Hy3-SF-INT8 is a rotated W8A8 INT8 checkpoint based on Tencent-Hunyuan/Hy3-FP8, published by SiliconFlow for efficient text generation and deployment.

Model information

  • Base model: Tencent Hy3
  • Architecture: 295B Mixture-of-Experts, 21B active parameters
  • Task: Text generation
  • Context length: 256K
  • Languages: Chinese and English
  • Weight format: Safetensors
  • License: Apache License 2.0

The tokenizer, chat template, generation configuration, rotation matrix, and one MTP layer are included in this repository.

Quantization

The checkpoint was produced with CrossingBits:

Item Setting
Weight / activation dtype INT8 / INT8
Granularity Per-channel
Recipe Rotation
Router BF16, excluded from quantization to preserve expert selection
Main LM head BF16
MTP Included and quantized

This is a CrossingBits checkpoint. Native Hy3 Hugging Face, vLLM, and SGLang loaders do not consume its internal W8A8 layout directly.

Verification

The saved artifact passed conversion-aware structural, numerical, and resident inference checks:

  • Manifest: 187,087 output tensors; 0 missing keys, unexplained extras, or shape/dtype mismatches.
  • Shard integrity: all 99 safetensors shards match model.safetensors.index.json; no missing or orphan shards.
  • All-zero sweep: 187,087 tensors checked with 0 corruption failures. The only all-zero output tensor is the MTP expert_bias, whose canonical source is also all zero.
  • Canonical W8A8 samples: eight weights spanning early, middle, late, and MTP layers achieved cosine similarity 0.999962–0.999964 and relative error 0.847%–0.870% against provider-FP8-dequantized, fused, rotated references.
  • Excluded state: all 80 BF16 routers passed forward-equivalence checks; 242 unchanged tensors matched exactly; 166 fused norm tensors were exact identity values.
  • Resident main decode: 273.8 GiB INT8 + 2.0 GiB BF16 + 0.8 GiB FP32 loaded across four H20 GPUs without CPU/disk offload. Prompt What is the capital of France? produced The capital of France is Paris.
  • MTP acceptance: the official five-sample evaluation covered 552 positions and measured Top-1 acceptance 0.5217 and Top-5 acceptance 0.7029.

The full-run calibration report measured KLD 0.0801 and rejection rate 0.0351.

Download

from modelscope import snapshot_download

model_dir = snapshot_download("siliconflow/Hy3-SF-INT8")

Or clone it with Git LFS:

git lfs install
git clone https://www.modelscope.cn/siliconflow/Hy3-SF-INT8.git

Usage

Use the CrossingBits loader or an inference runtime with explicit support for the CrossingBits Hy3 W8A8 format. A minimal resident smoke test is:

CUDA_VISIBLE_DEVICES=0,1,2,3 USE_MTP=0 python -m crossingbits.tools.tiny_infer \
  --model /path/to/Hy3-SF-INT8 \
  --prompt "What is the capital of France?" \
  --max-new-tokens 16 --dtype bf16

Evaluate the included MTP layer with:

CUDA_VISIBLE_DEVICES=0,1,2,3 USE_MTP=1 python evals/eval_mtp_acceptance.py \
  --model /path/to/Hy3-SF-INT8 --dtype bf16 --top-k 5

Use the included tokenizer and chat template for chat applications. Model capabilities and limitations generally follow the Hy3 base model; evaluate quality and performance for your target workload before production deployment.

License

This checkpoint follows the Apache License 2.0 terms of the Hy3 base model. See LICENSE for details.