Qwen3.8-Flash-Next · Orinfer E8P Q2A8

中文 · ModelScope

Quantized official Qwen3.8-Flash-Next for Jetson AGX Orin 64GB / CUDA SM87, using Orinfer 0.1.3. Supports text, images, multiple images, thinking, native MTP, prefix caching and continuous batching.

Weights

• Routed experts: our integer E8P near-2-bit representation with signed-block128 rotation and INT8 computation. Lossless K64 storage rearrangement also supports exact A8 bit-plane Tensor Core paths.
• Single-stream decode and MTP verification: selected GDN QKV/Z, QSA Q/gate and the shared output head use Q4A8. Prefill and concurrent projections retain their W8 representations.
• Shared FFN, sensitive output projections and draft Q/O remain W8; HC keeps BF16 values and a lossless packed read representation; persistent GDN state is FP32. Vision weights are FP16. KV cache is group-64 INT8 with FP16 scales.
• Context capacity: 262,144 tokens, including prompt, image and generated tokens. Quantization seed: 20261002.
• 35 standard safetensors weight files, approximately 52.94 GB total: 27 GPU shards (38.87 GB) and 8 CPU embedding/PLE shards (14.08 GB). CPU tables are read by row with bounded caches; they are not uploaded in full to the GPU.

Shards use stable component/layer names. The standard cache/weights/model.safetensors.index.json maps all GPU tensors. CPU row locations are described by cache/cpu/inputs.json. These are Orinfer physical quantization layouts, not a generic Transformers checkpoint. Repacking changes containers and indices without changing tensor bytes. Full BF16/FP8 quality benchmarking is not implied by performance measurements.

Deploy

Download this model snapshot, then download both the Orinfer aarch64 CLI archive and the Flash Next execution archive from release 0.1.3. Extract the CLI archive, synchronize its fixed CPU tooling dependencies with uv, and install the execution archive:

hf download immiq/Qwen3.8-Flash-Next-Orinfer-E8P-Q2A8 --local-dir ./qwen-flash-next-orinfer

tar -xzf orinfer-v0.1.3-linux-aarch64.tar.gz
cd orinfer-v0.1.3-linux-aarch64
uv venv --python python3.10 .venv
uv pip sync --python .venv/bin/python tools/build/cpu-requirements.txt
PYTHONPATH=. .venv/bin/python tools/model/package.py install \
  ../orinfer-v0.1.3-flash_next-int8-quality-sm87-execution.tar.gz ~/.cache/orinfer/packages
./bin/orinfer serve ../qwen-flash-next-orinfer \
  --listen 0.0.0.0:8088 --model qwen-flash-next \
  --cuda-graph decode_only --mtp-drafts 3 --prefix-cache-mib 1024 \
  --default-request-params '{"enable_thinking":true}'

The model repository contains no execution binaries. Its execution_package references external package digest 3ed6c474aeb21b0f0e34f204a31e8404ee835148d2f051b70d64f5e7aa6ffa08. The package is installed in ~/.cache/orinfer/packages, or the directory selected by ORINFER_EXECUTION_CACHE. No online Python or kernel compilation is required. Source-build and API instructions are in the Orinfer repository.

The OpenAI-compatible endpoint is http://localhost:8088/v1/chat/completions. Explicit request fields override the thinking default. Streaming is opt-in with stream: true. Images use image_url.

Performance reference

Measured on Jetson AGX Orin 64GB, MAXN, GPU 1300.5 MHz, CUDA 12.6, with matching optimized kernels. These are reference measurements, not guarantees for every workload.

The complete code-generation requests naturally stopped after 6176–7598 output tokens under a 16K total output budget. Decode counts committed thinking and answer tokens, excluding rejected drafts. Measurement methods, quality limitations and prefix-hit latency are documented in Orinfer performance.

License

Weights retain the upstream Qwen Community License. Orinfer and its separate execution package are LGPL-3.0-or-later; bundled third-party notices are provided with the software release.