Qwen3-VL-Reranker-8B-AWQ

This repository contains a language-only AWQ 4-bit quantized version of Qwen/Qwen3-VL-Reranker-8B.

The quantization target is practical deployment on constrained GPUs such as NVIDIA A10 24GB, especially when colocating the reranker model with the matching embedding model on the same card.

Quantization summary

• Base model: Qwen/Qwen3-VL-Reranker-8B
• Quantization: AWQ 4-bit
• Quantized modules: model.language_model.layers.* linear layers
• Preserved in fp16/bf16: visual tower, visual projector/merger, embeddings, normalization layers, lm_head
• AWQ config:

{
  "bits": 4,
  "group_size": 128,
  "modules_to_not_convert": ["visual", "lm_head"],
  "quant_method": "awq",
  "version": "gemm",
  "zero_point": true
}

Size and loading footprint

• Quantized repository size: about 6.8GB
• Observed GPU allocation in smoke loading test: about 6.77GiB

This makes it feasible to test this reranker together with the matching AWQ embedding model on a single 24GB GPU, with conservative context length and batch/concurrency settings.

Validation

A small text-only reranking smoke test was run against the original fp16/bf16 checkpoint.

Results:

num_pairs: 12
groups: 3 query groups x 4 docs
top1_match_rate: 1.0
Spearman per group: 0.8 / 0.8 / 0.8

The top-1 result matched for all test query groups. Some non-top1 internal ordering changed slightly, which is expected for a small AWQ smoke test.

Validation artifact:

reranker_awq_consistency_text_only.json

Note: this is a small text-only smoke test, not a full multimodal reranking benchmark. The visual tower is intentionally kept unquantized to reduce multimodal quality loss.

Loading example

This model was produced with a custom Qwen3-VL language-only AutoAWQ wrapper. The quantization script is included in this repository:

quant_qwen3vl_awq_language_only.py

Example:

import sys
sys.path.insert(0, ".")
from quant_qwen3vl_awq_language_only import register_qwen3vl_awq
register_qwen3vl_awq()

from awq import AutoAWQForCausalLM

model = AutoAWQForCausalLM.from_quantized(
    "yobo2u/Qwen3-VL-Reranker-8B-AWQ",
    trust_remote_code=True,
    safetensors=True,
    device_map={"": 0},
    fuse_layers=False,
)

Suggested A10 24GB deployment starting point

When colocating this reranker model with the matching embedding model on one A10 24GB GPU, start conservatively:

max_model_len: 2048-4096
max_num_seqs: 1
max_num_batched_tokens: 2048-4096
limit_mm_per_prompt: {"image": 1}

If serving two separate processes on one GPU, start around:

Embedding gpu_memory_utilization: 0.40-0.45
Reranker  gpu_memory_utilization: 0.40-0.45

Then increase context/concurrency empirically.

License

This derivative quantized model follows the original model license: Apache-2.0.

ModelScope remote snapshot test

This repository version was downloaded back from ModelScope with snapshot_download and tested locally after upload.

Test metadata:

repo_id: yobo2u/Qwen3-VL-Reranker-8B-AWQ
source: ModelScope snapshot_download
test_time: 2026-05-18 09:31:44 CST
max_length: 512
query_groups: 4
query_doc_pairs: 16

Load / memory:

load_ok: true
load_seconds: 7.647
gpu_alloc_gib_after_load: 6.761
gpu_reserved_gib_after_load: 6.812

Reranking smoke result:

top1_expected_relevant_match_count: 4 / 4
top1_expected_relevant_all_match: true
latency_seconds_min: 0.0529
latency_seconds_mean: 0.1646
latency_seconds_max: 1.5346

The remote ModelScope snapshot loaded successfully with the included quant_qwen3vl_awq_language_only.py registration helper and passed a small 4-query / 16-pair text-only reranking smoke test. This is a functional smoke test, not a full retrieval benchmark.

Full JSON result:

modelscope_remote_reranker_awq_test_results.json