MiniCPM-V-4.7-35B-A3B-Q4_K_M

This repository hosts the GGUF (llama.cpp) quantized version of MiniCPM-V 4.7 35B-A3B. For the original BF16 weights and the full model card, please refer to openbmb/MiniCPM-V-4.7-35B-A3B.

Get started

Model weights  ·  Inference  ·  Fine-tuning  ·  Live demo  ·  Local deployment  ·  API  ·  Cookbook

Installation

pip install "transformers[torch]>=5.18.0" torchvision torchcodec

Note on CUDA compatibility: torchcodec (used for video decoding) may have compatibility issues with certain CUDA versions. For example, torch>=2.11 bundles CUDA 13.1 by default, while environments with CUDA 12.x may encounter errors such as RuntimeError: Could not load libtorchcodec. Two workarounds:

1. Replace torchcodec with PyAV — supports both image and video inference without CUDA version constraints:

   pip install "transformers[torch]>=5.18.0" torchvision av
   
   

2. Pin the CUDA version when installing torch to match your environment (e.g. CUDA 12.8):

   pip install "transformers>=5.18.0" torchvision torchcodec --index-url https://download.pytorch.org/whl/cu128
   
   

Load the Model

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor

model_id = "openbmb/MiniCPM-V-4.7-35B-A3B"

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForImageTextToText.from_pretrained(
    model_id, torch_dtype="auto", device_map="auto"
)

# Flash Attention 2 is recommended for faster inference and lower GPU memory usage,
# especially in multi-image and video scenarios.
# model = AutoModelForImageTextToText.from_pretrained(
#     model_id,
#     torch_dtype=torch.bfloat16,
#     attn_implementation="flash_attention_2",
#     device_map="auto",
# )

Image Inference

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/openbmb/DemoCase/resolve/main/refract.png"},
            {"type": "text", "text": "What causes this phenomenon?"},
        ],
    }
]

downsample_mode = "16x"  # Using `downsample_mode="4x"` for Finer Detail

inputs = processor.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True,
    return_dict=True, return_tensors="pt",
    downsample_mode=downsample_mode,
    max_slice_nums=36,
    enable_thinking=False,
).to(model.device)

generated_ids = model.generate(**inputs, downsample_mode=downsample_mode, max_new_tokens=512)
generated_ids_trimmed = [
    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text[0])

Video Inference

messages = [
    {
        "role": "user",
        "content": [
            {"type": "video", "url": "https://huggingface.co/datasets/openbmb/DemoCase/resolve/main/football.mp4"},
            {"type": "text", "text": "Describe this video in detail. Follow the timeline and focus on on-screen text, interface changes, main actions, and scene changes."},
        ],
    }
]

downsample_mode = "16x"  # Using `downsample_mode="4x"` for Finer Detail

inputs = processor.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True,
    return_dict=True, return_tensors="pt",
    downsample_mode=downsample_mode,
    max_num_frames=128,
    stack_frames=1,
    max_slice_nums=1,
    use_image_id=False,
    enable_thinking=False,
).to(model.device)

generated_ids = model.generate(**inputs, downsample_mode=downsample_mode, max_new_tokens=2048)
generated_ids_trimmed = [
    out_ids[len(in_ids):] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text[0])

Advanced Parameters

You can customize image/video processing by passing additional parameters to apply_chat_template:

Note: downsample_mode must be passed to both apply_chat_template (for correct placeholder count) and generate (for the vision encoder). All other parameters only need to be passed to apply_chat_template.

Serving with transformers serve

Hugging Face Transformers includes a lightweight OpenAI-compatible server for quick testing and moderate-load deployment.

pip install "transformers[serving]>=5.18.0"

Start the server:

transformers serve openbmb/MiniCPM-V-4.7-35B-A3B --port 8000 --host 0.0.0.0

Send a request:

curl -s http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "openbmb/MiniCPM-V-4.7-35B-A3B",
    "messages": [{
      "role": "user",
      "content": [
        {"type": "image_url", "image_url": {"url": "https://huggingface.co/datasets/openbmb/DemoCase/resolve/main/refract.png"}},
        {"type": "text", "text": "What causes this phenomenon?"}
      ]
    }]
  }'

Tool calling example:

curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "openbmb/MiniCPM-V-4.7-35B-A3B",
  "messages": [{"role": "user", "content": [
    {"type": "text", "text": "the weather of Beijing"}
  ]}],
  "tools": [{
    "type": "function",
    "function": {
      "name": "get_weather",
      "description": "Get the current weather for a given location",
      "parameters": {
        "type": "object",
        "properties": {
          "location": {"type": "string", "description": "City name"}
        },
        "required": ["location"]
      }
    }
  }]
}'

Speculative decoding (DSpark)

Inference with OpenVINO™

OpenVINO™ is an open-source toolkit for optimizing and deploying AI inference on Intel CPU/GPU/NPU platforms. OpenVINO provides day-0 support for MiniCPM-V-4.7 on Intel platforms with high-performance inference. See the OpenVINO™ Toolkit to get started. You can install and run the example with this notebook (link TBD).

On-Device Deployment

MiniCPM-V 4.7 supports deployment on iOS, Android, and HarmonyOS, with day-0 support for Rockchip, Houmo, Intel, AMD, and Arm chip platforms. See the edge deployment repository for the apps and source code.

Inference and Fine-Tuning

For supported inference and training frameworks, see Supported Inference and Training Frameworks and API. The MiniCPM-V 4.7 Cookbook covers deployment, fine-tuning, and quantization workflows.

Task guides

• Single-image chat
• Multi-image chat
• Video understanding
• OCR
• PDF parsing
• Visual grounding

Acknowledgements

Model Weights

MiniCPM-V 4.7 is available in the following variants:

Model	Description	Download
MiniCPM-V 4.7 35B-A3B	The first open-source MoE model to run smoothly on-device without compromising performance, raising the bar for on-device performance.	🤗
MiniCPM-V 4.7 35B-A3B FP8	FP8 quantized version with lower GPU memory usage.	🤗
MiniCPM-V 4.7 35B-A3B int4	int4 quantized version with lower GPU memory usage.	🤗
MiniCPM-V 4.7 35B-A3B AWQ	AWQ (W4A16) quantized version with lower GPU memory usage.	🤗
MiniCPM-V 4.7 35B-A3B GPTQ	GPTQ (W4A16) quantized version with lower GPU memory usage.	🤗
MiniCPM-V 4.7 35B-A3B GGUF	GGUF version with lower memory usage for CPU inference.	🤗
MiniCPM-V 4.7 35B-A3B DSpark	DSpark draft model for speculative decoding.	🤗

Also available on Ollama.