MeanVC2: Robust Low-Latency Streaming Zero-Shot Voice Conversion

模型调用说明

这个目录只存放 MeanVC2 的模型权重和配置文件。

如果别人单独下载了这个目录，需要再配合 MeanVC2 的运行时代码一起用；仅有这些 checkpoint 文件本身不能直接完成推理。

这个目录里的主要文件

• meanvc2_120ms_40ms.safetensors: 当前更推荐的模型，延迟稍高一点，但音质和稳定性更好
• meanvc2_40ms_40ms.safetensors: 更低延迟版本
• config_120ms_40ms.json: 120ms 版本配置
• config_40ms_40ms.json: 40ms 版本配置
• vocos.pt: 声码器
• wavlm_large_cfg.pt / wavlm_large_finetune.pth: 参考音色相关模型
• fastu2pp_80ms.pt / fastu2pp_160ms.pt: 运行时依赖的前端模型

最小代码调用示例

下面这段示例使用的是本仓库现成的 MeanVC2Service 封装，适合想直接把这套 checkpoint 跑起来的人：

import numpy as np
import soundfile as sf

from app.services.meanvc2 import MeanVC2Service

service = MeanVC2Service(device="cuda", model_variant="120ms")
service.load(reference_audio_path="target.wav")

audio, sample_rate = sf.read("source.wav", dtype="float32")
audio = np.asarray(audio, dtype=np.float32)
if audio.ndim > 1:
    audio = audio.mean(axis=1)

chunk_size = service.get_stream_chunk_samples()
outputs = []
for start in range(0, len(audio), chunk_size):
    chunk = audio[start : start + chunk_size]
    converted, out_sr = service.process_stream_chunk(chunk, sample_rate)
    if converted.size:
        outputs.append(converted)

tail, out_sr = service.flush_stream()
if tail.size:
    outputs.append(tail)

if outputs:
    merged = np.concatenate(outputs).astype(np.float32, copy=False)
    sf.write("converted.wav", merged, out_sr)

也可以直接用官方脚本

如果你已经有完整的 MeanVC2 仓库，可以直接用它自带的脚本：

python src/infer/infer_e2e.py --model 120ms --source-wav source.wav --target-wav target.wav --output-wav output.wav
python runtime/run_rt.py --mode realtime --model 40ms

注意

• MeanVC2 是语音转语音模型，不是文本转语音
• 必须先提供参考音频，再处理源语音
• 更推荐 120ms 版本，质量通常更稳
• 不要随意修改这些文件名，运行时代码通常会按固定名字查找

MeanVC2 is a robust, low-latency streaming zero-shot voice conversion (VC) system built upon the diffusion-based conditional flow matching (CFM) framework. By introducing Future-Receptive Chunking (FRC) and a Universal Timbre Token Encoder (UTTE), MeanVC2 achieves high-fidelity voice conversion with an end-to-end pipeline latency of only 110 ms while maintaining superior speaker similarity and audio naturalness even with a 40 ms chunk size.

✨ Key Features

• 🚀 Ultra-Low Latency Streaming: 110 ms end-to-end first-packet latency with 40 ms chunk size; full pipeline RTF < 0.633 on single CPU core.
• ⚡ Single-Step Generation: Mean flows + 1-NFE ODE solving for high-quality mel-spectrogram synthesis.
• 🎯 Zero-Shot Capability: Convert to any unseen target speaker without re-training, robust under low-quality reference audio.
• 💾 Lightweight: Only 18M parameters — far smaller than competing streaming VC systems.
• 🔊 High Fidelity: Future-Receptive Chunking (FRC) with Universal Timbre Token Encoder (UTTE) for pronunciation-aware timbre modeling.

📁 Model Checkpoints

This repository hosts all pretrained models for MeanVC2:

Voice Conversion Models

Vocoder (JIT)

ASR Encoder (Fast-U2++ JIT)

💻 Quick Start

1. Clone and Install

git clone https://github.com/ASLP-lab/MeanVC2.git
cd MeanVC2

conda create -n meanvc2 python=3.11 -y
conda activate meanvc2

pip install torch==2.5.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu121
pip install -r requirements.txt

2. Download Pretrained Models

python initialization.py --task all

Or download selectively:

python initialization.py --task preprocess   # BN + SpkEmb extraction only
python initialization.py --task train_120ms  # preprocess + 120ms VC + vocoder
python initialization.py --task train_40ms   # preprocess + 40ms VC + vocoder

3. End-to-End Voice Conversion

No pre-extracted features needed — input two wavs, output converted audio:

# 120ms+40ms model (recommended for quality)
python src/infer/infer_e2e.py \
    --source-wav /path/to/source.wav \
    --target-wav /path/to/target.wav \
    --ckpt-path ckpts/pretrained_models/meanvc2_120ms_40ms.safetensors \
    --model-config src/config/config_120ms_40ms.json \
    --vocoder-ckpt-path ckpts/vocos/vocos.pt \
    --chunk-size 12 --steps 3 \
    --output-wav output.wav

# 40ms+40ms model (lower latency)
python src/infer/infer_e2e.py \
    --source-wav /path/to/source.wav \
    --target-wav /path/to/target.wav \
    --ckpt-path ckpts/pretrained_models/meanvc2_40ms_40ms.safetensors \
    --model-config src/config/config_40ms_40ms.json \
    --vocoder-ckpt-path ckpts/vocos/vocos.pt \
    --chunk-size 4 --steps 3 \
    --output-wav output.wav

4. Real-Time Streaming

cd runtime

# File mode
python run_rt.py --mode file --input in.wav --output out.wav --model 120ms

# Microphone mode
python run_rt.py --mode realtime --model 40ms

🏗️ Model Architecture

Total parameters: ~18M

📜 License & Disclaimer

MeanVC2 is released under the Apache License 2.0. This open-source license allows you to freely use, modify, and distribute the model, as long as you include the appropriate copyright notice and disclaimer.

MeanVC2 is designed for research and legitimate applications in voice conversion technology. Users must obtain proper consent from individuals whose voices are being converted or used as references. We strongly discourage malicious use including impersonation, fraud, or creating misleading audio content. Users are solely responsible for ensuring compliance with ethical standards and legal requirements.

📄 Citation

If you find our work helpful, please cite:

@article{ma2026meanvc2,
  title={MeanVC2: Robust Low-Latency Streaming Zero-Shot Voice Conversion},
  author={Ma, Guobin and Xia, Yuxuan and Jiang, Yuepeng and Guo, Dake and Xie, Hanke and Hu, Jingbin and Wang, Yanbo and Xie, Lei and Zhu, Pengcheng},
  journal={arXiv preprint arXiv:2606.09050},
  year={2026}
}

📧 Contact

For questions or collaborations, please contact: guobin.ma@mail.nwpu.edu.cn