MiniMax-H3 Ref2VA First-Frame Lineart LoRA

给 MiniMax-H3 的 Ref2VA 任务（参考 → 视频 + 音频）加上线稿首帧控制能力的 LoRA，共训练 1000 step。给一张首帧线稿和一段 prompt，即可生成以该线稿为开场构图、并把线稿"上色完成"为成片动画的视频 + 音频。

本模型同时是 MiniMax-H3-TrainingAdapter（DeCFG Training Adapter）方案在 Ref2VA 任务上的验证案例：训练时把一个 Ref2VA 版的 Training Adapter 作为 preset lora 融进基座，训练结束后只保留新训出的 LoRA。

LoRA 权重文件：model.safetensors

Prompt 与参考图写法

参考图是一张首帧线稿（RGB），通过 references 传入，顺序对应 prompt 里的 <Picture 1>、<Picture 2>……：

references=[{"type": "image", "image": Image.open("lineart.png").convert("RGB")}]

prompt 采用 Ref2VA 的多字段结构，字段名保持字面量原样书写。所有字段的内容都用英文书写，不要写中文：

• subject_definitions、retention_analysis：逐字固定的英文模板，直接照抄下方完整示例中的对应段落，不要改写
• summary：固定英文前缀加一句英文场景描述，前缀照抄下方完整示例
• detailed_description：唯一完全自由编写的字段，英文。画面与分镜描述，多镜头用 [Shot 1] / [Shot 2] 标记，第一个镜头要写从 <Picture 1> 开始
• overall_soundscape / non_diegetic_music：环境音与背景音乐，不需要时写 N/A

完整示例：

subject_definitions: <Subject 1> is the complete visible scene content in <Picture 1>, including all depicted characters, objects, environment elements, poses, outlines, spatial relationships, and camera framing.
<Picture 1> is the line-art first frame of [Shot 1] at 0.00 seconds and provides the opening composition for the target video.

summary: [keyframe completion] The target video begins from <Picture 1>, preserves the line-defined content and composition of <Subject 1>, and develops it as finished 2D animation. A sun-drenched rural road scene featuring lush greenery and warm, nostalgic lighting.

retention_analysis: <Subject 1> (appears in [Shot 1]): fully_preserved - the characters, objects, environment elements, poses, outlines, spatial relationships, and camera framing established by the line-art first frame are retained.
<Picture 1> ([Shot 1] first frame): partially_preserved - its composition and structural content are retained at 0.00 seconds while the monochrome line art is rendered with the target colors, lighting, textures, and detail.

detailed_description: A sun-drenched rural road scene featuring lush greenery and warm, nostalgic lighting.
[Shot 1] The shot begins from <Picture 1>, preserving <Subject 1>'s outlines, subject placement, poses, spatial relationships, and camera framing while rendering the line art as finished 2D animation. A girl with a dark ponytail, wearing a white shirt and dark skirt, stands on a cracked asphalt road next to a mossy wall. A delivery person on a yellow scooter with a blue crate approaches from behind. [Shot 2] At 00:04.000, A close-up shows the delivery person in a red uniform and helmet looking at the girl. [Shot 3] At 00:05.000, A high-angle view shows the delivery person riding the scooter toward the camera, passing the girl who stands by the guardrail overlooking a river.

overall_soundscape: N/A

non_diegetic_music: N/A

快速开始

1. 安装 DiffSynth-Studio

pip install diffsynth

或从源码安装：

git clone https://github.com/modelscope/DiffSynth-Studio.git
cd DiffSynth-Studio
pip install -e .

2. 推理

import torch
from PIL import Image
from diffsynth.pipelines.minimax_h3_audio_video import MiniMaxH3Pipeline, ModelConfig
from diffsynth.utils.data.audio_video import write_video_audio

pipe = MiniMaxH3Pipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[
        ModelConfig(model_id="MiniMax/MiniMax-H3", origin_file_pattern="Ref2VA/text_encoder/model*.safetensors"),
        ModelConfig(model_id="MiniMax/MiniMax-H3", origin_file_pattern="Ref2VA/transformer/model*.safetensors"),
        ModelConfig(model_id="MiniMax/MiniMax-H3", origin_file_pattern="Ref2VA/video_vae/source/model.safetensors"),
        ModelConfig(model_id="MiniMax/MiniMax-H3", origin_file_pattern="Ref2VA/audio_vae/model.safetensors"),
    ],
    processor_config=ModelConfig(model_id="MiniMax/MiniMax-H3", origin_file_pattern="Ref2VA/processor/"),
    vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 5,
)

pipe.load_lora(pipe.dit, ModelConfig(
    model_id="mibei0804/MiniMax-H3-Ref2VA-FirstFrame-Lineart",
    origin_file_pattern="model.safetensors",
))

prompt = """subject_definitions: <Subject 1> is the complete visible scene content in <Picture 1>, including all depicted characters, objects, environment elements, poses, outlines, spatial relationships, and camera framing.
<Picture 1> is the line-art first frame of [Shot 1] at 0.00 seconds and provides the opening composition for the target video.

summary: [keyframe completion] The target video begins from <Picture 1>, preserves the line-defined content and composition of <Subject 1>, and develops it as finished 2D animation. <your scene description>

retention_analysis: <Subject 1> (appears in [Shot 1]): fully_preserved - the characters, objects, environment elements, poses, outlines, spatial relationships, and camera framing established by the line-art first frame are retained.

detailed_description: <your scene description>
[Shot 1] The shot begins from <Picture 1>, preserving <Subject 1>'s outlines, subject placement, poses, spatial relationships, and camera framing while rendering the line art as finished 2D animation. <your shot description>

overall_soundscape: N/A

non_diegetic_music: N/A"""

video, audio = pipe(
    prompt=prompt,
    height=768, width=1344, num_frames=124,
    num_inference_steps=30, seed=0, cfg_scale=1.0,
    references=[{"type": "image", "image": Image.open("lineart.png").convert("RGB")}],
)
write_video_audio(
    video=video, audio=audio, output_path="lineart_ref2va.mp4",
    fps=24, audio_sample_rate=pipe.audio_vae.sample_rate,
)

# To unload the LoRA:
# pipe.clear_lora()

基座权重与 processor 均位于 Ref2VA/ 子目录。

训练方法

使用 Ref2VA 版 Training Adapter 作为 preset lora

MiniMax-H3 基座是经过 CFG 蒸馏的，直接在上面做 LoRA 微调容易不稳定，或者破坏蒸馏后的 CFG-free 行为。Training Adapter 在训练期间把蒸馏后的 DiT 临时拉回接近蒸馏前的状态，为新 LoRA 提供更好的优化面。

Ref2VA 任务使用 MiniMax-H3-TrainingAdapter 仓库下的 Ref2VA 版文件 model_ref2va.safetensors。

Training Adapter 只在训练时用，推理时不要加载——本仓库发布的 model.safetensors 是已经脱离 Training Adapter 的独立 LoRA，直接加载到原版 MiniMax-H3 Ref2VA 即可。

思路来源：DiffSynth-Studio 的 Differential LoRA Training，以及 ai-toolkit 的 Training Adapter 概念。

完整训练脚本

8 卡训练，共 1000 step。

先下载 Training Adapter：

modelscope download --model DiffSynth-Studio/MiniMax-H3-TrainingAdapter --local_dir ./models/DiffSynth-Studio/MiniMax-H3-TrainingAdapter

Stage 1：数据预处理

accelerate launch --num_processes 8 examples/minimax_h3/model_training/train.py \
  --dataset_base_path "<your_dataset_dir>" \
  --dataset_metadata_path "<your_dataset_dir>/metadata_train.jsonl" \
  --data_file_keys "video,input_audio,references" \
  --extra_inputs "input_audio,references" \
  --max_pixels 399360 \
  --num_frames 124 \
  --dataset_repeat 1 \
  --model_id_with_origin_paths "MiniMax/MiniMax-H3:Ref2VA/text_encoder/model*.safetensors,MiniMax/MiniMax-H3:Ref2VA/video_vae/source/model.safetensors,MiniMax/MiniMax-H3:Ref2VA/audio_vae/model.safetensors" \
  --processor_path "MiniMax/MiniMax-H3:Ref2VA/processor/" \
  --learning_rate 1e-4 \
  --num_epochs 1 \
  --remove_prefix_in_ckpt "pipe.dit." \
  --output_path "./models/train/lineart_ref2va_cache" \
  --lora_base_model "dit" \
  --lora_target_modules "blocks.25.attn.qkv_proj,blocks.25.attn.out_proj,blocks.25.mlp.fc1,blocks.25.mlp.fc2,blocks.26.attn.qkv_proj,blocks.26.attn.out_proj,blocks.26.mlp.fc1,blocks.26.mlp.fc2,blocks.27.attn.qkv_proj,blocks.27.attn.out_proj,blocks.27.mlp.fc1,blocks.27.mlp.fc2,blocks.28.attn.qkv_proj,blocks.28.attn.out_proj,blocks.28.mlp.fc1,blocks.28.mlp.fc2,blocks.29.attn.qkv_proj,blocks.29.attn.out_proj,blocks.29.mlp.fc1,blocks.29.mlp.fc2,blocks.30.attn.qkv_proj,blocks.30.attn.out_proj,blocks.30.mlp.fc1,blocks.30.mlp.fc2,blocks.31.attn.qkv_proj,blocks.31.attn.out_proj,blocks.31.mlp.fc1,blocks.31.mlp.fc2,blocks.32.attn.qkv_proj,blocks.32.attn.out_proj,blocks.32.mlp.fc1,blocks.32.mlp.fc2,blocks.33.attn.qkv_proj,blocks.33.attn.out_proj,blocks.33.mlp.fc1,blocks.33.mlp.fc2,blocks.34.attn.qkv_proj,blocks.34.attn.out_proj,blocks.34.mlp.fc1,blocks.34.mlp.fc2,blocks.35.attn.qkv_proj,blocks.35.attn.out_proj,blocks.35.mlp.fc1,blocks.35.mlp.fc2,blocks.36.attn.qkv_proj,blocks.36.attn.out_proj,blocks.36.mlp.fc1,blocks.36.mlp.fc2,blocks.37.attn.qkv_proj,blocks.37.attn.out_proj,blocks.37.mlp.fc1,blocks.37.mlp.fc2,blocks.38.attn.qkv_proj,blocks.38.attn.out_proj,blocks.38.mlp.fc1,blocks.38.mlp.fc2,blocks.39.attn.qkv_proj,blocks.39.attn.out_proj,blocks.39.mlp.fc1,blocks.39.mlp.fc2,blocks.40.attn.qkv_proj,blocks.40.attn.out_proj,blocks.40.mlp.fc1,blocks.40.mlp.fc2,blocks.41.attn.qkv_proj,blocks.41.attn.out_proj,blocks.41.mlp.fc1,blocks.41.mlp.fc2,blocks.42.attn.qkv_proj,blocks.42.attn.out_proj,blocks.42.mlp.fc1,blocks.42.mlp.fc2,blocks.43.attn.qkv_proj,blocks.43.attn.out_proj,blocks.43.mlp.fc1,blocks.43.mlp.fc2,blocks.44.attn.qkv_proj,blocks.44.attn.out_proj,blocks.44.mlp.fc1,blocks.44.mlp.fc2,blocks.45.attn.qkv_proj,blocks.45.attn.out_proj,blocks.45.mlp.fc1,blocks.45.mlp.fc2,blocks.46.attn.qkv_proj,blocks.46.attn.out_proj,blocks.46.mlp.fc1,blocks.46.mlp.fc2,blocks.47.attn.qkv_proj,blocks.47.attn.out_proj,blocks.47.mlp.fc1,blocks.47.mlp.fc2,blocks.48.attn.qkv_proj,blocks.48.attn.out_proj,blocks.48.mlp.fc1,blocks.48.mlp.fc2,blocks.49.attn.qkv_proj,blocks.49.attn.out_proj,blocks.49.mlp.fc1,blocks.49.mlp.fc2" \
  --lora_rank 16 \
  --use_gradient_checkpointing_offload \
  --task "sft:data_process"

Stage 2：训练 LoRA

accelerate launch --num_processes 8 examples/minimax_h3/model_training/train.py \
  --dataset_base_path ./models/train/lineart_ref2va_cache \
  --data_file_keys "video,input_audio,references" \
  --extra_inputs "input_audio,references" \
  --max_pixels 399360 \
  --num_frames 124 \
  --dataset_repeat 8 \
  --model_id_with_origin_paths "MiniMax/MiniMax-H3:Ref2VA/transformer/model*.safetensors" \
  --processor_path "MiniMax/MiniMax-H3:Ref2VA/processor/" \
  --learning_rate 1e-4 \
  --num_epochs 1 \
  --remove_prefix_in_ckpt "pipe.dit." \
  --output_path "./models/train/lineart_ref2va" \
  --lora_base_model "dit" \
  --lora_target_modules "blocks.25.attn.qkv_proj,blocks.25.attn.out_proj,blocks.25.mlp.fc1,blocks.25.mlp.fc2,blocks.26.attn.qkv_proj,blocks.26.attn.out_proj,blocks.26.mlp.fc1,blocks.26.mlp.fc2,blocks.27.attn.qkv_proj,blocks.27.attn.out_proj,blocks.27.mlp.fc1,blocks.27.mlp.fc2,blocks.28.attn.qkv_proj,blocks.28.attn.out_proj,blocks.28.mlp.fc1,blocks.28.mlp.fc2,blocks.29.attn.qkv_proj,blocks.29.attn.out_proj,blocks.29.mlp.fc1,blocks.29.mlp.fc2,blocks.30.attn.qkv_proj,blocks.30.attn.out_proj,blocks.30.mlp.fc1,blocks.30.mlp.fc2,blocks.31.attn.qkv_proj,blocks.31.attn.out_proj,blocks.31.mlp.fc1,blocks.31.mlp.fc2,blocks.32.attn.qkv_proj,blocks.32.attn.out_proj,blocks.32.mlp.fc1,blocks.32.mlp.fc2,blocks.33.attn.qkv_proj,blocks.33.attn.out_proj,blocks.33.mlp.fc1,blocks.33.mlp.fc2,blocks.34.attn.qkv_proj,blocks.34.attn.out_proj,blocks.34.mlp.fc1,blocks.34.mlp.fc2,blocks.35.attn.qkv_proj,blocks.35.attn.out_proj,blocks.35.mlp.fc1,blocks.35.mlp.fc2,blocks.36.attn.qkv_proj,blocks.36.attn.out_proj,blocks.36.mlp.fc1,blocks.36.mlp.fc2,blocks.37.attn.qkv_proj,blocks.37.attn.out_proj,blocks.37.mlp.fc1,blocks.37.mlp.fc2,blocks.38.attn.qkv_proj,blocks.38.attn.out_proj,blocks.38.mlp.fc1,blocks.38.mlp.fc2,blocks.39.attn.qkv_proj,blocks.39.attn.out_proj,blocks.39.mlp.fc1,blocks.39.mlp.fc2,blocks.40.attn.qkv_proj,blocks.40.attn.out_proj,blocks.40.mlp.fc1,blocks.40.mlp.fc2,blocks.41.attn.qkv_proj,blocks.41.attn.out_proj,blocks.41.mlp.fc1,blocks.41.mlp.fc2,blocks.42.attn.qkv_proj,blocks.42.attn.out_proj,blocks.42.mlp.fc1,blocks.42.mlp.fc2,blocks.43.attn.qkv_proj,blocks.43.attn.out_proj,blocks.43.mlp.fc1,blocks.43.mlp.fc2,blocks.44.attn.qkv_proj,blocks.44.attn.out_proj,blocks.44.mlp.fc1,blocks.44.mlp.fc2,blocks.45.attn.qkv_proj,blocks.45.attn.out_proj,blocks.45.mlp.fc1,blocks.45.mlp.fc2,blocks.46.attn.qkv_proj,blocks.46.attn.out_proj,blocks.46.mlp.fc1,blocks.46.mlp.fc2,blocks.47.attn.qkv_proj,blocks.47.attn.out_proj,blocks.47.mlp.fc1,blocks.47.mlp.fc2,blocks.48.attn.qkv_proj,blocks.48.attn.out_proj,blocks.48.mlp.fc1,blocks.48.mlp.fc2,blocks.49.attn.qkv_proj,blocks.49.attn.out_proj,blocks.49.mlp.fc1,blocks.49.mlp.fc2" \
  --lora_rank 16 \
  --use_gradient_checkpointing_offload \
  --find_unused_parameters \
  --task "sft:train" \
  --save_steps 100 \
  --audio_loss_weight 0 \
  --preset_lora_path "./models/DiffSynth-Studio/MiniMax-H3-TrainingAdapter/model_ref2va.safetensors" \
  --preset_lora_model "dit"

Stage 2 从缓存目录读数据，所以不再需要 --dataset_metadata_path。

Note

1. 如何加 Training Adapter：在 Stage 2 的训练命令里加上 --preset_lora_path <training_adapter>.safetensors --preset_lora_model dit，训练框架会在训练开始前把 Training Adapter 融进 DiT。训练结束后只保留新训出的 LoRA，Training Adapter 不再需要，推理时也不要加载它。
2. 音频 loss：如果没有想要学习的音频目标，可以将 --audio_loss_weight 设置为 0，只优化视频分支，可以加快训练速度。
3. LoRA 注入位置：--lora_target_modules 只覆盖 blocks.25 ~ blocks.49（DiT 共 50 层）的后半段。这么设计是为了加速训练——未被 LoRA 覆盖的前半段模块不参与反向传播，效果也不错。
4. 训练显存：如果显存不够，可以尝试开启 DeepSpeed 等方式降低训练显存，参考 Low-VRAM Training。

效果示例

生成结果，统一参数：768×1344、124 帧 @ 24 fps、num_inference_steps=30、seed=0、cfg_scale=1.0。每行左为输入线稿，右为生成结果。

相关链接

• 基座模型：MiniMax/MiniMax-H3
• Training Adapter：DiffSynth-Studio/MiniMax-H3-TrainingAdapter
• 训练框架：DiffSynth-Studio