Hugging Peers

Qwen-Image-i2L

☆ Save
by DiffSynth-Studio text-to-image-synthesis Pytorch Apache License 2.0
32.5K downloads 114 likes on ModelScope updated

Model card

Qwen-Image-i2L (Image to LoRA)

模型介绍

i2L (Image to LoRA) 模型是我们以疯狂的思路设计的模型结构。模型的输入为一张图片，输出为这张图片训练出的 LoRA 模型。 更多关于训练策略和实现细节，欢迎查看我们的技术博客。

本次开源的模型包括四个：

  • Qwen-Image-i2L-Style
    • 简介：这是我们第一个称得上训练成功的模型，它的细节保持能力非常弱，但这反而让它能够有效地提取出图片中的风格信息，因此这个模型可用于风格迁移。
    • 图像编码器：SigLIP2、DINOv3
    • 参数量：2.4B
  • Qwen-Image-i2L-Coarse
    • 简介：这个模型是 Qwen-Image-i2L-Style 的 scaling up 版本。它产生的 LoRA 已经能够保留图片中的内容信息，但细节并不完善。如果将这个模型用于风格迁移，则需要输入更多图片，否则模型将倾向于生成输入图片中的内容。我们不建议单独使用这个模型。
    • 图像编码器：SigLIP2、DINOv3、Qwen-VL（分辨率为 224 x 224）
    • 参数量：7.9B
  • Qwen-Image-i2L-Fine
    • 简介：这个模型是 Qwen-Image-i2L-Coarse 的增量更新版本，必须与 Qwen-Image-i2L-Coarse 同时使用。它将 Qwen-VL 的图像编码分辨率提高到了 1024 x 1024，从而获得更多细节信息。
    • 图像编码器：SigLIP2、DINOv3、Qwen-VL（分辨率为 1024 x 1024）
    • 参数量：7.6B
  • Qwen-Image-i2L-Bias
    • 简介：这个模型是静态的补充 LoRA。由于 Qwen-Image-i2L-Coarse 与 Qwen-Image-i2L-Fine 的训练数据与 Qwen-Image 基础模型的训练数据分布不同，因此其生成的 LoRA 生成的图片与 Qwen-Image 的偏好对齐不一致。使用此 LoRA 模型后，生成的图片会更接近 Qwen-Image 的风格。
    • 图像编码器：无
    • 参数量：30M

这些模型仍有诸多局限，在泛化能力、细节保持能力等方面还有很大改进空间，我们将这些模型开源，以启发更多创新性的研究工作。

效果展示

Style

Qwen-Image-i2L-Style 模型可用于快速生成风格 LoRA，只需输入几张风格统一的图像。以下是我们生成的结果，随机种子都是 0。

风格1：抽象矢量

输入图像：

生成图像：

a cat a dog a girl

风格2：黑白素描

输入图像：

生成图像：

a cat a dog a girl

风格3：潦草手绘

输入图像：

生成图像：

a cat a dog a girl

风格4：蓝底扁平

输入图像：

生成图像：

a cat a dog a girl

Coarse + Fine + Bias

Qwen-Image-i2L-Coarse、Qwen-Image-i2L-Fine、Qwen-Image-i2L-Bias 三者组合后可以生成保留图像内容和细节信息的 LoRA 权重，这组权重作为 LoRA 训练的初始化权重能够加速收敛速度。

LoRA 数据集：小狗背包

训练数据：

训练中的样图生成：

训练步数：100 训练步数：200 训练步数：300 训练步数：400 训练步数：500
随机初始化
Image to LoRA 初始化

LoRA 数据集：小熊玩偶

训练数据：

训练中的样图生成：

训练步数：100 训练步数：200 训练步数：300 训练步数：400 训练步数：500
随机初始化
Image to LoRA 初始化

LoRA 数据集：蓝莓小碗

训练数据：

训练中的样图生成：

训练步数：100 训练步数：200 训练步数：300 训练步数：400 训练步数：500
随机初始化
Image to LoRA 初始化

推理代码

安装 DiffSynth-Studio：

git clone https://github.com/modelscope/DiffSynth-Studio.git  
cd DiffSynth-Studio
pip install -e .

Qwen-Image-i2L-Style

from diffsynth.pipelines.qwen_image import (
    QwenImagePipeline, ModelConfig,
    QwenImageUnit_Image2LoRAEncode, QwenImageUnit_Image2LoRADecode
)
from modelscope import snapshot_download
from safetensors.torch import save_file
import torch
from PIL import Image

vram_config_disk_offload = {
    "offload_dtype": "disk",
    "offload_device": "disk",
    "onload_dtype": "disk",
    "onload_device": "disk",
    "preparing_dtype": torch.bfloat16,
    "preparing_device": "cuda",
    "computation_dtype": torch.bfloat16,
    "computation_device": "cuda",
}

# Load models
pipe = QwenImagePipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[
        ModelConfig(model_id="DiffSynth-Studio/General-Image-Encoders", origin_file_pattern="SigLIP2-G384/model.safetensors", **vram_config_disk_offload),
        ModelConfig(model_id="DiffSynth-Studio/General-Image-Encoders", origin_file_pattern="DINOv3-7B/model.safetensors", **vram_config_disk_offload),
        ModelConfig(model_id="DiffSynth-Studio/Qwen-Image-i2L", origin_file_pattern="Qwen-Image-i2L-Style.safetensors", **vram_config_disk_offload),
    ],
    processor_config=ModelConfig(model_id="Qwen/Qwen-Image-Edit", origin_file_pattern="processor/"),
    vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 0.5,
)

# Load images
snapshot_download(
    model_id="DiffSynth-Studio/Qwen-Image-i2L",
    allow_file_pattern="assets/style/1/*",
    local_dir="data/examples"
)
images = [
    Image.open("data/examples/assets/style/1/0.jpg"),
    Image.open("data/examples/assets/style/1/1.jpg"),
    Image.open("data/examples/assets/style/1/2.jpg"),
    Image.open("data/examples/assets/style/1/3.jpg"),
    Image.open("data/examples/assets/style/1/4.jpg"),
]

# Model inference
with torch.no_grad():
    embs = QwenImageUnit_Image2LoRAEncode().process(pipe, image2lora_images=images)
    lora = QwenImageUnit_Image2LoRADecode().process(pipe, **embs)["lora"]
save_file(lora, "model_style.safetensors")

Qwen-Image-i2L-Coarse、Qwen-Image-i2L-Fine、Qwen-Image-i2L-Bias

from diffsynth.pipelines.qwen_image import (
    QwenImagePipeline, ModelConfig,
    QwenImageUnit_Image2LoRAEncode, QwenImageUnit_Image2LoRADecode
)
from diffsynth.utils.lora import merge_lora
from diffsynth import load_state_dict
from modelscope import snapshot_download
from safetensors.torch import save_file
import torch
from PIL import Image

vram_config_disk_offload = {
    "offload_dtype": "disk",
    "offload_device": "disk",
    "onload_dtype": "disk",
    "onload_device": "disk",
    "preparing_dtype": torch.bfloat16,
    "preparing_device": "cuda",
    "computation_dtype": torch.bfloat16,
    "computation_device": "cuda",
}

# Load models
pipe = QwenImagePipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[
        ModelConfig(model_id="Qwen/Qwen-Image", origin_file_pattern="text_encoder/model*.safetensors", **vram_config_disk_offload),
        ModelConfig(model_id="DiffSynth-Studio/General-Image-Encoders", origin_file_pattern="SigLIP2-G384/model.safetensors", **vram_config_disk_offload),
        ModelConfig(model_id="DiffSynth-Studio/General-Image-Encoders", origin_file_pattern="DINOv3-7B/model.safetensors", **vram_config_disk_offload),
        ModelConfig(model_id="DiffSynth-Studio/Qwen-Image-i2L", origin_file_pattern="Qwen-Image-i2L-Coarse.safetensors", **vram_config_disk_offload),
        ModelConfig(model_id="DiffSynth-Studio/Qwen-Image-i2L", origin_file_pattern="Qwen-Image-i2L-Fine.safetensors", **vram_config_disk_offload),
    ],
    processor_config=ModelConfig(model_id="Qwen/Qwen-Image-Edit", origin_file_pattern="processor/"),
    vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 0.5,
)

# Load images
snapshot_download(
    model_id="DiffSynth-Studio/Qwen-Image-i2L",
    allow_file_pattern="assets/lora/3/*",
    local_dir="data/examples"
)
images = [
    Image.open("data/examples/assets/lora/3/0.jpg"),
    Image.open("data/examples/assets/lora/3/1.jpg"),
    Image.open("data/examples/assets/lora/3/2.jpg"),
    Image.open("data/examples/assets/lora/3/3.jpg"),
    Image.open("data/examples/assets/lora/3/4.jpg"),
    Image.open("data/examples/assets/lora/3/5.jpg"),
]

# Model inference
with torch.no_grad():
    embs = QwenImageUnit_Image2LoRAEncode().process(pipe, image2lora_images=images)
    lora = QwenImageUnit_Image2LoRADecode().process(pipe, **embs)["lora"]
    lora_bias = ModelConfig(model_id="DiffSynth-Studio/Qwen-Image-i2L", origin_file_pattern="Qwen-Image-i2L-Bias.safetensors")
    lora_bias.download_if_necessary()
    lora_bias = load_state_dict(lora_bias.path, torch_dtype=torch.bfloat16, device="cuda")
    lora = merge_lora([lora, lora_bias])
save_file(lora, "model_coarse_fine_bias.safetensors")

使用生成的 LoRA 生成图像

from diffsynth.pipelines.qwen_image import QwenImagePipeline, ModelConfig
import torch

vram_config = {
    "offload_dtype": "disk",
    "offload_device": "disk",
    "onload_dtype": torch.bfloat16,
    "onload_device": "cpu",
    "preparing_dtype": torch.bfloat16,
    "preparing_device": "cuda",
    "computation_dtype": torch.bfloat16,
    "computation_device": "cuda",
}
pipe = QwenImagePipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[
        ModelConfig(model_id="Qwen/Qwen-Image", origin_file_pattern="transformer/diffusion_pytorch_model*.safetensors", **vram_config),
        ModelConfig(model_id="Qwen/Qwen-Image", origin_file_pattern="text_encoder/model*.safetensors", **vram_config),
        ModelConfig(model_id="Qwen/Qwen-Image", origin_file_pattern="vae/diffusion_pytorch_model.safetensors", **vram_config),
    ],
    tokenizer_config=ModelConfig(model_id="Qwen/Qwen-Image", origin_file_pattern="tokenizer/"),
    vram_limit=torch.cuda.mem_get_info("cuda")[1] / (1024 ** 3) - 0.5,
)
pipe.load_lora(pipe.dit, "model_style.safetensors")
image = pipe("a cat", seed=0, height=1024, width=1024, num_inference_steps=50)
image.save("image.jpg")