简介
本模型是一个面向 FLUX.1-dev 的多目标对齐 LoRA，主要优化 GenEval、OCR 和 PickScore 三项能力，基于改进版 Flow-GRPO 训练得到。
在保持生成质量的同时，模型在复杂文本-图像对齐、文字渲染和人类偏好相关指标上取得了较好的综合表现。
视觉对比
Prompt 1: A moonlit Venetian canal scene with a gondola floating on the left, a candlelit table for two on a stone terrace to the right, and flower boxes under arched windows in the peach-colored building behind, with shimmering reflections in dark water; romantic cinematic mood and the restaurant sign reading "LUNA SERENA"
| FLUX.1-dev | Ours |
|---|---|
 |
 |
Prompt 2: A beautiful Mediterranean kitchen with white plaster walls, a blue ceramic bowl of oranges on a stone island, a copper kettle behind the bowl, a folded olive-green towel on the right edge, and a sunlit window with herbs in terracotta pots along the sill; warm natural light, tasteful rustic elegance, and the towel embroidered with "CASA LUNA"
| FLUX.1-dev | Ours |
|---|---|
 |
 |
Prompt 3: A glowing winter Christmas market square, with a decorated evergreen tree in the center, a stall of gingerbread on the left, a carousel on the right, and strings of warm lights above snowy cobblestones; festive cinematic charm and the stall sign reading "HOLIDAY TREATS"
| FLUX.1-dev | Ours |
|---|---|
 |
 |
Prompt 4: A dreamy alpine breakfast by a panoramic window, with a wooden tray on a bed holding a croissant, a pot of coffee, and a bowl of apricots, while a cream blanket folds across the lower edge and snowy peaks fill the view beyond; soft luxury travel styling, bright crisp light, and a small jar labeled "MOUNTAIN HONEY"
| FLUX.1-dev | Ours |
|---|---|
 |
 |
定量分析
我们将该模型与其他多目标对齐方法进行了对比。结果表明，本模型在 GenEval、OCR 和 PickScore 等指标上表现突出，具有较强的综合对齐能力。
| Method | Single object | Two. | Counting | Colors | Position | Color-Attr. | GenEval-Overall | OCR | PickScore |
|---|---|---|---|---|---|---|---|---|---|
| FLUX.1-dev | 0.988 | 0.748 | 0.713 | 0.745 | 0.200 | 0.470 | 0.644 | 0.591 | 21.993 |
| Diffusion-Blend | 1.000 | 0.899 | 0.738 | 0.819 | 0.380 | 0.680 | 0.753 | 0.732 | 23.083 |
| SELMA | 1.000 | 0.950 | 0.838 | 0.800 | 0.530 | 0.680 | 0.799 | 0.879 | 22.947 |
| MapReduce-LoRA | 1.000 | 0.980 | 0.875 | 0.915 | 0.820 | 0.670 | 0.877 | 0.951 | 23.015 |
| Flow-OPD | 1.000 | 0.970 | 0.913 | 0.862 | 0.810 | 0.700 | 0.876 | 0.951 | 23.454 |
| Ours | 1.000 | 0.939 | 0.925 | 0.883 | 0.930 | 0.700 | 0.896 | 0.962 | 23.576 |
推理代码
git clone https://github.com/modelscope/DiffSynth-Studio.git
cd DiffSynth-Studio
pip install -e .
- 推理示例
import torch
from diffsynth.pipelines.flux_image import FluxImagePipeline, ModelConfig
pipe = FluxImagePipeline.from_pretrained(
torch_dtype=torch.bfloat16,
device="cuda",
model_configs=[
ModelConfig(model_id="black-forest-labs/FLUX.1-dev", origin_file_pattern="flux1-dev.safetensors"),
ModelConfig(model_id="black-forest-labs/FLUX.1-dev", origin_file_pattern="text_encoder/model.safetensors"),
ModelConfig(model_id="black-forest-labs/FLUX.1-dev", origin_file_pattern="text_encoder_2/*.safetensors"),
ModelConfig(model_id="black-forest-labs/FLUX.1-dev", origin_file_pattern="ae.safetensors"),
],
)
pipe.load_timestep_lora(ModelConfig(model_id="DiffSynth-Studio/MultiAlign-FLUX.1-dev", origin_file_pattern="adapter_model.safetensors"))
prompt = "A moonlit Venetian canal scene with a gondola floating on the left, a candlelit table for two on a stone terrace to the right, and flower boxes under arched windows in the peach-colored building behind, with shimmering reflections in dark water; romantic cinematic mood and the restaurant sign reading 'NOTTE SERENA'"
image = pipe(prompt=prompt, seed=0, num_inference_steps=30, t5_sequence_length=128)
image.save("FLUX.1-dev-Timestep-LoRA.jpg")