scepter_scedit

🪄SCEdit: Efficient and Controllable Image Diffusion Generation via Skip Connection Editing

Zeyinzi Jiang · Chaojie Mao · Yulin Pan · Zhen Han · Jingfeng Zhang


Alibaba Group

SCEdit is an efficient generative fine-tuning framework proposed by Alibaba TongYi Vision Intelligence Lab. This framework enhances the fine-tuning capabilities for text-to-image generation downstream tasks and enables quick adaptation to specific generative scenarios, saving 30%-50% of training memory costs compared to LoRA. Furthermore, it can be directly extended to controllable image generation tasks, requiring only 7.9% of the parameters that ControlNet needs for conditional generation and saving 30% of memory usage. It supports various conditional generation tasks including edge maps, depth maps, segmentation maps, poses, color maps, and image completion.

Code Example

git clone https://github.com/modelscope/scepter.git
cd scepter
PYTHONPATH=. python scepter/tools/run_train.py --cfg scepter/methods/SCEdit/t2i_sdxl_1024_sce.yaml

To prepare the training dataset.

# pip install modelscope
from modelscope.msdatasets import MsDataset
ms_train_dataset = MsDataset.load('style_custom_dataset', namespace='damo', subset_name='3D', split='train_short')
print(next(iter(ms_train_dataset)))

BibTeX

@article{jiang2023scedit,
    title = {SCEdit: Efficient and Controllable Image Diffusion Generation via Skip Connection Editing},
    author = {Jiang, Zeyinzi and Mao, Chaojie and Pan, Yulin and Han, Zhen and Zhang, Jingfeng},
    year = {2023},
    journal = {arXiv preprint arXiv:2312.11392}  
}