diffusionpen-matan-ft
DiffusionPen (Nikolaidou et al., ECCV 2024) fine-tuned on Hebrew handwritten words. It produces one word per image (64×256 px) in the style of a given writer. The style is set by 5 reference word crops.
- Base: warm-started from
cyttic/diffusionpen-matan, the same architecture fine-tuned further on the same data. - Training data:
cyttic/matan-hebrew-handwriting-words, 22,148 word crops by 491 writers. - Architecture: UNet with 491 style classes and a 29-symbol vocabulary (
character_classes.json). Text is encoded with CANINE-c. It works in the latent space of the SD-1.5 VAE. Style comes from the DiffusionPen MobileNetV2 style encoder. - Characters: the 27 Hebrew letters, including final forms. No punctuation, digits or Latin.
Files
| file | what |
|---|---|
ema_ckpt.pt |
UNet EMA weights (use this one for sampling) |
ckpt.pt, optim.pt |
raw weights and optimizer state (only needed to resume training) |
sample.py |
minimal runnable sampler |
unet.py, feature_extractor.py |
model code from the DiffusionPen repo (unchanged, MIT, see LICENSE-DiffusionPen) |
style_models/iam_style_diffusionpen.pth |
DiffusionPen style encoder (original release) |
character_classes.json, writers_dict_train.json |
vocabulary and writer-id → style-class maps |
Quick start
pip install -r requirements.txt # torch, diffusers, transformers, timm, einops, ...
python sample.py --text "שלום עולם" --style w100 --n 2
python sample.py --text "שלום" --refs r1.png r2.png r3.png r4.png r5.png # your own style references
On the first run, sample.py downloads everything it needs from the Hub:
- this repo;
- the reference crops (from the dataset above);
- the SD-1.5 VAE and DDIM scheduler (
stable-diffusion-v1-5/stable-diffusion-v1-5); google/canine-c.
It runs on GPU, or on CPU (slowly). Output goes to ./samples/. Each word is generated separately; to build a line, concatenate the words right to left.
Examples (examples/, writer w100, 30 DDIM steps):
