Hugging Peers

dreamsim-ensemble

☆ Save
by bigshanedogg image-feature-extraction transformers mit
27 downloads 1 likes on Hugging Face updated

Model card

DreamSim (ensemble) — HuggingFace format

Unofficial HuggingFace-format port of the DreamSim perceptual-similarity metric (ensemble variant). Loads via trust_remote_code with no dreamsim / peft runtime dependency; weights are the LoRA-merged ensemble stored as model.safetensors.

Not an official DreamSim release. DreamSim was created by Shobhita Sundaram, Netanel Tamir, Stephanie Fu, and Richard Zhang — see the original repository https://github.com/ssundaram21/dreamsim and paper "DreamSim: Learning New Dimensions of Human Visual Similarity using Synthetic Data" (NeurIPS 2023). This repo re-packages their released ensemble for transformers.

What it is

The ensemble concatenates the features of three ViT-B/16 backbones — DINO (cls), CLIP (embedding), OpenCLIP (embedding) — each LoRA-adapted, then mean/L2-normalizes (embed_size = 1792). Perceptual distance between two images is 1 − cos.

This port has been verified against the official dreamsim package on 1,000 real images: max |Δ embedding| < 3e-7 and max |Δ distance| < 2e-7 — identical up to float32 rounding.

Usage

import torch
from PIL import Image
from transformers import AutoModel, AutoImageProcessor

model = AutoModel.from_pretrained("bigshanedogg/dreamsim", trust_remote_code=True).eval()
proc = AutoImageProcessor.from_pretrained("bigshanedogg/dreamsim", trust_remote_code=True)

img_a = proc(images=Image.open("a.png"), return_tensors="pt")["pixel_values"]
img_b = proc(images=Image.open("b.png"), return_tensors="pt")["pixel_values"]

emb = model(pixel_values=img_a).embeddings          # (1, 1792)
dist = model.compute_distance(img_a, img_b)          # 1 - cos (higher = more different)

The image processor resizes to 224×224 (BICUBIC) and scales to [0, 1]; the per-backbone mean/std normalization is applied inside the model.

Licensing

  • Port + original DreamSim code/weights: MIT — see LICENSE.
  • Vendored ViT backbone code (adapted from DINO, Meta AI): Apache-2.0 — full text in LICENSE.apache-2.0.txt; attribution + modification statement in NOTICE.
  • Merged backbone weights derive from DINO (Apache-2.0), CLIP (MIT), and OpenCLIP (MIT code, LAION-based weights). Review the upstream backbone terms — and, for OpenCLIP, the LAION data provenance — before any commercial use.