[Community Pipeline] UnCLIP image / text interpolations

[patrickvonplaten]

Model/Pipeline/Scheduler description

Copied from #1858:

UnCLIP / Karlo: https://huggingface.co/spaces/kakaobrain/karlo gives some very nice and precise results when doing image generation and can strongly outperform Stable Diffusion in some - see:
https://www.reddit.com/r/StableDiffusion/comments/zshufz/karlo_the_first_large_scale_open_source_dalle_2/

Another extremely interesting aspect of Dalle 2 is its ability to interpolate between text and or image embeddings. See e.g. section 3.) of the Dalle 2 paper: https://cdn.openai.com/papers/dall-e-2.pdf . This PR now allows to directly pass text embeddings and image embeddings which should enable those tasks!

I think we could create a super cool community pipeline. The pipeline could allow to automatically create interpolations between two text prompts and similarly we could create one to do interpolations between two images.

In terms of design to stay as efficient as possible the following would make sense:

• 1. The user passes two text prompts and a num_interpolations input.
• 2. The pipeline then embeds those two text prompts into the text embeddings x_0 and x_N and num_interpolations x_1, x_2, ... x_N-1 are created using the slerp function .
• 3. Then we have num_interpolations + 2 text embeddings that should be passed in a batch through the model to create a nice interpolation of images.
• 4. It'd be important to make use of enable_cpu_offload() to save memory.

It's probably easier to start with the UnCLIPImageInterpolationPipeline since image embeddings are just a single 1-d vector where as for text embeddings two latent vectors are used.

Would be more than happy to help if someone is interested in giving this a try - think it'll make for some super cool demos.

Open source status

• The model implementation is available
• The model weights are available (Only relevant if addition is not a scheduler).

Provide useful links for the implementation

No response

[Abhinay1997]

Hi @patrickvonplaten and @williamberman, are you working on this or can I pick this up ?

[patrickvonplaten]

@Abhinay1997 feel free to pick it up! We re more than happy to help if needed ☺️

[Abhinay1997]

This makes sense at the top level. But just so that my understanding is correct, we want a community pipeline that can interpolate between prompts/images like the StableDiffusionInterpolation but using the unCLIPPipeline ( a.k.a Dall-E 2)

The interpolation also makes sense, we generate the embeddings for the 2 prompts, p1 and p2 let's say. They would correspond to x_0 and x_N of the interpolation sequence and using slerp I would interpolate between them for N outputs in total for a pair of prompts/images.

[patrickvonplaten]

That's exactly right @Abhinay1997 :-)

[Abhinay1997]

@patrickvonplaten sorry it took so long.

The UnCLIPTextInterpolation pipeline is actually more straight forward imo. For the UnCLIPImageInterpolation pipeline, would we not need the CLIP model that was used for training the UnCLIPPipeline ?

Because I see the CLIP model in the original codebase but not in the HuggingFace hub model

I am working on the text interpolation right now. ETA: 24th Jan.

[patrickvonplaten]

cc @williamberman here

[williamberman]

Thanks for your work @Abhinay1997!

We do still use clip for the unclip pipeline(s). Note that we just import it from transformers.

In the text to image pipeline, we use the text encoder for encoding the prompt

diffusers/src/diffusers/pipelines/unclip/pipeline_unclip.py

Lines 70 to 71 in ac3fc64

	text_encoder: CLIPTextModelWithProjection
	tokenizer: CLIPTokenizer

In the image variation pipeline, we use the image encoder for encoding the input image and the text encoder for encoding an empty prompt. Note that we also optionally allow directly passing image embedding to the pipeline to skip encoding an input image.

diffusers/src/diffusers/pipelines/unclip/pipeline_unclip_image_variation.py

Lines 75 to 78 in ac3fc64

	text_encoder: CLIPTextModelWithProjection
	tokenizer: CLIPTokenizer
	feature_extractor: CLIPFeatureExtractor
	image_encoder: CLIPVisionModelWithProjection

The image interpolation pipeline should work similarly to the image variation pipeline in that it should be passed either the two sets of images which would be encoded via clip or the two sets of pre-encoded latents. Note that similarly to the image variation pipeline, we would have to encode the empty text prompt for the image interpolation pipeline.

LMK if that makes sense!

[Abhinay1997]

Hi @williamberman thank you for the details. It makes sense. I was under the impression that I needed to use the actual CLIP checkpoint that the UnCLIP model learns to invert its decoder over. So I got confused.

Will share the work in progress notebooks soon.

[Abhinay1997]

Hi @williamberman can you review the UnCLIPTextInterpolation notebook when you have time ?

Questions:-

1. When prompts have different lengths, what attention mask should be used for the intermediate interpolation steps ?
2. Will I be able to instantiate this community pipeline using DiffusionPipeline(custom_pipelein='....') as I am inheriting from both DiffusionPipeline[ CommunityPipeline requirement] and UnCLIPPipeline[ to be able to use UnCLIP modules] ?

[williamberman]

Great work so far @Abhinay1997 !

Hi @williamberman can you review the UnCLIPTextInterpolation notebook when you have time ?

Questions:-

1. When prompts have different lengths, what attention mask should be used for the intermediate interpolation steps ?

That's a good point, and I don't know off the top of my head or from googling around. I would recommend for now just using the mask of the longer prompt. cc @patrickvonplaten

2. Will I be able to instantiate this community pipeline using DiffusionPipeline(custom_pipelein='....') as I am inheriting from both DiffusionPipeline[ CommunityPipeline requirement] and UnCLIPPipeline[ to be able to use UnCLIP modules] ?

We actually want all pipelines to be completely independent so please do not inherit from the UnCLIPPipeline :)

[Abhinay1997]

@williamberman, just to clarify, can I still import UnCLIPPipeline inside the methods and use it for generation ?

[williamberman]

Nope!

We want all pipelines to be as self contained as possible. If any methods are exactly the same, we have the # Copied from mechanism (which we should document a bit better) which will let you copy and paste the method and keep the two methods in sync in ci.

We've articulated some of our rationale in the philosophy doc

https://github.com/huggingface/diffusers/blob/main/docs/source/en/conceptual/philosophy.mdx#pipelines
https://github.com/huggingface/diffusers/blob/main/docs/source/en/conceptual/philosophy.mdx#tweakable-contributor-friendly-over-abstraction