Multidiffusion creates corrupted images with different schedulers

[Isotr0py]

Thanks for the great implement of StableDiffusionPanoramaPipeline.
However, I found that StableDiffusionPanoramaPipeline will create corrupted images when using some specific schedulers.

For example, if we use DPMSolverMultistepScheduler, a corrupted image will be generated.

import torch
from diffusers import StableDiffusionPanoramaPipeline
from diffusers import DPMSolverMultistepScheduler

seed = 33
model_ckpt = "stabilityai/stable-diffusion-2-base"
prompt = "a photo of the dolomites"

pipe = StableDiffusionPanoramaPipeline.from_pretrained(model_ckpt, torch_dtype=torch.float16).to("cuda")
pipe.scheduler = DPMSolverMultistepScheduler.from_config(pipe.scheduler.config)
pipe.to("cuda")
generator = torch.Generator(device="cuda").manual_seed(seed)
image = pipe(prompt, generator=generator, num_inference_steps=25,width=1024,height=512).images[0]
image.save("DPM++2M.png")
display(image)

But if we switch to EulerDiscreteScheduler, it can create a normal panorama image like DDIM scheduler.

import torch
from diffusers import StableDiffusionPanoramaPipeline
from diffusers import EulerDiscreteScheduler

seed = 33
model_ckpt = "stabilityai/stable-diffusion-2-base"
prompt = "a photo of the dolomites"

pipe = StableDiffusionPanoramaPipeline.from_pretrained(model_ckpt, torch_dtype=torch.float16).to("cuda")
pipe.scheduler = EulerDiscreteScheduler.from_config(pipe.scheduler.config)
pipe.to("cuda")
generator = torch.Generator(device="cuda").manual_seed(seed)
image = pipe(prompt, generator=generator, num_inference_steps=25,width=1024,height=512).images[0]
image.save("Euler.png")
display(image)

[Isotr0py]

Well, I think I found what's wrong with this. I will create a PR to solve it later.

[alexblattner]

@Isotr0py what was the problem?

[Isotr0py]

@alexblattner We crop latents into blocks and sample them seperately in PanoramaPipeline.

However, if we use dpmsolver scheduler, which used self.model_outputs to save model_outputs in previous and current timestep, the self.model_outputs will become model_outputs in previous and current block in PanoramaPipeline instead.

Since the self.model_outputs mismatched, panorama image was corrupted. So we could rematch self.model_outputs in blocks sampling loop to solve it.

[patrickvonplaten]

cc @sayakpaul for panorama pipeline

[sayakpaul]

#3499 should fix it. Thanks so much @Isotr0py.

Closing hence.