Inpainting model

[benedlore]

Is there a new inpainting model released for researchers or is it still currently the original latent diffusion model that is current most released?

[gregturk]

So far as I know, inpainting is not a capability that is specific to any particular trained model (e.g. set of network weights). Rather, at the heart of inpainting it is a piece of code that "freezes" one part of the image as it is being generated. There is actually code to do inpainting in the "scripts" directory ("inpaint.py"). I looked it over briefly, and it looks like you just have to supply a mask, which is a PNG file. The puzzling thing about this script is that it takes very few parameters. Maybe they just have hard-wired in some reasonable defaults.

My guess is that somebody could cobble together an inpainting example in a colab notebook without too much trouble. If somebody does this, please let the rest of us know!

[benedlore]

Ah, so instead of a separate model, it would just be the new SD model itself being used for the inpainting as well? I was under the impression it was a separate model.

[1blackbar]

No dood, its already in there , you just need weights but theres no GUI for it

[karray]

where can I find the weights?

[scyheidekamp]

When trying to run the inpainting script, I'm missing a file called last.ckpt
Is this already available somewhere? Because placing the sd-v1-4.ckpt there doesn't seem to work

[karray]

I found the weights in a nearby repository:

wget -O models/ldm/inpainting_big/last.ckpt https://heibox.uni-heidelberg.de/f/4d9ac7ea40c64582b7c9/?dl=1

[banteg]

are there any specific requirements for the inpainting model input? the result looked garbled up for me, as if the dimensions were improperly translated.

[Sanster]

No dood, its already in there , you just need weights but theres no GUI for it

You can try Lama Cleaner, it integrates multiple inpainting models, including LDM.

[karray]

this link doesnt seem to be working anymore, anybody got an updated link?

There is also a download script:

wget -O models/ldm/inpainting_big/model.zip https://ommer-lab.com/files/latent-diffusion/inpainting_big.zip

but I didn't try it.

[javismiles]

does anybody have a version of this inpainting script that involves also a text prompt so that the masked parts of the image are pushed in the direction of the text prompt, and only those areas, but taking into account the surroundings etc, thank you

[benedlore]

So hold on, someone said we could use the SD model, but then that didn't work, so people started downloading the old LDM model for inpainting?

[Jellybit]

So hold on, someone said we could use the SD model, but then that didn't work, so people started downloading the old LDM model for inpainting?

I found something that claims to be using Stable Diffusion. Here's the walkthrough video:
https://www.youtube.com/watch?v=N913hReVxMM

And here's the colab notebook:
https://colab.research.google.com/drive/1R2HJvufacjy7GNrGCwgSE3LbQBk5qcS3?usp=sharing

[karray]

So hold on, someone said we could use the SD model, but then that didn't work, so people started downloading the old LDM model for inpainting?

SD is based on LDM. I guess the inpainting is a legacy example from that project