Improve pipeline_stable_diffusion_inpaint_legacy.py

[cyber-meow]

This is basically a one-line change to the file pipeline_stable_diffusion_inpaint_legacy.py.
I allow the possibility of replacing
init_latents_proper = self.scheduler.add_noise(init_latents_orig, noise, torch.tensor([t]))
by
init_latents_proper = self.scheduler.add_noise(init_latents_orig, noise_pred_uncond, torch.tensor([t]))
with the argument add_predicted_noise now default to True (i.e., use the latter option by default).
(Maybe it is better to keep the default argument as False, and maybe we can find a better name for this argument.)

What I am doing here is that I use the unconditionally predicted noise instead of the initial noise to create the diffused samples used in the reverse diffusion process, and as argued in this paper this leads to more coherent result.
It turns out however that I did fail two of the three tests when I run tests/pipelines/stable_diffusion/test_stable_diffusion_inpaint_legacy.py. These are assertions errors AssertionError: assert 0.012784755849838236 < 0.01 and AssertionError: assert 0.01317704176902773 < 0.01.
I doubt this is a natural consequence of the change of the algorithm and as the difference is small I don't think that is a severe problem.

Anyway, the difference in inpainted result is very pronouncing.
In the following, I compare results from Automatic1111 webui DDIM inpainting (top row), the original inpaint_legacy script (middle row), and the modified inpaint_legacy script (bottom row). For the latter two I use the same random seeds.
Denoising strength is set to 1 so that the masked area are completely ignored

Example 1
model: SD 1.4
cfg: 10
Input:

Prompt: guinea pig

Example 2
model: Linaqruf/anything-v3.0
cfg: 7.5
Input:

Prompt: anime girl with blue hair in dress

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[averad]

I tried to apply your suggested settings to the Onnx Legacy Inpainting Pipeline but the noise_pred_uncond isn't defined until after the noise is generated.

UnboundLocalError: local variable 'noise_pred_uncond' referenced before assignment

Any thoughts on adding this to Onnx Legacy Inpainting Pipeline as well?

pipeline_onnx_stable_diffusion_inpaint_legacy.py

Line 238

add_predicted_noise: Optional[bool] = True,

Line 270

add_predicted_noise (`bool`, *optional*, defaults to True):
    Use predicted noise instead of random noise when constructing noisy versions of the original image in
    the reverse diffusion process

Line 378

# add noise to latents using the timesteps
noise = generator.randn(*init_latents.shape).astype(latents_dtype)
if add_predicted_noise:
	init_latents = self.scheduler.add_noise(
	torch.from_numpy(init_latents), torch.from_numpy(noise_pred_uncond), torch.from_numpy(timesteps)
	)
else:
	init_latents = self.scheduler.add_noise(
	torch.from_numpy(init_latents), torch.from_numpy(noise), torch.from_numpy(timesteps)
	)
init_latents = init_latents.numpy()

[cyber-meow]

The line to change is 417-419 when defining init_latents_proper, i.e. the recurrent step and not the initialization step.

[cyber-meow]

I also compare with the inpainting model using Automatic1111-webui (second row). For the guinea pig example the inpainting model works the best as it perfectly falls within the training distribution.

For the second one I do a custom merge and it works reasonably well as well. However inpaint_legacy works for any model trained without inpainting in mind and can be beneficial for more customized dreambooth model.

[averad]

Code change is similar to PR #1583 - Linking so both are reviewed at the same time.

[patrickvonplaten]

Thanks for the PR!