Skip to content

Correct timestep for img2img initial noise addition#1946

Merged
lstein merged 4 commits intoinvoke-ai:mainfrom
wfng92:patch-3
Dec 15, 2022
Merged

Correct timestep for img2img initial noise addition#1946
lstein merged 4 commits intoinvoke-ai:mainfrom
wfng92:patch-3

Conversation

@wfng92
Copy link
Copy Markdown
Contributor

@wfng92 wfng92 commented Dec 12, 2022

There seems to be difference between huggingface's diffusers and compvis's implementation for img2img as the wrong timestep is used for the additive noise for image-2-image.

Summary from the relevant PR raised by the HuggingFace team:

t+1 is used as the timestep to add noise to the original image, but [0, ..., t] is used afterwards for the denoising process. We should however also use t when adding the noise to the original image.
This can be quite easily verified by doing the following. Run a img2img with a small number of update steps and a very low strength because then differences between t and t+1 become quite clear.

From

torch.tensor([t_enc]).to(self.model.device)

to

torch.tensor([t_enc - 1]).to(self.model.device)

For example, using the following init_img and configuration:

sketch-mountains-input

  • prompt: A fantasy landscape, trending on artstation
  • strength: 0.1
  • seed: 42
  • sampler_name: DDIM
  • steps: 10

The existing code will generate an image with the following noise:

old_code

The new implementation will produce an image with lesser noise:

new_code

This PR affects only the img2img script. I realized that text2img2img and inpainting share the same code base but I am not sure if the same implementation should be applied as well.

@lstein
Copy link
Copy Markdown
Collaborator

lstein commented Dec 12, 2022

Thanks so much for finding this problem. I will test it out the proposed solution later today.

Copy link
Copy Markdown
Collaborator

@lstein lstein left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for finding and fixing this. I have applied your fix to the inpaint and txt2img2img paths as well.

@lstein lstein enabled auto-merge (rebase) December 13, 2022 19:32
@lstein lstein disabled auto-merge December 15, 2022 20:59
@lstein lstein merged commit 2a2f38a into invoke-ai:main Dec 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants