Skip to content

Avoiding graph break by changing the way we infer dtype in vae.decoder #12512

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Avoiding graph break by changing the way we infer dtype in vae.decoder #12512

Changes from 2 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
c813353
Changing the way we infer dtype to avoid force evaluation of lazy ten…
ppadjinTT Oct 20, 2025
e84a341
changing way to infer dtype to ensure type consistency
ppadjinTT Oct 22, 2025
3a860ae
more robust infering of dtype
ppadjinTT Oct 23, 2025
0c04973
removing the upscale dtype entirely
ppadjinTT Oct 23, 2025
6cc4ae8
Merge branch 'main' into fix-vae-lazy-tensor-evaluation
ppadjinTT Oct 24, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion src/diffusers/models/autoencoders/vae.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -286,7 +286,7 @@ def forward(

sample = self.conv_in(sample)

upscale_dtype = next(iter(self.up_blocks.parameters())).dtype
upscale_dtype = self.up_blocks[0].resnets[0].norm1.weight.dtype
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Think current failing tests in the CI are due to the fact that not every decoder block has a norm1 with a weight. Hence the use of the generator here to avoid such cases.

@ppadjinTT I noticed you initially used self.conv_out.weight here? What was the issue you ran into with that?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay, I will change that too, tnx! I intially changed the self.conv_out.weight because there are some tests that check what happens when conv_out and upscale_blocks have different dtypes

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you point me to those tests? Seems like setting to conv_out is more robust.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup, these are the tests pytest -svvv tests/models/autoencoders/test_models_autoencoder_kl.py

This is one of the tests from this test set that fails tests/models/autoencoders/test_models_autoencoder_kl.py::AutoencoderKLTests::test_layerwise_casting_inference

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added better logic for inferring dtype, to capture the case where it doesn't work

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm I think we can remove upscale_type entirely here. I think all tests should still pass without it.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

okay let's try that, i'm pushing the change

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you think it's okay now? @DN6

if torch.is_grad_enabled() and self.gradient_checkpointing:
# middle
sample = self._gradient_checkpointing_func(self.mid_block, sample, latent_embeds)
Expand Down
Loading