Skip to content

[Qwen LoRA training] fix bug when offloading #12440

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Oct 7, 2025
+1 −1
Merged

[Qwen LoRA training] fix bug when offloading #12440

Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
3f665ac
fix bug when offload and cache_latents both enabled
linoytsaban Oct 6, 2025
e5c98cd
fix bug when offload and cache_latents both enabled
linoytsaban Oct 6, 2025
248a763
fix bug when offload and cache_latents both enabled
linoytsaban Oct 6, 2025
97dac51
fix bug when offload and cache_latents both enabled
linoytsaban Oct 6, 2025
5da70fc
fix bug when offload and cache_latents both enabled
linoytsaban Oct 7, 2025
302a273
Merge branch 'main' into qwen-lora-fix
linoytsaban Oct 7, 2025
01498ad
fix bug when offload and cache_latents both enabled
linoytsaban Oct 7, 2025
4a238de
Merge remote-tracking branch 'origin/qwen-lora-fix' into qwen-lora-fix
linoytsaban Oct 7, 2025
73453e9
fix bug when offload and cache_latents both enabled
linoytsaban Oct 7, 2025
050a2c8
fix bug when offload and cache_latents both enabled
linoytsaban Oct 7, 2025
b1d5489
fix bug when offload and cache_latents both enabled
linoytsaban Oct 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion examples/dreambooth/train_dreambooth_lora_qwen_image.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1338,7 +1338,7 @@ def compute_text_embeddings(prompt, text_encoding_pipeline):
batch["pixel_values"] = batch["pixel_values"].to(
accelerator.device, non_blocking=True, dtype=vae.dtype
)
latents_cache.append(vae.encode(batch["pixel_values"]).latent_dist)
latents_cache.append(vae.encode(batch["pixel_values"]).latent_dist)
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so that vae & batch["pixel_values"] are both on the same device. w/o this change the vae is back on cpu but batch["pixel_values"] are on accelerator.device

if train_dataset.custom_instance_prompts:
with offload_models(text_encoding_pipeline, device=accelerator.device, offload=args.offload):
prompt_embeds, prompt_embeds_mask = compute_text_embeddings(
Expand Down