forked from huggingface/diffusers
-
Couldn't load subscription status.
- Fork 0
Mirage Pipeline #1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
DavidBert
wants to merge
70
commits into
main
Choose a base branch
from
mirage
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 1 commit
Commits
Show all changes
70 commits
Select commit
Hold shift + click to select a range
ec5449f
Support both huggingface_hub `v0.x` and `v1.x` (#12389)
Wauplin 016316a
mirage pipeline first commit
4ac274b
use attention processors
904debc
use diffusers rmsnorm
122115a
use diffusers timestep embedding method
4588bbe
[CI] disable installing transformers from main in ci for now. (#12397)
sayakpaul e3fe0e8
remove MirageParams
85ae87b
checkpoint conversion script
9a697d0
ruff formating
9c09445
[docs] slight edits to the attention backends docs. (#12394)
sayakpaul 041501a
[docs] remove docstrings from repeated methods in `lora_pipeline.py` …
sayakpaul 19085ac
Don't skip Qwen model tests for group offloading with disk (#12382)
sayakpaul 0a15111
Fix #12116: preserve boolean dtype for attention masks in ChromaPipe…
akshay-babbar 64a5187
[quantization] feat: support aobaseconfig classes in `TorchAOConfig` …
sayakpaul ccedeca
[docs] Distributed inference (#12285)
stevhliu c07fcf7
[docs] Model formats (#12256)
stevhliu 76d4e41
[modular]some small fix (#12307)
yiyixuxu 20fd00b
[Tests] Add single file tester mixin for Models and remove unittest d…
DN6 0e12ba7
fix 3 xpu failures uts w/ latest pytorch (#12408)
yao-matrix b596545
Install latest prerelease from huggingface_hub when installing transf…
Wauplin d7a1a03
[docs] CP (#12331)
stevhliu cc5b31f
[docs] Migrate syntax (#12390)
stevhliu 34fa9dd
remove dependencies to old checkpoints
5cc965a
remove old checkpoints dependency
d79cd8f
move default height and width in checkpoint config
f2759fd
add docstrings
394f725
if conditions and raised as ValueError instead of asserts
54fb063
small fix
c49fafb
nit remove try block at import
7e7df35
mirage pipeline doc
814d710
[tests] cache non lora pipeline outputs. (#12298)
sayakpaul 9ae5b62
[ci] xfail failing tests in CI. (#12418)
sayakpaul b429796
[core] conditionally import torch distributed stuff. (#12420)
sayakpaul 7242b5f
FIX Test to ignore warning for enable_lora_hotswap (#12421)
BenjaminBossan 941ac9c
[training-scripts] Make more examples UV-compatible (follow up on #12…
linoytsaban 2b7deff
fix scale_shift_factor being on cpu for wan and ltx (#12347)
vladmandic c3675d4
[core] support QwenImage Edit Plus in modular (#12416)
sayakpaul ce90f9b
[FIX] Text to image training peft version (#12434)
SahilCarterr 7f3e9b8
make flux ready for mellon (#12419)
sayakpaul cf4b97b
[perf] Cache version checks (#12399)
cbensimon 0974b4c
[i18n-KO] Fix typo and update translation in ethical_guidelines.md (#…
braintrue 2d69bac
handle offload_state_dict when initing transformers models (#12438)
sayakpaul de03851
update doc
a69aa4b
rename model to photon
1066de8
[Qwen LoRA training] fix bug when offloading (#12440)
linoytsaban 2dc3167
Align Flux modular more and more with Qwen modular (#12445)
sayakpaul 35e538d
fix dockerfile definitions. (#12424)
sayakpaul 345864e
fix more torch.distributed imports (#12425)
sayakpaul 9e099a7
mirage pipeline first commit
6e10ed4
use attention processors
866c6de
use diffusers rmsnorm
4e8b647
use diffusers timestep embedding method
472ad97
remove MirageParams
97a231e
checkpoint conversion script
35d721f
ruff formating
775a115
remove dependencies to old checkpoints
1c6c25c
remove old checkpoints dependency
b0d965c
move default height and width in checkpoint config
235fe49
add docstrings
a6ff579
if conditions and raised as ValueError instead of asserts
3a91503
small fix
e200cf6
nit remove try block at import
2ea8976
mirage pipeline doc
26429a3
update doc
0abe136
rename model to photon
fe0e3d5
add text tower and vae in checkpoint
855b068
update doc
d2c6bdd
Merge branch 'mirage' of https://github.com/Photoroom/diffusers into …
89beae8
update photon doc
2df0e2f
ruff fixes
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -22,18 +22,12 @@ Mirage is a text-to-image diffusion model using a transformer-based architecture | |
|
|
||
| Key features: | ||
|
|
||
| - **Transformer Architecture**: Uses a modern transformer-based denoising model with attention mechanisms optimized for image generation | ||
| - **Flow Matching**: Employs flow matching with Euler discrete scheduling for efficient sampling | ||
| - **Simplified MMDIT architecture**: Uses a simplified MMDIT architecture for image generation where text tokens are not updated through the transformer blocks | ||
| - **Flow Matching**: Employs flow matching with discrete scheduling for efficient sampling | ||
| - **Flexible VAE Support**: Compatible with both Flux VAE (8x compression, 16 latent channels) and DC-AE (32x compression, 32 latent channels) | ||
| - **T5Gemma Text Encoder**: Uses Google's T5Gemma-2B-2B-UL2 model for text encoding with strong text-image alignment | ||
| - **T5Gemma Text Encoder**: Uses Google's T5Gemma-2B-2B-UL2 model for text encoding offering multiple language support | ||
| - **Efficient Architecture**: ~1.3B parameters in the transformer, enabling fast inference while maintaining quality | ||
| - **Modular Design**: Text encoder and VAE weights are loaded from HuggingFace, keeping checkpoint sizes small | ||
|
|
||
| <Tip> | ||
|
|
||
| Make sure to check out the Schedulers [guide](../../using-diffusers/schedulers) to learn how to explore the tradeoff between scheduler speed and quality, and see the [reuse components across pipelines](../../using-diffusers/loading#reuse-a-pipeline) section to learn how to efficiently load the same components into multiple pipelines. | ||
|
|
||
| </Tip> | ||
|
|
||
| ## Loading the Pipeline | ||
|
|
||
|
|
@@ -46,7 +40,7 @@ from diffusers import MiragePipeline | |
| pipe = MiragePipeline.from_pretrained("path/to/mirage_checkpoint") | ||
| pipe.to("cuda") | ||
|
|
||
| prompt = "A digital painting of a rusty, vintage tram on a sandy beach" | ||
| prompt = "A vibrant night sky filled with colorful fireworks, with one large firework burst forming the glowing text “Photon” in bright, sparkling light" | ||
|
||
| image = pipe(prompt, num_inference_steps=28, guidance_scale=4.0).images[0] | ||
| image.save("mirage_output.png") | ||
| ``` | ||
|
|
@@ -123,11 +117,11 @@ Key parameters for image generation: | |
| ```py | ||
| # Example with custom parameters | ||
| image = pipe( | ||
| prompt="A serene mountain landscape at sunset", | ||
| prompt="A vibrant night sky filled with colorful fireworks, with one large firework burst forming the glowing text “Photon” in bright, sparkling light", | ||
| num_inference_steps=28, | ||
| guidance_scale=4.0, | ||
| height=1024, | ||
| width=1024, | ||
| height=512, | ||
| width=512, | ||
| generator=torch.Generator("cuda").manual_seed(42) | ||
| ).images[0] | ||
| ``` | ||
|
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess we'll be able to store the checkpoint on Hugging Face as well, right? If yes, we should not forget to update the paths here to the official one, to make this truly copy-paste and run.