[New Feature] Flux PuLID

[SahilCarterr]

PuLID: Pure and Lightning ID Customization via Contrastive Alignment

Paper:Link
Code:Link

This has astonishing result with FLUX Pipeline

I Think it Should be added to Main Pipeline. @yiyixuxu

[maxs-kan]

Hello. Initially, I wanted to create a pr for this task. However, in the process, I realized that the amount of changes needed is more than what should be in the public pr. Anyway, I have started the process and hope this pieces of code will be useful.

I redesigned cross-attention blocks from the author's code to diffusers and made the weights loading. I added changes to transformer, pipeline. Here is the test on one person, in my work I did the test on several persons, and everything worked as expected.

In my opinion, we need to create a 'LoaderMixin' like for IP adapter. The main problem is that cross attention with id_emb in PuLID is done differently compared to IP adapter it is separated from cross attention with text, and done after the whole block. Because of this, I do not know how best to proceed, to create a separate class like 'IPAdapterLoaderMixin', only for PuLID or to embed in the logic of the IP adapter.

[newideas99]

Thank you for doing this!! Any luck with adding lora support?

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[nitinmukesh]

This is interesting feature to add in diffusers

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[bghira]

not stale

[Mikerhinos]

Taking this out from the grave as I'm sure I read a few months ago that it was added in the diffusers and that's why the demo is twice as fast ?