Hunyuan Lora in Comfy

[hashu786]

is there any plans to release some sort of a conversion script to make a Hunyuan lora trained with finetrainers compatible with comfy? Right now I just get a bunch of "lora key not loaded" errors.

[a-r-r-o-w]

Currently, Comfy does not support loading diffusers-format loras (conversion script to diffusers format from original is available here).

So, the only way to be able to load it is by writing a reverse script of above to go from diffusers format to original. I don't think I have the bandwidth to do so myself atm, so would be super cool if someone could help out with the script. I will try and find the time if no one picks it up

[neph1]

I've one for ltx loras, and I could probably adapt it to hunyuan as well (and make a PR for it in comfy). I've however not been able to run hunyuan locally yet, so I don't know what the issue is. Can you post some of the keys it complains about and I can try to make a blind fix. Alt, if there is a small lora I can test with.

[cseti007]

Hi @a-r-r-o-w and @neph1 !
As I see Kohya has a convert script from diffusers to default. Could this do it? https://github.com/kohya-ss/musubi-tuner/blob/main/convert_lora.py

[hashu786]

not my area of expertise but I created a quick convert script to work with comfy and a test example lora trained on all target modules. Seems to be working for me for now but I am sure lots of the script can be improved to be made better.

https://huggingface.co/hashu786/cine/tree/main

[a-r-r-o-w]

@cseti007 The Kohya one is not really a full conversion script, and I'm about 95% sure it shouldn't work OOTB with comfy that way.

FWIW, I wrote a partial conversion script here: huggingface/diffusers#10376 (comment)

It should account for most loras being trained but still requires handling a complete layer to make compatible with original format. I'll try to finish it up soon. Thanks for offering to help @neph1 - would be awesome if you can bring it to completion but I suspect it might require some debugging so please do not spend time on this unless purely for educational purposes or wanting to continue improving this 🤗

[spacepxl]

I tried @hashu786 's conversion script (which looked like it was based on @a-r-r-o-w 's partial script?) and found that it wasn't able to give comparable results in comfyui vs diffusers. I'll admit I might be missing some understanding here, but the solution that seems to be working was to concatenate the lora_A weights and make a block diagonal matrix from the lora_B weights. This effectively increases the rank for the fused weights by 3x (double) or 4x (single) vs the split weights.

Here's my diffusers lora -> original/comfyui lora conversion script: https://gist.github.com/spacepxl/b6b8be274ef289250106e5bd85a92959

It does inflate file sizes a bit due to all the zero entries in the lora_B weights but the results in comfyui are practically identical to diffusers inference. I've tested it with all attention and mlp weights in the double and single blocks. Did not bother implementing the token refiner but it should use a similar pattern to the double blocks.

[a-r-r-o-w]

@spacepxl Looks super cool and thank you very much for taking this up 🤗 While I love comfyanonymous' work, his comments on not wanting to support other lora formats is disheartening. We're supportive of everything open source, so opposed to what he thinks, we'd love to have a conversion script that goes from diffusers format -> original format (would not call it comfy format since it's not a format he invented and since they use the original code directly instead of standardized model implementations that are more preferable among researchers/students). Both comfy and diffusers have their own usecases, but trying to force people to use one thing over the other is not justifiable, but seems like comfyanonymous is very adamant and no one can convince him otherwise. Would you be able to open a PR with your script to a new folder in finetrainers: maybe something like examples/conversions/hunyuan_video/lora_block_diagonal.py or whatever you prefer?

[spacepxl]

@a-r-r-o-w PR submitted. I actually totally get comfy's perspective on this, supporting a bunch of different formats is a ton of busy work that could be better spent elsewhere, and it makes more sense to only support the original weights and push the conversion to the training stage.

I do love diffusers for the modularity though, much easier to write custom training code on it :)

[Reginald-L]

how about ltxv lora model

[a-r-r-o-w]

The original LTX model implementation (which Comfy supports natively I think) was based on Diffusers-style so should almost fully be compatible with Comfy already. These are the changes in the transformer: https://github.com/huggingface/diffusers/blob/57ac6738028004143cf19c362a81d7d135d1de24/scripts/convert_ltx_to_diffusers.py#L19

[a-r-r-o-w]

I believe that we reached a conclusion here with @spacepxl contributing the conversion script (thanks!).

In general, I don't think Comfy wants to have anything to do with diffusers format weights going forward (atleast not natively), so someone will have to maintain a lora loader node if there's interest for training with finetrainers but using in Comfy. On our end, a colleague is working on two-way weight conversion natively in diffusers so that we can maintain our standardized implementations as well as allow for conversion to original-authors format for anyone that needs it.

There are multiple UIs and it's hard to try to be compatible with them all, so the current design of finetrainers supports writing your own custom model specifications quite easily -- the main trainer logic handles most formats of dataset & loading them, distributedness for scaling training, checkpointing, validation, etc. in a model agnostic manner (well about 80% of it since some fill-in-the-blanks need to be done by the user with ModelSpecification's). It comes with some sacrifice of hackability, but that was never the goal.

I really appreciate the work some folks have done to take steps into enabling cross-compatibility, thank you!