Any plan for training code?

[shileims]

Hi Author,
This is a really amazing repo.
I check the progress every day and hope to get it to try.
Do you have an estimation deadline for finishing the training code?
Thank you so much!

[lucidrains]

hey yea, i'll circle back to it

you should give it a try though. hard part is done and all the pieces are there, for both single and multi-concept

[IndolentKheper]

Would you mind giving a little advice on how to run it for us non-developers? Like I understand how to install and all that, but actually running the training isn't going to be easy for me and a lot of others to figure out on our own.

[ViddleShtix]

+1 as I am also very interested if you have the time. I'm mostly wondering the broad order of what has to be done, and how we can go about saving the "100kb file" and apply it to a diffusers pipeline.

[lucidrains]

i would recommend starting with xiao's dreambooth / SD repository first, since that is what the authors used. Yoad also offered to further debug once we had something wired together

at this point, you'll still need to find a developer / researcher / ML engineer. i'll try to get it to the point where it is just a command line when i find time

[lucidrains]

broad order, for any entry level ML engineer who is looking to up their skills

1. run a subset of laion dataset texts through the function for generating the input covariance
2. determine the super concept id (must be 1 token id) from embedding table
3. wrap the open CLIP instance with OpenClipEmbedWrapper, passing it also your superclass_string
4. modify the BPE string to id mapping for your new *concept*
5. wrap all keys and values in the cross attentions with the Rank1EditModule
6. pass the outputs from the OpenClipEmbedWrapper through the cross attention, and into the Rank1EditModule from step 5
7. use get_finetune_optimizer on your wired up stable diffusion instance. it should automatically extract only the small number of finetuneable parameters
8. the 100kb of data per fine tuned concepts can be extracted and restored with save and load functions (just pass it in the wired stable diffusion instance)

[lucidrains]

i realize there's still a lot of steps, i'll try to do a bit more damage next week. just writing that up, i realize for number 6, i could also take care of the whole fetching the embedding with the concept substituted with super class concept in the embedding wrapper.

[lucidrains]

actually, i'll take care of 6 tomorrow morning. that's probably the remaining tool needed before operating on SD

[lucidrains]

the input covariance also shouldn't change, so perhaps someone can run this for the clip being used in SD, and we can just check in the tensor, use it as a default. that would save us from passing around this extra matrix

[ViddleShtix]

Here's what I got for the input covariance by taking the first 100k captions from the LAION 400m dataset and running it through the function (perhaps it can be verified in the future):
tensor([[ 9.3166e+00, -4.6043e-02, -1.1625e-01, ..., 6.4781e-02, -1.4543e+00, 8.6031e-01], [-4.6043e-02, 1.0780e+01, -2.8464e-01, ..., 7.0342e-01, -2.0617e-01, -3.6634e-01], [-1.1625e-01, -2.8464e-01, 1.0130e+01, ..., 3.4655e-01, -2.7999e-01, 2.1443e-03], ..., [ 6.4781e-02, 7.0342e-01, 3.4655e-01, ..., 1.0315e+01, -6.7360e-01, 1.0505e+00], [-1.4543e+00, -2.0617e-01, -2.7999e-01, ..., -6.7360e-01, 1.2800e+01, -2.2284e+00], [ 8.6031e-01, -3.6634e-01, 2.1443e-03, ..., 1.0505e+00, -2.2284e+00, 1.4173e+01]])

Please note there is a slight bug in the calculate function. The first line needs to be removed since it tries to take in all the data instead of batching it, causing it to max out memory and crash.

I calculated the covariance on the ViT-L-14 CLIP model since that's what SD 1.5 uses. My OpenClipAdapter looks like this:
OpenClipAdapter(name="ViT-L-14", pretrained="laion400m_e32", tokenizer_name="ViT-L-14")

I saved the output with torch.save and have attached it here. Hope this helps.

input_covariance_ViT-L-14.zip

[lucidrains]

@BradVidler amazing Brad! thank you so much 🙏

i'll get this checked in

[lucidrains]

@BradVidler checked it in for version 0.1.6! we can now scratch the first step off the list haha

[lucidrains]

also addressed returning the prompt with superclass embed, if in training mode

[lucidrains]

also added a save and load function this morning. if you read the file, should be self explanatory. since the keys are locked to the super class concept output, there is still room to whittle down the size of the saved package even more

will get around to it next week

[irowberry]

Would love to contribute where I can, it looks like you're knocking out that list quickly though.

[lucidrains]

@irowberry sure, you can submit a PR at any time, or try wiring it up to SD 1.5 and give it a go