Training code for lambda-ECLIPSE

[j-min]

Hi, thanks for sharing the inference code for lambda-ECLIPSE; nice work!

Do you plan to release the training code for the lambda-ECLIPSE? Such resource-efficient training would be very useful for many low-resource groups. I'll be looking forward to trying this.

[Maitreyapatel]

Hello @j-min, thanks for showing interest in our work.

TL;DR: We are building an end-to-end project that can help us utilize the true potential of CLIP models for T2I. But public release may take some time.

Long answer:

Current progress/hype in T2I is largely due to three works: SDXL-Turbo (inference time efficiency), DALL-E 3 (SOTA), and Playground v2.5 (highly aesthetic). Majority of the such works use cleaver tricks (like synthetic captions, DPO, etc.) which we haven't explored yet in ECLIPSE/UnCLIP settings. Hence, the true potential is still a mystery to us!!

Due to the efforts involved in building the end-to-end system, we are projecting the partial release in May. We are also looking for help/collaborations to speed up the process and explore its use cases in unknown territories. If you are interested please reach out to me at: maitreya.patel@asu.edu

Alternative: The training is pretty straightforward and uses a mix of contrastive learning and projection loss as described in the paper.