Idea: Conditional Averaging of two prompts

[softtaco1]

Flux seems to follow a narrow vector, allowing great prompt adherence without bleed- but in doing so works on its own terms.

It seems to miss certain ideas given in a prompt due to this narrow vector.

As a fix, it'd be awesome if either by native feature or extension we could combine two prompts with conditional averaging, like is able to be done in ComfyUI.

The other implication of creating a conditional average is to combine both clip_l and t5xxl without lost prompt adherence as using them together negatively impacts prompt adherence. For example, the user would create two separate prompts. One prompt would use t5xxl (natural narrative language) and one would use clip (language separated by tags, more similar to SDXL). Then, an average would be created between the two, leading to (in theory) better prompt adherence. This is something I would have to further test to see if it has merit. See this for further analysis on how using both encoders negatively impacts prompt adherence:

#1182

The images and some of the info were sourced from Latent Vision's video, highly recommend for those using Flux: https://youtu.be/tned5bYOC08?si=jloiEz-dR8ecYFJG

[lllyasviel]

this is supported in forge and proposed by auto 2 years ago way before those
https://github.com/AUTOMATIC1111/stable-diffusion-webui/wiki/Features#composable-diffusion

[softtaco1]

Wow, I have no idea how I missed this! Thanks so much for taking the time to reply with a way to do exactly what I was looking to do :)

Love to see that the idea for dual prompting in both Clip and T5 language was taken up by a member of the Forge community, if anyone sees this in the future and is looking for that in particular, please check this extension by DenOfEquity out:

#1182 (comment)

https://github.com/DenOfEquity/forgeFlux_dualPrompt

Thanks to lllyasviel also for being courteous and extending some help on how to best code for Forge in reference to that extension in particular. This could be helpful for anyone else looking to develop extensions for Forge:

#1182 (comment)

Thank you as well for all your work toward Forge, lllyasviel! you're a gift to this community!