Use AB results to nudge CLIP output

[chrisgoringe]

The AB tool effectively creates a mapping from image to aesthetic value. We've found that training a model to map from the CLIP encodes of those images to aesthetic value can be quite accurate.

So that suggests we could generate a direction in the CLIP space that is liked (sum of the CLIP vectors weighted by their aesthetic score, for instance).

Could this then be added to the CLIP encoding of a prompt?

[chrisgoringe]

SDXL uses two CLIP encoders, l and g. They return torch.Size([B, 77, 768]) and torch.Size([B, 77, 1280]) respectively, which are concatenated torch.cat([l_out, g_out], dim=-1) to produce cond of shape (B,77,2048)

[chrisgoringe]

Could this be done as a LoRA? By patching the end of the CLIP?