Loss function for face training

[kidol]

I've noticed the loss function takes two 64,64,4 vectors. I guess those are the latent vectors produced by the autoencoder. To generate an actual image, a 64,64,4 vector is given to the autoencoder (decoder part) to produce the final 512,512,3 vector.

Now in context of hypernetworks and the task of face training, would it be beneficial to directly apply a loss between input image from training and output image from network? While that would require to do inference at training time, certain loss functions like cosine loss could be applied.

Thoughts?

[Nacurutu]

I dont know much about how TI or HN works, but maybe this could be interesting for you, I think is related to what you asked:

#6700 (comment)

[kidol]

Yes it's related, thanks!

[Nacurutu]

More than welcome!

[Shondoit]

Currently the VAE is only used to get the training data into latent space, afterwards the VAE can be unload and all training can be done a lot quicker.

If you're talking about latent to pixel image to CLIP image embedding and comparing the image embedding to the text embedding using cosine similarity, I've wondered the same thing: I wonder if it can learn better details or learn to generalize better.

It's something I would like to try in the future. But I'm not sure that avenue will be fruitful.

[kidol]

@Shondoit No, I think about full inference. For example, using prompt "face" to generate a random output image. Then detect the face and it's orientation and pick the image from training set which matches best and calculate cosine loss. Not sure if it's possible to flow the loss backwards then?