Positional Stickiness

[afiaka87]

For lack of a better word; I've noticed during training that the VitGAN tends to get stuck on one, two, or three (i don't see four happen very often/at all) "positional blobs" for lack of better words.

Does this match your experience? Effectively what I'm see is that the VitGAN needs to slide from one generation to the next in its latent space. In doing so - it seems to find that it's easier to just sort of create two "spots" in the image that are highly likely to contain specific concepts from each caption.

Does this match your experience? Any idea if this is bad/good? In my experience with the "chimera" examples; it seems to hurt things.

I hope you can see what I mean - there's a position in particular that seems designated for the "head" of the animal. But it also biases the outputs from other captions as well; for instance -

tri - x 4 0 0 tx a cylinder made of coffee beans . a cylinder with the texture of coffee beans .

[mehdidc]

Yes I see exactly what you mean and noticed this in all the models I trained (both VitGAN and mlp_mixer), this was the reason why I started by the way to work on the diversity loss, but it does not work very well so far (quality drops and if the coeficient is small enough it only starts to output the same objects with different palettes if the noise vector changes, if the coeficient is bigger then it just outputs random textures unrelated to the prompt). Yes, I also thought similarly that it was a kind of shortcut the model finds to optimize the loss, and the first thing I could think of to fix it is to explicitly add an additional constraint on the loss (diversity).

[mehdidc]

It could also be related to the architectures themselves (VitGAN and mlp_mixer), not sure. Another way to make the constraint even more explicit is to add a diversity loss on the "shapes" of the objects (rather than the "style") to avoid the model sticking on one shape.

[afiaka87]

@mehdidc interesting so this happens with mlp-mixer too? Hm. There are of course examples to this which sort of defy the notion that it's a universal problem - if the captions are sufficiently different you can get it to output something quite different. For instance - the "photo of san francisco" captions tend to produce wildly different outputs - although you can sort of see that it's still getting caught up on the same position:

I think in this example - rather than "animal head goes here - other animal's body goes there" - it is changing to "foreground goes here - background goes there"

[mehdidc]

"For instance - the "photo of san francisco" captions tend to produce wildly different outputs " Ah okay, so what are the text prompts here where you observed different outputs, you mix "photo of san francisco" with the different attributes that you mentioned before such as "8k resolution"?

[afiaka87]

@mehdidc Indeed - but I saw very similar results from the prepend all captions with minimalism checkpoint and I believe it would still happened with no modifications to the captions.

[afiaka87]

Generating a video from the training samples is maybe the best way to illustrate the issue:

ffmpeg -framerate 15 -pattern_type glob -i '*.png' -c:v libx264 -pix_fmt yuv420p training_as_video.mp4

[mehdidc]

Yes exactly!

[mehdidc]

Another way to see is through interpolation, here is an video showing interpolation (of text encoded features) from "the sun is a vanishing point absorbing shadows" to "the moon is a vanishing point absorbing shadows": https://drive.google.com/file/d/16yreg0jajmC4qwJGmiGJypp_L_VQbhq5/view?usp=sharing

[johndpope]

Not an answer - but perhaps a direction for enquiry in this repo by @nerdyrodent
https://github.com/nerdyrodent/VQGAN-CLIP

in the readme - he spells out a way to alter the weights of the words being passed in.

python generate.py -p "A painting of an apple in a fruit bowl | psychedelic | surreal:0.5 | weird:0.25"

for me - this coefficient tacked on to the end - provided a way to directed output with more control.

tri - x 400tx a cylinder made of coffee beans :0.5
. a cylinder with the texture of coffee beans

Need to dig into the code - but this may help transcend this blob problem.

[afiaka87]

Another possible direction: @kevinzakka has a colab notebook here for getting saliency maps out of CLIP from a specific text-image prompt.
https://github.com/kevinzakka/clip_playground/blob/main/CLIP_GradCAM_Visualization.ipynb

[mesw]

I actually like this so called "stickiness" effect. It creates some kind of continuity over large amounts of generated images.

Is there a way to influence the "sticky spacial shape"? Is it shaped during training, or can it be modified somehow?

[afiaka87]

@mesw As far as I can tell it is inherited from the biases present in CLIP and perhaps in the specific captions used to train this repo.

@mehdidc oh yes I forgot to mention. @crowsonkb suggested that the loss used in CLOOB "InfoLOOB", would be appropriate for these methods and may help output more diverse generations for a given caption.

The blog linked mentions the problem of "explaining away" which seems correlated with this issue perhaps?

edit: Implementation here https://github.com/ml-jku/cloob

[pwaller]

I put in the whole of /usr/share/dict and did a nearest neighbour search over Z for the term nebula, which gave this, which I think shows a hint as to the positional stickiness. You get a similar result for any word you can think of - lots of images that look very similar.

What I would like to see is a 'seed' input alongside the text input, which provokes the net into producing a completely different output for the same textual input. Is there a way to train that? Could you, for example, have a loss which makes it so that if the net is fed prompt=(foobar) cross product seed=(1, 2, 3, 4, 5), the loss is higher the greater the distance of the GAN coordinates are for the different seeds?

You can provoke the net into producing different images by random vectors to H, but it's not the same! :D

nebula, meteoric, interstellar, insignificant, splatters
astronomers, elemental, geode, trinities, transcendence
foretold, rift, glitters, atom, glistens
enthralls, divinity, angelically, opalescent, mystique

[pwaller]

It's interesting that all of the above images have a black spot in the bottom right. What's that about? :)