Medium- and Small-sized Faces

[rstockm]

Hi, I get stellar results with my models trained on a face using "portrait" and/or "closeup" prompts - when the face covers most of the 512px area or is even bigger. Really impressive.

Bit when I try to render the same model on a more wide-angle shot with more of the body visible, things change dramatically. There is a faint resemblance at best, more often no likeness at all. Sometimes to the extend of "cursed faces" with washed out features.

Things I tried:

• throwing some pictures from midrange into the learning mix
• prompt optimization
• different output resolutions
• more/less training pictures and learning steps

...but to no avail. So Is there a secret I'm missing? Somne best practice for medium sized/small faces?

[TheLastBen]

you can train on a 1024 resolution images, you'll get better results, but at inference you shouldn't go below 1024

[JustAnOkapi]

is there any other way?

[rstockm]

OK, I will try. What do you mean by "inference"? (I'm using Draw Things on a M2Pro)

[TheLastBen]

inference or when generating

[THX1139b]

The way I deal with this is to upscale and then manually crop out the face in Photoshop and do an img2img on that new image before pasting the result back into the original upscaled image in Photoshop. I haven’t done it that much lately tho since it’s a lot kore work and I’m lazy and I prefer a face to be in the foreground anyway, so typically I just add the words closeup, portrait or headshot to my prompts and generate more new images.

I’m going to use this method again this upcoming week because I’m making some posters that will be enlarged to… well, poster size, so then I’ll be OK with more of a landscape with the person being a smaller part of the overal image area. But you have to really look away from the fact that your subject doesn’t look good at all and work on the image anyway because the overall image and composition and everything else is great.

Also I have to say that often img2img simply doesn’t work for me and I get no improved details that way for some reason. It happens.

[SorenTruelsen]

I use the batch-face-swap extention. When ever I try to use upscale, it changes the picture too much and I have a hard time finding the sweet spot.

[dr-formalyst]

I am having the same results and my guess -maybe I'm wrong- is because Stable Diffusion does not have idea what the face (or any other concept) is and that it should be resized. Stable Diffusion works by adding noise to images (when training) and progressively denoising them (when generating new images). This is a gross oversimplification but it can help understand why generation of small faces would not work - simply it can't recognize a small patch of say 100x125px as a "candidate" for your face, if you trained it on your photos where your face mostly fills 512x512px area. It will though throw other people faces in that 100x125px area simply because of many photos it was trained on, where some of them had smaller faces to begin with.
I am not sure if this is a correct interpretation, but it feels logical. Neither do I know what solution may be. Maybe train on the full body shots where the head is small in the format (similar to what the size of your generated "heads' should be on the format)? I haven't tried this, but I am considering doing it, to see if it can fix it.

[kgonia]

It's probably wrong explanation. This project https://github.com/NVlabs/ODISE shows that SD uderstands what's in the image.

[meunumerotim333]

I have tried some "cheat" for some specific images that maybe you should try. I generate 1024x768 picture very close to the face and them I used Dalle-2 to outpaint the cenary and them I use inpaint on SD to restore the image to the according style. It's harder but I've found this solution for specfic results.

[dr-formalyst]

Thanks, @meunumerotim333 this is an interesting workflow! I am wondering, why are you using Dalle-2 for outpainting?

[meunumerotim333]

Because I use Colab and I don't know how to generate good outpaint on Colab.

[jimys]

if I generate large picture, every thing goes wrong,

[meunumerotim333]

Use ControlNET

[jimys]

also multiple head, extra legs