The fight against non-deterministic results

[sarahhemail]

So, hello community.

I'm aware that several commandline args (such as xformers) cause non-deterministic behaviours.
What is most likely less common knowledge is for example the torch.backends.cudnn.benchmark flag in devices.py, as it might cause non-deterministic behavoir across sessions (relaunching the WebUI) (see doc here https://pytorch.org/docs/stable/notes/randomness.html)
I must not, but may give different results after relaunching (could confirm it in 10/10 cases, with flag = true and false)

Now for the more important part, GPU. That is the biggest problem in my opinion. The results I get from a rtx 3060Ti extremely far away from the ones on rtx 2080 Super. And I can't affort to buy tons of machines just to see wich gives me close results.

If I get it right, the GPU, wich creates the noise in response to the give seed. Given the same seed a 2nd, without having anything changed at all, would produce the exact same noise, resulting lastly, in a 100% identical output.

Now a different GPU, has the same behaviour, but produces different noise for the very same seed. Kinda similar to different search engines, same input, different output, more or less.

Now I wonder about following
A) what would happen if I have 2 different machines, but both of em having the examt same GPU, drivers and CUDA kids (and OS)
B) the information on the pytorch doc, and these lines in 'progressing.py':
# randn results depend on device; gpu and cpu get different results for same seed;
# the way I see it, it's better to do this on CPU, so that everyone gets same result;
# but the original script had it like this, so I do not dare change it for now because
# it will break everyone's seeds.
noise = devices.randn(seed, noise_shape)
lead me to the idea, that it could be eventually done via a offset??? (i think that is what people doing using ensd without knowing, event though it doesn't change anything for me when change ensd)

C) anyone tried on 2 different CPU's ?

[Sakura-Luna]

I didn't disable benchmarks, but have never seen an example on a local device where a reboot would change things, and even if I reproduce the results now, they are 100% the same.

I feel that A) may still have a small amount of variance. PyTorch does not guarantee consistency in different environments. The documentation only shows the means to reduce uncertainty, and your resistance may be in vain.

Maybe you can try to compare the results in CPU mode by running locally and on cloud services (such as colab).

[Sakura-Luna]

This PR pointed out that he got similar results, you can give it a try.

[Trevor-Z]

Maybe we could run a test, several people state their settings and hardware and run the same prompt in the same model, etc, and post the result?

Or someone could create an extension for that, like the one that does benchmarks and collects the info.

[Sakura-Luna]

Given the fact that it's hard, maybe someone can make a Docker image and distribute it to keep the software variance as low as possible.

[sarahhemail]

Maybe we could run a test, several people state their settings and hardware and run the same prompt in the same model, etc, and post the result?

Or someone could create an extension for that, like the one that does benchmarks and collects the info.

the last time it was:

• Aurora R11, Rtx 2080 super, I7-10700KF, Windows 10
  vs
• Legion T5, rtx 3060 Ti, i5-12400F, Windwos 11
• both automatic a9fed7c

I tried everything I could think of aside of messing arround in the code of the WebUI and results were totally different, with both fixed and random seed.

People reporting different experiences, Some say they get almost identical results (even between 10xx and 30xx series), some fail miserable, like me. I currently exchanged the LegtionT5 to another one, but hasn't been delivered yet.

When I got it, I can run some tests, on both hardware and software end.
Maybe turning off 'torch.backends.cudnn.benchmark' might at least help to get closer. But it doesn't explain why for some people its so, for some people not.

[sarahhemail]

here are some tests.
it's a alienwareR11 (RTX 2080, Win10) vs (RTX 3070, Win11)
torch.backends.cudnn.benchmark has been disabled

1. cpu vs. cpu: minimum differences, regardless, no option as simply by far too slow and obvious seed killer
2. gpu vs gpu: medium differences, the superior image is the same but there is constant style shift
3. gpu vs gpu*: same as nr.2 but with the images resembling the cpu test
4. swapping gpu: each gpu produced 100% reproducable result in the opposite machine
5. downgrading from Win11 > Win10: no influence, results were 100% reproducable
6. trying some different drivers (including different cuda driver versions) no influence, results were 100% reproducable

So it appears it's really just a hardware thing. Sadly, as I get somewhat of a style shift, as mentioned above. And that's the biggest problem I see. After a 500 batch, that what clearly noticeable in most of the pictures. Not a single one looked as good as it does when I work with the RTX 2080.

-* Initial noise has been done via cpu by modifying devices.py. I was hoping this gives a more stable, less different result, but unfortunately it does not. And since the "starting shot" comes from the cpu, it's very seed braking.

Guess I try to get another 2nd rtx 2080 -.-

[noisefloordev]

Having results be worse (and not just different) suggests something more than just different PRNG in the noise, right? If you tried generating the noise on the CPU and it's still different, does that point to something other than noise entirely?

(Just throwing darts, it's probably hard to get help on this since most people don't have multiple GPUs to compare...)

[sarahhemail]

I don't think so. Getting noticable differences Gpu <>Cpu is what i did expect. And for Gpu<>Gpu, like I said, noticing a style shift in most of the images. It's not huge but still annoying.

[Trevor-Z]

Also, you should post the resulting images in png, and the prompt and settings you used, so others can try and compare.

[sarahhemail]

I try to make some new ones, deleted everything. However, I think it's not helping much. I've proven that non-deterministic influences can be reduced to Gpu only.

I try to get another rtx2080 and see if the results are 100% identical (they should be...)

Actually my attempt is to minimize differences between different Gpu's. But i'm out of ideas. Maybe not possible in the first place.

[Trevor-Z]

Xformers 0.0.19 is said to be deterministic, and indeed in the same session I get the same result when regenerating the same prompt.

But when I close and restart SD, I get the usual slightly divergent result, although once more constant for the duration of the session.

[Kadah]

I've seen more deterministic results on my setup by enabling Extra with a fixed seed and Variation strength set to 0.