Replies: 29 comments 55 replies
-
|
Currently, the speed of Nvidia version 531.79 is normal during testing. I'm not sure if the old version 531.68.61 will be faster. I'll test it later. |
Beta Was this translation helpful? Give feedback.
-
|
Tested with versions 531.79-68-64-41-29, the speed difference does not exceed 10 seconds. The 532 version has a speed reduction of 20-50%, and the 535 version has a speed reduction of over 9000%. This situation may be encountered on graphics cards with less than 12GB of memory. |
Beta Was this translation helpful? Give feedback.
-
|
Confirms what I thought was a slowdown after upgrading to 532.03. Rolling back to 531.79. Is there any difference using the 535 studio driver? Not that it would help me. My rig is gaming + AI. |
Beta Was this translation helpful? Give feedback.
-
|
in my test, [email protected] work well |
Beta Was this translation helpful? Give feedback.
-
|
My generations werent really slower but I had a lot of artifacting when using tiled vae or img2img. Its gone now, I rolled back by installing an older studio driver |
Beta Was this translation helpful? Give feedback.
-
|
Driver ver 535 is a shame, A1111 runs tooooo slowwwwww on that driver version. I had to get back to 532. Problem solved. |
Beta Was this translation helpful? Give feedback.
-
|
It would be a great feature if efficient shared VRAM usage could be achieved by the driver alone. The premise is that Nvidia provides a switch and at the same time cooperates with the program to optimize the use of logic. |
Beta Was this translation helpful? Give feedback.
-
|
Nvidia released 536.99 a few days ago, oddly the impact on SD speed is no longer mentioned in the logs for this version, which is weird, no one knows if Nvidia thinks it's fixed. |
Beta Was this translation helpful? Give feedback.
-
|
Nvidia's goal/solution is likely that we adapt our workflow to use their olive tensor implementation (and don't run projects large enough to dip into system memory). |
Beta Was this translation helpful? Give feedback.
-
|
In fact, a FP32 model generates with 2x the IT/s than FP16 that i pruned from it. So the driver indeed screwed with speeds. An sadly, the newest drivers has diablo 4 optimizations, so i'd have to have SD on a separate windows install lol.
I would rather it dip into system memory than using pagefile on my poor nvme whenever video memory runs out from a generation heh. |
Beta Was this translation helpful? Give feedback.
-
|
DON'T UPGRADE TO v535.** |
Beta Was this translation helpful? Give feedback.
-
Thank you for testing. I still don't know what exactly the reason why nvidia 535 and 536 conflict with A1111 rendering |
Beta Was this translation helpful? Give feedback.
-
|
Tested it on 3060 12GB with 32GB System Ram and this tools: A1111 its not fixed generating needs 8-14 times the time than normal What ever Nvidia has done i think they done it because desktop open source tools are compeding with their Partners that want money for online services, this services use most of the time NVIDIA A100 Tensor GPUs and when you test them in GPU for Rent Websites they run faster than before.......... so i think this is an software attak on open source tools for SD. We can only hope someone makes an tool in C++ and that support the new AMD GPU or even Intel GPU....... |
Beta Was this translation helpful? Give feedback.
-
|
I think so too. |
Beta Was this translation helpful? Give feedback.
-
@mattehicks How so? something is wrong with your setup I guess, using 3090 I can generate 1920x1080 pic with SDXL on A1111 in under a minute and 1024x1024 in 8 seconds. All with the 536.99 latest nvidia driver and xformers. It's true that the newest drivers made it slower but that's only if you're filling up your VRAM and when shared memory gets used. |
Beta Was this translation helpful? Give feedback.
-
|
I'm well aware that something is wrong, I think something got triggered when I added a new CPU with onboard graphics (to get a 5th monitor working with my 4090). But I've since removed it, to troubleshoot this problem. |
Beta Was this translation helpful? Give feedback.
-
|
Hello, Sorry if this not in the correct place. I had installed Konya_ss with the new installer. Very good. In middle I install the NVIDIA library : that run very fast my 3090. Can be of interest to incorporate to A1111 Hope this info be good for this community. THANKS FOR ALL ! |
Beta Was this translation helpful? Give feedback.
-
|
Which folder location should I place the "cudnn_windows" folder? |
Beta Was this translation helpful? Give feedback.
-
|
Could also be the source of sudden NaN errors when using Adetailer, maybe triggered by mixing Adetailer's models with ControlNet or the mixing of FP16 with FP32. It's a shame because I'd rather it dip into shared memory when going above VRAM limit at the temporary cost of speed, instead of just stopping altogether, but without the extra bugs. |
Beta Was this translation helpful? Give feedback.
-
|
If you have actually triggered this bug, you will know that it is better to directly report an OOM error. |
Beta Was this translation helpful? Give feedback.
-
|
Oh I know, just saying this would be better without that bug, not only for SD but also oobabooga, audiocraft, etc. I'd rather it be like this: go over VRAM due to memory use spike, etc. > slow down for a bit > task still gets finished Of course right now with this bug it's like this: If Nvidia actually fixes the bug, it would be a nice feature to have, I haven't gotten OOM errors in any AI UI since the new drivers. |
Beta Was this translation helpful? Give feedback.
-
|
This doesn't matter much because Windows 11 insider drivers are mostly limited to insider builds and their changes are all over the place with a mix of older than stable branch code and new, despite having higher build numbers than regular drivers, but Nvidia Insider driver 545.29 doesn't seem to have this memory management change. |
Beta Was this translation helpful? Give feedback.
-
|
I've been noticing Stable Diffusion rendering slowdowns since updating to the latest nvidia GRD but it gets more complicated than that. I had heard from a reddit post that rolling back to 531.79 would solve the speed reduction and it did but a reboot undid that and returned me to slow-land. This was especially noticeable as I had finally gotten xformers (0.0.17) to load without errors and instead of a speed boost I was getting painfully unusably slow render speeds. Turns out reinstalling the drivers without reboot, regardless whether the latest GRD or the rollback version people have been suggesting, fixes the problem temporarily, at least for the current desktop session, and then the problems return after reboot. The workaround for this is to reinstall nvidia drivers prior to working with stable diffusion, but we shouldn't have to do this. The silver lining is that the latest nvidia drivers do indeed include the memory management improvements that eliminate OOM errors by hitting shared gpu (system) RAM instead of crashing out with OOM, but at the expense of speed when it's fallen back to system memory. This general slowdown does not appear related to the slowdowns one would expect when falling back to slower memory, but the slowdowns are similar. You can watch this play out on the performance tab of Task Manager when memory overflow is the reason for your slowdowns. I still have no idea why failing to reinstall nvidia drivers prior to an SD session results in speeds comparable to using system RAM, or if they are even related. On my system I get over 15it/s with a blank prompt at 512 square using default A1111 settings, euler a, 20 steps, CFG 7. This is only after a reinstall of the latest drivers from nvidia, which were already installed. This is repeatable. Is it unique to my system or are others noticing the same things with their A1111 install? When it's running without first reinstalling nvidia drivers (without reboot), I get more like 1.2it/s, which as you might guess is really noticeable. One more thing. I was told by someone that the new version of torch makes xformers kind of irrelevant but it doesn't seem that way from my perspective. I tested some renders without it turned on in launch args and they seem a little slower than expected but it seems like xformers is still boosting things when I have it turned on. I have to test whether a fresh reinstall of nvidia drivers changes that assessment, but I know from testing that it has accelerated things back to 15it/s with --xformers on (and loaded). I will update this when I've had a chance to test that scenario against xformers being turned off and see if the performance returns the same as it does when reinstalling nvidia drivers with it turned off. UPDATE: I just removed --xformers from my a1111 startup args and did a test render post-nvidia-reinstall and got about 10-11it/s, so it seems xformers (I'm running 0.0.17) is not obsoleted by the torch upgrade, at least not entirely. I still get 15-16it/s with it turned on, as long as the drivers have been reinstalled during the current session, otherwise things slow down almost entirely, to 1.2it/s or the even worse flipside of that, seconds per iteration. I put it back on and 15+ it/s as expected. I don't know if that's fast or slow but at least it's back to the speeds I got used to before I ever got xformers to load properly. Until reboot at least. UPDATE: Well, now I'm getting what I would consider normal render speeds without having to reinstall the nvidia driver so this is where I throw up my hands and shrug. Device name DESKTOP-QH9LO7U |
Beta Was this translation helpful? Give feedback.
-
|
@w-e-w yah I am mostly using only 1.5 models. By 4k I mean 3840x2160 (wallpapers for my 4k monitor), I remember when 1.3.0 had SDP by default but it was the option "without memory efficient attention" and it indeed caused OOM issues. But using just SDP solves the OOM error. You can simply test it against doggettx or xformers, keep raising the resolution until you get OOM, at that same resolution SDP won't throw OOM error while being the same speed as xformers (and much faster than doggettx). I mostly use DPM++ SDE Karaas or 2S A Karaas but the OOM with xformers and SDP without mem efficiency happend on various sampling methods IIRC even euler a. Edit: just noticed that on Vlad SD.Next SDP is the default optimization. |
Beta Was this translation helpful? Give feedback.
-
|
all I can say is that things are getting weirder and weirder we know that if you have a better CPU relative to GPU xformers should be slightly faster and the other way around this is why I pretty much given up figuring out which is the best |
Beta Was this translation helpful? Give feedback.
-
|
Thanks a lot for testing, I know things like this can be complicated especially with the significant variation of hardware among different rigs. All I can say is that SDP has been the best experience for me and that doggettx was the worst (not only because of OOM but it's too slow), xformers is fine but had a few OOM errors with it with the workflow I do (generating large images with hires fix and using various upscaling methods). |
Beta Was this translation helpful? Give feedback.
-
|
@w-e-w I know this is late now but I thought it's worth mentioning that after I updated the Nvidia drivers to the latest (536.67) I tried xformers again and it's actually much better than I remember it be. It's on average ~30% faster than SDP/SDP-no-mem and it no longer OOMs on large resolutions like it used to for me. So right now I highly recommend xformers. |
Beta Was this translation helpful? Give feedback.
-
|
lol |
Beta Was this translation helpful? Give feedback.
-
|
Nvidia is aware of performance issues with the new memory management and will provide a fix in future driver updates. |
Beta Was this translation helpful? Give feedback.
-
|
Great, where did this news come from |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
What driver does run well? I had way better performance im march than now. I went back to 531 version, but it did not help. |
Beta Was this translation helpful? Give feedback.
-
Clean uninstall your driver, restart and reinstall Game Ready Driver v531 or 532. They are both work well for me (RTX 30 series) |
Beta Was this translation helpful? Give feedback.
-
RTX 30 series here Installed 536.40 and immediately noticed that image generation was taking 4 or even more times longer than the previous driver. But the interesting part, was that I also noticed that the GPU simply wasn't being stressed nearly as much either. I have the RGB bar on my GPU set to change colors based on Temperature, and normally my GPU temp sky rockets during image generation. And with the new driver the temps stayed in the low range for the GPU without ever increasing. Rolled back to 536.23 and the performance gain was immediate along with the temps that I was used to seeing. Edit: Only noticed after my post that this was in reference to the driver released even further back. Will roll back even further to see if I get more of a boost than rolling back to 536.23. |
Beta Was this translation helpful? Give feedback.
-
|
Dunno what's going on exactly. I have the latest Nvidia drivers and Comfy absolutely blasts SDXL renders out. Faster than 1.5 on A1111. |
Beta Was this translation helpful? Give feedback.
-
|
I did some of my own tests with 536.* and 531.79. Performance differences were trivial in the short tests I did. Where I think people may be seeing the slowdown is on my rig, using 531.79, SD will not use shared gpu memory and will simply throw an OOM error when dedicated gou memory is exhausted. 536 drivers, shared memory will be used if needed. This allows for much larger gens etc however, once SD bleeds into shared memory, even by a little bit, generation time will suffer severely. People should check their task manager and ensure the jobs are not hitting shared gpu memory if you are on newer drivers before concluding the driver is a problem. Mid gen screenshots to show. First is before HiRes fix, second is after, shared memory is being used and will slow things down. The large spike is the VAE I think. |
Beta Was this translation helpful? Give feedback.
-
|
Some have suggested that the shared memory usage appears at lower required
memory usages than 531. This hasn’t been addressed in drivers yet but is on
their radar.
Depending on how things evolve with SDXL we might just hit a point when low
memory cards aren’t viable anymore.
…On Thu, Jul 20, 2023 at 14:08 gunnerman1 ***@***.***> wrote:
I did some of my own tests with 536.* and 531.79. Performance differences
were trivial in the short tests I did.
Where I think people may be seeing the slowdown is on my rig, using
531.79, SD will not use shared gpu memory and will simply throw an OOM
error when dedicated gou memory is exhausted. 536 drivers, shared memory
will be used if needed. This allows for much larger gens etc however, once
SD bleeds into shared memory, even by a little bit, generation time will
suffer severely. People should check their task manager and ensure the jobs
are not hitting shared gpu memory if you are on newer drivers before
concluding the driver is a problem.
Mid gen screenshots to show. First is before HiRes fix, second is after,
shared memory is being used and will slow things down. The large spike is
the VAE tiling.
[image: image]
<https://user-images.githubusercontent.com/11324979/254983968-629a9be5-378e-4d3d-bc82-df0d700d869e.png>
[image: image]
<https://user-images.githubusercontent.com/11324979/254984491-68a1fd74-1707-40bf-bb7e-68d531c53ef9.png>
—
Reply to this email directly, view it on GitHub
<#11062 (reply in thread)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AOKLG4Z65DFTLTZ73OKRB5LXRF62NANCNFSM6AAAAAAY4S3UA4>
.
You are receiving this because you commented.Message ID:
<AUTOMATIC1111/stable-diffusion-webui/repo-discussions/11062/comments/6502774
@github.com>
|
Beta Was this translation helpful? Give feedback.
-
|
RTX 3060 12GB user here. As soon as i removed I know many people say you should never use |
Beta Was this translation helpful? Give feedback.
-
|
Sure thing! In most cases! |
Beta Was this translation helpful? Give feedback.
-
|
Driver 536.99 acknowledges the issue and says a future update should fix it. "This driver implements a fix for creative application stability issues seen during heavy memory usage. We’ve observed some situations where this fix has resulted in performance degradation when running Stable Diffusion and DaVinci Resolve. This will be addressed in an upcoming driver release. [4172676]" |
Beta Was this translation helpful? Give feedback.
-
|
Has 536.99 been fixed yet? |
Beta Was this translation helpful? Give feedback.
-
|
@wzgrx obviously not yet |
Beta Was this translation helpful? Give feedback.
-
|
They have dropped a mention of Stable Diffusion in known issues in the release notes. |
Beta Was this translation helpful? Give feedback.
-
lol what's a shame Nvidia! |
Beta Was this translation helpful? Give feedback.
-
|
Excerpt from the freshly released 537.13 driver: [DaVinci Resolve] This driver implements a fix for creative application stability issues So it seems they tried to implement a fix for other apps like SD? My GPU just died, but can someone test and see if things improved? Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
No, the previously integrated "fix" is the root of the problem for SD and Davinci etc. and will be adressed in a later version. At least that's how I understand it ^^ |
Beta Was this translation helpful? Give feedback.
-
|
Yeah I think you're right. People tested it on Reddit, and didn't see any improvement over previous releases. :( |
Beta Was this translation helpful? Give feedback.
-
|
Still living with 531.79 till this day, It seems NVIDIA wont even bother since this is on the lower end of GPUs 😑 |
Beta Was this translation helpful? Give feedback.
-
|
which is a pisser because it effectively means I can never upgrade these drivers again or face a broken SD experience....sigh...oh well |
Beta Was this translation helpful? Give feedback.
-
|
why on the "lower end", it's noticeable on 4090s as well, thats not the lower end :D if they say it will be fixed, thats enough for me to believe them till now. |
Beta Was this translation helpful? Give feedback.
-
|
Does anyone test the latest 537.42 (WHQL) ? Is the issue fixed? |
Beta Was this translation helpful? Give feedback.
-
|
Thank you for your feedback and for saving us some time. :) |
Beta Was this translation helpful? Give feedback.
-
|
Update: I just tried to use the latest driver but with Studio version instead. Surprise that, the performance of the studio version is better. Still slower (not much) than v.531 but if you guys want to work with the latest driver, just try studio version instead! |
Beta Was this translation helpful? Give feedback.
-
|
I found this issue when I upgrade from 531 Studio to 537 Studio. My 2060 slowdown from 1-3s/it to as much as 60s/it! |
Beta Was this translation helpful? Give feedback.
-
@KrisadaFantasy did you try with --medvram to see if it helps with 537? |
Beta Was this translation helpful? Give feedback.
-
|
@eniora I'm already on --medvram before update so I'm quite sure it won't help. |
Beta Was this translation helpful? Give feedback.
-
|
I've installed studio 537.58 and have generated 5 images at 680x512 with hires fix at 2.2 with various samplers and a bunch of loras and plug ins (SAG, Anti-burn, Tiled Vae) and all of them have been consistently the same speed and without any noticeable slowdown or difference from 531 so far. GTX4050TI laptop with 4gb of VRAM and 64GB RAM. Will keep on this driver and see how it goes. Would appreciate other opinions, I'm now paranoid and twitchy and utterly beholden to subejctiveness where this issue is concerned :-) |
Beta Was this translation helpful? Give feedback.
-
|
RTX 4070 ti |
Beta Was this translation helpful? Give feedback.
-
UpdateUsers are reporting the issue have been fixed in 537.58 |
Beta Was this translation helpful? Give feedback.
-
|
After testing, 537.58 did not fix this issue. |
Beta Was this translation helpful? Give feedback.
-
|
Studio version was fixed a bit |
Beta Was this translation helpful? Give feedback.
-
|
I tested 545.84 (game ready) which resolved the problem for me. It was not fixed for 537.58 |
Beta Was this translation helpful? Give feedback.
-
|
seem NVIDIA update it, workaround for 546.01 please check https://nvidia.custhelp.com/app/answers/detail/a_id/5490 |
Beta Was this translation helpful? Give feedback.
-
|
Still slower than 531.79. Shame Nvidia |
Beta Was this translation helpful? Give feedback.
-
|
Just tested. We're back baby!! |
Beta Was this translation helpful? Give feedback.
-
|
Amazing, thank you. Would have never found this myself. At least I'm able to run highres fix again. |
Beta Was this translation helpful? Give feedback.
-
|
Didn't work for me :-( I tried the workaround NVIDIA described in the link above (there are 2 python.exe running - one from my system path at c:\python310\python.exe and the other from %SD_HOME%\venv\scripts\python.exe, applied on both) - but still an animatediff run with 4xUltraScale Upscaler just hangs at 50%. Switched to 4x_NMKD_SuperScaler and finally SD went past this point and finished. Using no hires.fix at all also works every time. GTX 4060 TI with 16 GB GPURAM, SD 1.8, Commit Hash: bef51ae and NVIDIA drivers 551.76 on Windows 10 22H2, 32 GB RAM |
Beta Was this translation helpful? Give feedback.
and others
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Update (2023-10-31)
This issue should now be entirely resolved. NVIDIA has made a help article to disable the system memory fallback behavior. Please upgrade to the latest driver (546.01) and follow the guide on their website: https://nvidia.custhelp.com/app/answers/detail/a_id/5490
Update (2023-10-17)
there seems to be some reports saying that the issue is still not fixed
comments
#11063 (comment)
#11063 (comment)
Update (2023-10-14)
Users are reporting the issue have been fixed in 537.58
#11063 (comment)
some users have reported some issues related to the latest (535) Nvidia drivers
nVidia drivers change in memory management vladmandic#1285
#11050 (comment)
if you have been experiencing generation slowdowns or getting stuck, consider downgrading to driver version 531 or below
NVIDIA Driver Downloads
Beta Was this translation helpful? Give feedback.
All reactions