Help : torch._C._cuda_emptyCache() RuntimeError: CUDA error: the launch timed out and was terminated

[JeNeSuisPasDeveloppeur]

Hello,

It's my first time posting here, I am not quite sure I am posting in the correct place, I'm sorry.

I've been using AUTOMATIC1111 for a few weeks now without any major problems. I only have 4Gb of dedicated GPU memory, but I managed to create several hundreds of 850x850 to 1000x1000 images, until a couple of days ago.

Now, every time I run anything above 200x200, I got this error :
File "C:\Program Files (x86)\Stable Diffusion\stable-diffusion-webui\venv\lib\site-packages\torch\cuda\memory.py", line 133, in empty_cache torch._C._cuda_emptyCache() RuntimeError: CUDA error: the launch timed out and was terminated CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.

I have tried many things, Nvidia drivers, Cuda toolkit, uninstall & reinstall the entire venv folder, and try to add various lines in the webui-user.bat file. I've read every related topic here and elsewhere, and I couldn't find a solution to this.

I have many extensions, but the problem appeared long after installing and using all of them. Nothing changed really, between the moment it was working and the moment it was not.

Can anybody help me with this ? Thank you

Here is a little more info on the packages :
"Packages": [ "absl-py==1.4.0", "accelerate==0.21.0", "addict==2.4.0", "aenum==3.1.15", "aiofiles==23.2.1", "aiohttp==3.8.5", "aiosignal==1.3.1", "altair==5.1.1", "antlr4-python3-runtime==4.9.3", "anyio==3.7.1", "async-timeout==4.0.3", "attrs==23.1.0", "basicsr==1.4.2", "beautifulsoup4==4.12.2", "bidict==0.22.1", "blendmodes==2022", "boltons==23.0.0", "cachetools==5.3.1", "certifi==2023.7.22", "cffi==1.15.1", "charset-normalizer==3.2.0", "clean-fid==0.1.35", "click==8.1.7", "clip==1.0", "colorama==0.4.6", "contourpy==1.1.0", "controlnet-aux==0.0.3", "cssselect2==0.7.0", "cycler==0.11.0", "deprecation==2.1.0", "diffusers==0.16.1", "einops==0.4.1", "exceptiongroup==1.1.3", "facexlib==0.3.0", "fastapi==0.94.0", "ffmpy==0.3.1", "filelock==3.12.3", "filterpy==1.4.5", "flask-cors==4.0.0", "flask-socketio==5.3.6", "flask==2.2.3", "flaskwebgui==0.3.5", "flatbuffers==23.5.26", "fonttools==4.42.1", "frozenlist==1.4.0", "fsspec==2023.9.0", "ftfy==6.1.1", "future==0.18.3", "fvcore==0.1.5.post20221221", "gdown==4.7.1", "gfpgan==1.3.8", "gitdb==4.0.10", "gitpython==3.1.32", "google-auth-oauthlib==1.0.0", "google-auth==2.22.0", "gradio-client==0.5.0", "gradio==3.41.2", "greenlet==2.0.2", "grpcio==1.57.0", "h11==0.12.0", "httpcore==0.15.0", "httpx==0.24.1", "huggingface-hub==0.16.4", "idna==3.4", "imageio==2.31.3", "importlib-metadata==6.8.0", "importlib-resources==6.0.1", "inflection==0.5.1", "iopath==0.1.9", "itsdangerous==2.1.2", "jinja2==3.1.2", "jsonmerge==1.8.0", "jsonschema-specifications==2023.7.1", "jsonschema==4.19.0", "kiwisolver==1.4.5", "kornia==0.6.7", "lama-cleaner==1.2.3", "lark==1.1.2", "lazy-loader==0.3", "lightning-utilities==0.9.0", "llvmlite==0.40.1", "lmdb==1.4.1", "loguru==0.7.2", "lpips==0.1.4", "lxml==4.9.3", "markdown-it-py==3.0.0", "markdown==3.4.4", "markupsafe==2.1.3", "matplotlib==3.7.2", "mdurl==0.1.2", "mediapipe==0.10.3", "mpmath==1.3.0", "multidict==6.0.4", "networkx==3.1", "numba==0.57.1", "numpy==1.23.5", "oauthlib==3.2.2", "omegaconf==2.2.3", "open-clip-torch==2.20.0", "opencv-contrib-python==4.8.0.76", "opencv-python-headless==4.8.0.76", "opencv-python==4.8.0.76", "orjson==3.9.5", "packaging==23.1", "pandas==2.1.0", "piexif==1.1.3", "pillow==9.5.0", "pip==22.2.1", "platformdirs==3.10.0", "portalocker==2.7.0", "protobuf==3.20.0", "psutil==5.9.5", "py-cpuinfo==9.0.0", "pyasn1-modules==0.3.0", "pyasn1==0.5.0", "pycparser==2.21", "pydantic==1.10.12", "pydub==0.25.1", "pygments==2.16.1", "pyparsing==3.0.9", "pysocks==1.7.1", "python-dateutil==2.8.2", "python-engineio==4.7.1", "python-multipart==0.0.6", "python-socketio==5.9.0", "pytorch-lightning==1.9.4", "pytz==2023.3.post1", "pywavelets==1.4.1", "pywin32==306", "pyyaml==6.0.1", "realesrgan==0.3.0", "referencing==0.30.2", "regex==2023.8.8", "reportlab==4.0.4", "requests-oauthlib==1.3.1", "requests==2.31.0", "resize-right==0.0.2", "rich==13.5.2", "rpds-py==0.10.2", "rsa==4.9", "safetensors==0.3.1", "scikit-image==0.21.0", "scipy==1.11.2", "seaborn==0.12.2", "segment-anything==1.0", "semantic-version==2.10.0", "sentencepiece==0.1.99", "setuptools==63.2.0", "simple-websocket==0.10.1", "six==1.16.0", "smmap==5.0.0", "sniffio==1.3.0", "sounddevice==0.4.6", "soupsieve==2.5", "sqlalchemy==2.0.20", "starlette==0.26.1", "supervision==0.14.0", "svglib==1.5.1", "sympy==1.12", "tabulate==0.9.0", "tb-nightly==2.15.0a20230902", "tensorboard-data-server==0.7.1", "termcolor==2.3.0", "tifffile==2023.8.30", "timm==0.9.2", "tinycss2==1.2.1", "tokenizers==0.13.3", "tomesd==0.1.3", "tomli==2.0.1", "toolz==0.12.0", "torch==2.0.1+cu118", "torchdiffeq==0.2.3", "torchmetrics==1.1.1", "torchsde==0.2.5", "torchvision==0.15.2+cu118", "tqdm==4.66.1", "trampoline==0.1.2", "transformers==4.30.2", "typing-extensions==4.7.1", "tzdata==2023.3", "ultralytics==8.0.172", "urllib3==1.26.16", "uvicorn==0.23.2", "wcwidth==0.2.6", "webencodings==0.5.1", "websockets==11.0.3", "werkzeug==2.3.7", "wheel==0.41.2", "whichcraft==0.6.1", "win32-setctime==1.1.0", "wsproto==1.2.0", "yacs==0.1.8", "yapf==0.40.1", "yarl==1.9.2", "zipp==3.16.2"

[jskye]

i just did a basic install and it worked for one small generation.
then i tried to do another one and im getting this same cache error now.
seems like cuda isnt releasing the cache it uses after every gen.
can manually release cache using garbage collector in python, so may need to do that in the script?
i tried doing it just in a python shell and it didnt fix the error so it must be the cache only for cuda/pytorch or the local project.
not sure why it happened only after one generation on my machine..