How can I train with a 7900xt/ xtx?

[Anothergazz]

I saw that zluda is being integrated, so thought I would give it a go. Unfortunately it does not work when I start training, I get the following. Any ideas ?

activating venv C:\AI\OneTrainer\OneTrainer\venv
C:\AI\OneTrainer\OneTrainer\venv\lib\site-packages\transformers\utils\generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
_torch_pytree._register_pytree_node(
C:\AI\OneTrainer\OneTrainer\venv\lib\site-packages\transformers\utils\generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
_torch_pytree._register_pytree_node(
ZLUDA device failed to pass basic operation test: index=None, device_name=AMD Radeon RX 7900 XTX [ZLUDA]
Clearing cache directory C:/AI/OneTrainer/OneTrainer/training_samples/JL/Cache! You can disable this if you want to continue using the same cache.
TensorFlow installation not found - running with reduced feature set.
Some weights of the model checkpoint were not used when initializing CLIPTextModel:
['text_model.embeddings.position_ids']
Some weights of the model checkpoint were not used when initializing CLIPTextModelWithProjection:
['text_model.embeddings.position_ids']
Serving TensorBoard on localhost; to expose to the network, use a proxy or pass --bind_all
TensorBoard 2.15.1 at http://localhost:6006/ (Press CTRL+C to quit)
Traceback (most recent call last):
File "C:\AI\OneTrainer\OneTrainer\modules\ui\TrainUI.py", line 480, in __training_thread_function
trainer.start()
File "C:\AI\OneTrainer\OneTrainer\modules\trainer\GenericTrainer.py", line 123, in start
self.model_setup.setup_model(self.model, self.config)
File "C:\AI\OneTrainer\OneTrainer\modules\modelSetup\StableDiffusionXLLoRASetup.py", line 127, in setup_model
model.optimizer = create.create_optimizer(
File "C:\AI\OneTrainer\OneTrainer\modules\util\create.py", line 327, in create_optimizer
optimizer = torch.optim.AdamW(
File "C:\AI\OneTrainer\OneTrainer\venv\lib\site-packages\torch\optim\adamw.py", line 68, in init
raise RuntimeError("fused=True requires all the params to be floating point Tensors of "
RuntimeError: fused=True requires all the params to be floating point Tensors of supported devices: ['cuda', 'xpu', 'privateuseone'].

[Calamdor]

I am unsure ZLUDA can used AdamW in fused mode. You can try turning fused off in the optimizer settings (... button next to the optimizer name) and then try again.

[Anothergazz]

Thanks for the suggestion (especially telling me were the button is) and it did help, but it now seems to be stuck. After 20 mins it has not moved on with the second 'enumerating sample paths" (the first part was like lightening) and now the machine is just sitting here with only 8% CPU utilisation.

activating venv C:\AI\OneTrainer\OneTrainer\venv
C:\AI\OneTrainer\OneTrainer\venv\lib\site-packages\transformers\utils\generic.py:441: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
_torch_pytree._register_pytree_node(
C:\AI\OneTrainer\OneTrainer\venv\lib\site-packages\transformers\utils\generic.py:309: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
_torch_pytree._register_pytree_node(
ZLUDA device failed to pass basic operation test: index=None, device_name=AMD Radeon RX 7900 XTX [ZLUDA]
Clearing cache directory C:/AI/OneTrainer/Workspace/JL/Cache! You can disable this if you want to continue using the same cache.
No backup found, continuing without backup...
TensorFlow installation not found - running with reduced feature set.
Some weights of the model checkpoint were not used when initializing CLIPTextModel:
['text_model.embeddings.position_ids']
Some weights of the model checkpoint were not used when initializing CLIPTextModelWithProjection:
['text_model.embeddings.position_ids']
Serving TensorBoard on localhost; to expose to the network, use a proxy or pass --bind_all
TensorBoard 2.15.1 at http://localhost:6006/ (Press CTRL+C to quit)
Could not enable memory efficient attention. Make sure xformers is installed correctly and a GPU is available: CUDA error: named symbol not found
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.

enumerating sample paths: 100%|██████████████████████████████████████████████████████████████████| 1/1 [00:00<?, ?it/s]
enumerating sample paths: 0%| | 0/1 [00:00<?, ?it/s]
caching: 0%|

[Calamdor]

I believe also that ZLUDA needs to use SDP and not xformers.

[Anothergazz]

OK I changed the drop down in training>Attention to SDP, but it is exactly the same.
I notice that there is a message early on
"ZLUDA device failed to pass basic operation test: index=None, device_name=AMD Radeon RX 7900 XTX [ZLUDA]"
Maybe that is a clue

[Anothergazz]

No I have nothing else using ZLUDA. I have been using Stability Matrix which only uses WindowsML so there is no support for OneTrainer on AMD. I may well have not installed it correctly. I have not found a clear documented instal process, The ZLUDA page says very little and there are specific examples for programs other than OneTrainer, or this
[https://github.com/vosen/ZLUDA/discussions/82)

Not certain that works with the star-ui.bat file I have been using for oneTrainer up till now

[Heasterian]

AI projects are not using main ZLUDA repo as it lacks some features, it's fork. You need to have HIP SDK installed to make it work. https://www.amd.com/en/developer/resources/rocm-hub/hip-sdk.html I'll try write about it later in readme.

[Anothergazz]

Thanks for your time.
I thought I had already installed HIP SDK, still I downloaded it again and installed it. I got the latest version of ZLUDA V3.7 https://github.com/lshqqytiger/ZLUDA/releases unzipped to C:\AI\zluda. I had already added to PATH in the ENV variables %HIP_PATH%bin and C:\AI\zluda
and it still seems to stop at the same point as described above
What am I missing?

[Heasterian]

I would give a try with installing OneTrainer in new location from scratch. It can be something like broken venv or changed branch.

[Anothergazz]

I deleted the old version of OneTrainer and installed a fresh copy in a different location. Exactly the same result. I am going to reset/re-instal this copy of windows and start from scratch. Unless you have any other ideas

[Anothergazz]

Thanks for your help. Lets hope some else has got it working

[Heasterian]

If you're willing to test some more solutions, we probably should ping author of fork of zluda used by project.

[Anothergazz]

Thanks for the suggestion but until something official gets posted. I dont want zluda to take over my life :)

[hansolocambo]

That's when you realize that buying Nvidia since day one, especially for anything related to (pro) 3D and now AI is a not a bad idea.
There's no plan B to Nvidia, 0 competitor for hardware innovations focused solely on professional artists using a PC.
Intel+Nvidia and you'll never have issues anymore.

[Heasterian]

There are valid reasons to chose any of vendors. None of them is connected to this issue, so please keep this discussion out of it.

[Anothergazz]

Just to re-enforce Haesterian. I bought my 7900xtx to play games as it was considerable cheaper than the Nvidia alternative, some time before I discovered AI. Im not a professional just an interested engineer.

[tnginako]

Imagine buying Intel chips

[xcrashdayx]

@Anothergazz if you still need help with onetrainer on amd cards let me know! i got it working on a 6800xt on windows!

[kirian555]

@Anothergazz if you still need help with onetrainer on amd cards let me know! i got it working on a 6800xt on windows!
Hi, I need help to make it work with a 7800XT, how did you do it?

[Anothergazz]

@kirian555 Thanks for the offer. I spent ages battling with the shifting layers of AMD SW, In the end I gave in and bought an RTX4090, (The price recently dropped and my AMD has held its)

[5P33DY1]

I have a similar card. How did you get OneTrainer working on Windows?

[Calamdor]

https://github.com/Nerogar/OneTrainer/wiki/Lessons-Learnt-and-Tutorials#zluda-notes-by-calamdor

or, as the 7900XTX is one of the few AMD cards that works with RocM and WSL, try that.

https://rocm.docs.amd.com/projects/radeon/en/latest/docs/install/wsl/install-radeon.html

[HillQi]

@Anothergazz if you still need help with onetrainer on amd cards let me know! i got it working on a 6800xt on windows!

Can you please brief your steps to made it. I am struggling on my 7900xtx. Thanks.

[Anothergazz]

@Anothergazz if you still need help with onetrainer on amd cards let me know! i got it working on a 6800xt on windows!

Can you please brief your steps to made it. I am struggling on my 7900xtx. Thanks.

The short answer is I gave up a bought an Nvidia card.
Long answer
I found that it kept breaking, (This is not just a onetrainer thing) typically an updated broke something, and I then wasted time trying to identify what is happening and then waiting for the fix. Most users seem to be Linux based which meant windows fixes took even longer. Hopefully all the changes have settled down. A few month ago the 7900xtx price went up and a I sold it for the same price I paid for it and at the same time RTX4090 prices dropped (still a silly price), so the gap was doable for me. I see RTX4090 prices have gone up by 40% since I bought mine, ouch.

[HillQi]

@Anothergazz if you still need help with onetrainer on amd cards let me know! i got it working on a 6800xt on windows!

Can you please brief your steps to made it. I am struggling on my 7900xtx. Thanks.

The short answer is I gave up a bought an Nvidia card. Long answer I found that it kept breaking, (This is not just a onetrainer thing) typically an updated broke something, and I then wasted time trying to identify what is happening and then waiting for the fix. Most users seem to be Linux based which meant windows fixes took even longer. Hopefully all the changes have settled down. A few month ago the 7900xtx price went up and a I sold it for the same price I paid for it and at the same time RTX4090 prices dropped (still a silly price), so the gap was doable for me. I see RTX4090 prices have gone up by 40% since I bought mine, ouch.

........ maybe you got the best solution. The VERSION Magic going crazy! Compatible between versions has gone for so many years. In the past new version means bug fix and improvement, but today it means brand new system!

To those who may fighting for your AMD & windows, please check my share: https://pan.quark.cn/s/ab761fea7314. This is comfyUI portable pack and works for me out of box. ROCm 5.7 + Zluda + torch 2.3.1 + cud118.

This pack in Chinese, please fix it yourself.

[XDM-Inc]

trying to get this working on amd is a NIGHTMARE! i have been studying for 2 months and STILL nothing but error codes and jsut freezing with no error output

[Heasterian]

On Linux, you should use ROCm, not Zluda. PyTorch 2.5 did have some really high VRAM usage spikes on ROCm. Try to upgrade OneTrainer to make sure you're using PyTorch 2.6.

[Heasterian]

Zluda on Linux afaik require recompiled PyTorch. If OneTrainer is trying to use Zluda on Linux it means you have Cuda Pytorch instead of ROCm for some reason.

[XDM-Inc]

Zluda on Linux afaik require recompiled PyTorch. If OneTrainer is trying to use Zluda on Linux it means you have Cuda Pytorch instead of ROCm for some reason.

That's what I was thinking, I'm thinking it's using zluda by default instead of rocm but I think that's how onetrainer is set up by default. Which would further prove the claim of why I'm running out of vram so quickly zluda is very heavy on the vram

[Calamdor]

did you use the requirements-rocm.txt? Only Heasterian really has AMD, so it is not something that comes up much.

the install.sh should check for this if it was used:

function get_platform_requirements_path {
    # NOTE: The user can override our platform detection via the environment.
    local platform_reqs="${OT_PLATFORM_REQUIREMENTS}"
    if [[ "${platform_reqs}" == "detect" ]]; then
        # NOTE: We MUST prioritize NVIDIA first, since machines that contain
        # *both* AMD and NVIDIA GPUs are usually running integrated AMD graphics
        # that's built into their CPU, whereas their *dedicated* GPU is NVIDIA.
        if [[ -e "/dev/nvidia0" ]] || can_exec nvidia-smi || can_exec "/usr/lib/wsl/lib/nvidia-smi"; then
            # NVIDIA graphics.
            #  "/dev/nvidia0": The "first" detected NVIDIA GPU in the system.
            #  "nvidia-smi": Driver tool for all NVIDIA GPUs made after 2010.
            #  "nvcc": CUDA SDK compiler. Not included in the drivers.
            #  "/usr/lib/wsl/lib/nvidia-smi": WSL's NVIDIA path (isn't in $PATH).
            # SEE: https://docs.nvidia.com/cuda/wsl-user-guide/
            platform_reqs="requirements-cuda.txt"
        elif [[ -e "/dev/kfd" ]]; then
            # AMD graphics.
            platform_reqs="requirements-rocm.txt"
        else
            # No GPU acceleration.
            platform_reqs="requirements-default.txt"
        fi
    fi

[XDM-Inc]

yea in the setup it is using rocm

[OneTrainer] + python -m pip install --upgrade --upgrade-strategy eager -r requirements-global.txt -r requirements-rocm.txt
Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/rocm6.2
Obtaining diffusers from git+https://github.com/huggingface/diffusers.git@c944f06#egg=diffusers (from -r requirements-global.txt (line 19))
  Cloning https://github.com/huggingface/diffusers.git (to revision c944f06) to ./venv/src/diffusers

[XDM-Inc]

i dont under stand why it cant manage the vram correctly?

torch.OutOfMemoryError: HIP out of memory. Tried to allocate 1.25 GiB. GPU 0 has a total capacity of 23.98 GiB of which 1.27 GiB is free. Of the allocated memory 19.80 GiB is allocated by PyTorch, and 2.13 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_HIP_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
it doesn't clear and allocate the vram

[O-J1]

The error isn't indicating what you're suggesting—please avoid making authoritative statements if you're not completely sure. The AMD driver is problematic, and you'll have to accept that.

[XDM-Inc]

Update: ADAFACTOR is making a MASSIVE difference! i was able to do 1 epoch and counting so far! avg vram usage 19-20gb.all i gotta do is speed it up with some learning rates

[Heasterian]

Depending on model, training with full precision (FP32) is worth a try. On my RX6900 XT SDP is now disabling flash attention by default with mixed precision so FP32 is faster than it without fiddling with code.

Also, AdamW with fused backpass did have for some reason lower VRAM usage than Adafactor (but it was with Pytorch 2.5 so it can be different now).