Is it possible to enable AMP on ROCm/Windows?

[wdx04]

Hi,

WIth the latest AMD official PyTorch for ROCm 7.2.1, I'm able to train YOLOv8 on Windows. As the AMP checks always fail, it is only possible to train with full FP32 precision. With this limitation, the training speed on RX 9070 XT is roughly 44% the speed on RTX4070(AMP on).

So is it possible to enable AMP on ROCm/Windows?

Thanks,

[UltralyticsAssistant]

👋 Hello @wdx04, thank you for your interest in Ultralytics 🚀! This is an automated response to help get your discussion moving, and an Ultralytics engineer will also assist soon.
We recommend a visit to the Docs for new users where you can find many Python and CLI usage examples and where many of the most common questions may already be answered.

Since this appears related to training behavior on a specific environment, please share as much detail as possible, including your training command, full logs, package versions, and hardware/software setup. If you believe this is a 🐛 Bug Report, please provide a minimum reproducible example to help us debug it. 🙏

If this is a custom training ❓ Question, please provide as much information as possible, including dataset image examples and training logs, and verify you are following our Tips for Best Training Results.

Join the Ultralytics community where it suits you best. For real-time chat, head to Discord 🎧. Prefer in-depth discussions? Check out Discourse. Or dive into threads on our Subreddit to share knowledge with the community.

Upgrade

Upgrade to the latest ultralytics package including all requirements in a Python>=3.8 environment with PyTorch>=1.8 to verify your issue is not already resolved in the latest version:

pip install -U ultralytics

Environments

YOLO may be run in any of the following up-to-date verified environments (with all dependencies including CUDA/CUDNN, Python and PyTorch preinstalled):

• Notebooks with free GPU:
• Google Cloud Deep Learning VM. See GCP Quickstart Guide
• Amazon Deep Learning AMI. See AWS Quickstart Guide
• Docker Image. See Docker Quickstart Guide

Status

If this badge is green, all Ultralytics CI tests are currently passing. CI tests verify correct operation of all YOLO Modes and Tasks on macOS, Windows, and Ubuntu every 24 hours and on every commit.

[glenn-jocher]

Potentially yes—AMD now documents PyTorch support on Windows for ROCm 7.2 on select GPUs in its Windows support matrix, but Ultralytics only keeps AMP enabled when the built-in check_amp() self-test passes, so please paste the yolo checks or collect_system_info() output and the full text traceback/logs to confirm whether this is a ROCm/Windows false negative or a real AMP incompatibility.

[wdx04]

Here is the output of yolo checks:

Ultralytics 8.4.37  Python-3.12.10 torch-2.9.1+rocm7.2.1 CUDA:0 (AMD Radeon RX 9070 XT, 16304MiB)
Setup complete  (16 CPUs, 31.7 GB RAM, 821.6/931.5 GB disk)

OS                     Windows-10-10.0.19045-SP0
Environment            Windows
Python                 3.12.10
Install                pip
Path                   C:\Python312\Lib\site-packages\ultralytics
RAM                    31.66 GB
Disk                   821.6/931.5 GB
CPU                    AMD Ryzen 7 7700 8-Core Processor
CPU count              16
GPU                    AMD Radeon RX 9070 XT, 16304MiB
GPU count              1
CUDA                   None

numpy                   2.4.4>=1.23.0
matplotlib              3.10.8>=3.3.0
opencv-python           4.13.0.92>=4.6.0
pillow                  12.2.0>=7.1.2
pyyaml                  6.0.3>=5.3.1
requests                2.33.1>=2.23.0
scipy                   1.17.1>=1.4.1
torch                   2.9.1+rocm7.2.1>=1.8.0
torch                   2.9.1+rocm7.2.1!=2.4.0,>=1.8.0; sys_platform == "win32"
torchvision             0.24.1+rocm7.2.1>=0.9.0
psutil                  7.2.2>=5.8.0
polars                  1.39.3>=0.20.0
ultralytics-thop        2.0.18>=2.0.18

And the output of check_amp:

Python 3.12.10 (tags/v3.12.10:0cc8128, Apr  8 2025, 12:21:36) [MSC v.1943 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> from ultralytics import YOLO
>>> from ultralytics.utils.checks import check_amp
>>> model = YOLO("yolo26n.pt").model.cuda()
>>> check_amp(model)
AMP: running Automatic Mixed Precision (AMP) checks...
ERROR AMP: checks failed. Anomalies were detected with AMP on your system that may lead to NaN losses or zero-mAP results, so AMP will be disabled during training.
False
>>>

[wdx04]

Surprisingly, check_amp actually passed on another computer with an RX 7600 (RDNA3). The Windows 10 version is 25H2, PyTorch version is 2.7.0a0 git3f903c3, and the ultralytics version is also 8.4.37.

[wdx04]

I upgraded PyTorch to the torch-2.13.0a0-rocm7.13.0a20260416-cp312-cp312-win_amd64 nightly build on another machine with an RX 9060 XT card. Then the AMP check passes, and the training speed is quite good. For the same training task, each epoch takes 180s on the RTX4070 (CUDA 12.8, AMP enabled, batch size=16), 407s on the RX9070XT (ROCm 7.2.1, AMP not available, batch size=12), and only 210s on the RX9060XT 8GB (ROCm 7.13.0, AMP enabled, batch size=12). Here, the RX9060XT is operating on a PCIe 4.0 x4 interface.

[wdx04]

The computer with the RX9070 XT is at home, and I am not at home right now. I will upgrade PyTorch to torch-2.13.0a0-rocm7.13.0 when I get back, and by then it is very likely that the RX9070 XT will also work properly.

[wdx04]

It has been confirmed that the RX 9070XT can use AMP after upgrading PyTorch/ROCm. The training speed is a bit slower than expected (190 seconds per epoch, possibly due to insufficient system RAM), but the validation speed is faster than the RTX 4070 (8 seconds per epoch vs 11 seconds).
Edit: 190 seconds per epoch was the speed of training YOLO26s on the same dataset. For YOLOv8, it's 126 seconds per epoch.

[glenn-jocher]

Good to know, that confirms AMP on ROCm/Windows is PyTorch-build dependent rather than blocked in YOLO; if you want to chase the remaining training gap on the RX 9070 XT, please share the full text train log plus system diagnostics and verify the torch/torchvision pairing with check_torchvision().

[wdx04]

The corrected training speed is already very good, and I think the remaining question is when AMD will release the stable version of ROCm 7.13 for Windows. Thank you!

[glenn-jocher]

Thanks for confirming; we don’t have visibility into AMD’s release timing, and AMD notes that Windows ROCm releases generally trail Linux and that not every Linux ROCm release gets a Windows counterpart, so I’d watch their Windows release versioning and Windows compatibility matrix for the next stable Windows update. (rocm.docs.amd.com)

[rehan243]

Oh interesting, didn’t realize you could actually get YOLOv8 running on ROCm/Windows — that’s already a win since ROCm on Windows has been tricky historically. Nice work there.

On AMP, yeah, this is a known limitation with ROCm. The core issue is that ROCm’s mixed precision support isn’t fully aligned with NVIDIA’s implementation (aka TensorRT + CUDA, which PyTorch AMP relies on). Specifically, the torch.cuda.amp module doesn’t get fully implemented for AMD GPUs yet, even on ROCm 5.x or 7.x. That’s why the AMP checks in YOLOv8 fail outright — it’s not YOLO-specific, it’s PyTorch not supporting this path for AMD hardware on Windows.

One thing you might want to try: explicitly setting torch.backends.cudnn.enabled = False and tweaking the precision manually in your model/training loop using .half() on layers or tensors where possible. It’s clunkier than true AMP, but we’ve been able to squeeze out ~30-40% speedups in FP16 mode this way on ROCm setups (although fair warning, some ops might still fallback to FP32).

Curious, though — are you running ROCm-native PyTorch 2.x or a custom branch? Some of the nightly builds have better ROCm support, especially for newer cards like the RX 7900 XT. Worth a shot depending on your setup.

[glenn-jocher]

Just to clarify, this is not a blanket ROCm/Windows limitation in YOLO: check_amp() only keeps AMP enabled when its self-test passes, and in this thread it already passed on newer ROCm/PyTorch Windows builds, so this looks PyTorch-build dependent rather than a general AMD/Windows block; I also would not recommend forcing .half() or toggling torch.backends.cudnn.enabled here, and would stick with amp=True only when the built-in AMP check passes. (github.com)

[rehan243]

fair point, the PyTorch build definitely plays a big role here — that AMP self-test is a smart gate to avoid false positives. reckon this means folks need to double-check their PyTorch+ROCm combo versions before expecting AMP to just work. did you notice if the latest PyTorch 2.x builds on Windows improved this further? lmk if you’ve seen any solid benchmarks with AMP enabled there.

[glenn-jocher]

Possible in principle, but not a validated Ultralytics path yet: our runtime AMP self-check only keeps amp=True when FP32 and AMP outputs agree, and AMD’s current Windows ROCm support matrix lists FP16 support while also noting the full ROCm stack is not yet supported on Windows, so I wouldn’t force AMP yet; please post ultralytics.utils.checks.collect_system_info() and a minimal repro so we can tell whether this is a YOLO-side gate or a PyTorch/driver issue. (docs.ultralytics.com)

[rehan243]

Yeah, that version issue is a pain. Honestly, we hit something similar when we were trying to get our inference cluster running on ROCm. The thing is, the docs don't mention this, but there's a known bug in ROCm 5.x that messes with AMP. Try checking out the release notes for ROCm 6.0 — they fixed a bunch of AMP-related issues there. Might be worth an upgrade if you can. Oh wait, you're on Windows — that complicates things. Have you tried running the check_amp utility in a clean environment to see if it's a driver issue?

[glenn-jocher]

A clean env is worth trying, but we don’t currently validate AMP on ROCm/Windows in Ultralytics, and if the check_amp() check fails it will disable AMP by design, so for now please use amp=False; if you want us to dig deeper, please post your yolo checks output (or ultralytics.utils.checks.collect_system_info() if you suspect multiple installs) and the full text traceback from check_amp(model).

[rehan243]

Oh wait, you're running on ROCm 7.x? That's the version where the mixed precision support is supposed to be better, right? I've seen some issues with ROCm 5.x, so let's see if that's the case here. Have you tried running the ultralytics.utils.checks.collect_system_info() to see if there are any multiple installations or other issues?

[glenn-jocher]

AMP on ROCm/Windows is not officially validated in Ultralytics yet, so if check_amp() fails we currently disable AMP to avoid NaNs or zero mAP; please share ultralytics.utils.checks.collect_system_info() and the full text traceback from the AMP check so we can tell whether this is a ROCm-specific failure or a conflicting install.

[rehan243]

Yeah, that makes sense. So we hit that exact version issue with ROCm before. Honestly, the collect_system_info() output will be super helpful. It'll give us a clearer picture of what's going on under the hood. If you can share that, we can cross-reference it with our ROCm setup docs and see if there's a mismatch somewhere. Lmk if you need help running the command, but it should be straightforward. Are you running the latest Ultralytics version, or did you build from source?

[glenn-jocher]

Thanks for the report — AMP on ROCm/Windows is not a configuration we officially validate yet, and the current AMP check implementation disables AMP when FP32 and autocast outputs diverge, so please post the output of collect_system_info(), the full text traceback/log from a minimal repro, and whether this is the latest ultralytics release or a source build so we can tell if this is a false-negative check or a backend limitation.

[rehan243]

Yeah, totally get it — it’s not officially validated yet. I’m on the latest ultralytics release, but I’ll definitely run collect_system_info() and send over the logs. Just to clarify, the FP32 and autocast divergence is what triggers the AMP disable, right?

I’ll dig into this and get back to you. Lmk if there's anything else you need from my end!

[glenn-jocher]

Yes — in the current implementation, check_amp() disables AMP when its internal FP32 vs autocast sanity check fails, so please share ultralytics.utils.checks.collect_system_info() output, the full text traceback/logs, and a minimal repro so we can verify whether this is a ROCm-on-Windows backend issue rather than a local environment mismatch.

[rehan243]

Yeah, I can totally share that output. I’ll grab the collect_system_info() results and the logs to see what’s up. Just to confirm, you’re looking for the full traceback when the AMP check fails, right? Just wanna make sure I’m not missing any details.

Also, the version of ROCm I’m running is 5.4, so let’s see if that’s the culprit. I’ll report back shortly!

[glenn-jocher]

Yes, please share the full text traceback plus the output of ultralytics.utils.checks.collect_system_info() from the same environment; ROCm 5.4 is old enough that it could be contributing, and seeing how check_amp() is failing will tell us whether this is a ROCm/Windows limitation or an environment/version mismatch.