[rocm7.0_internal_testing] skip 3D NCHW FP16 batchnorm test due to Native accuracy issue #2370
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Jenkins build for 2f9e18c5fb255cbd1f554c070bcf6d852ab9b848 commit finished as FAILURE |
dnikolaev-amd
changed the title
pruthvistony
approved these changes
Jul 19, 2025
Author
|
! cherry-pick --onto release/2.7 |
Author
|
! cherry-pick --onto release/2.6 |
okakarpa
pushed a commit
that referenced
this pull request
Jul 21, 2025
…tive accuracy issue (#2370) Skip for `test_nn.py::TestNN.test_batchnorm_3D_train_NCHW_vs_native_mixed_float16` Test failed on `weight gradient` comparison MIOpen/CuDNN vs Native batchnorm. But CPU test `test_batchnorm_3D_train_NCHW_vs_cpu_mixed_float16` passed. It looks like FP16 Native batchnorm issue. Failed on MI200/MI300 and V100 It passed somehow on Navi (with enabled MIOpen) Fixes SWDEV-541024, SWDEV-539171 ``` python test_nn.py -v -k test_batchnorm_3D_train_NCHW_vs_native_mixed_float16 test_batchnorm_3D_train_NCHW_vs_native_mixed_float16 (__main__.TestNN) ... skipped '3D float16 NCHW train failed on CUDA and ROCm due to Native batchnorm accuracy issue SWDEV-541024' OK (skipped=1) ```
Collaborator
|
Created branch autogenerated/release/2.7_cherry-pick_pr-2370 and #2390 |
okakarpa
pushed a commit
that referenced
this pull request
Jul 21, 2025
…tive accuracy issue (#2370) Skip for `test_nn.py::TestNN.test_batchnorm_3D_train_NCHW_vs_native_mixed_float16` Test failed on `weight gradient` comparison MIOpen/CuDNN vs Native batchnorm. But CPU test `test_batchnorm_3D_train_NCHW_vs_cpu_mixed_float16` passed. It looks like FP16 Native batchnorm issue. Failed on MI200/MI300 and V100 It passed somehow on Navi (with enabled MIOpen) Fixes SWDEV-541024, SWDEV-539171 ``` python test_nn.py -v -k test_batchnorm_3D_train_NCHW_vs_native_mixed_float16 test_batchnorm_3D_train_NCHW_vs_native_mixed_float16 (__main__.TestNN) ... skipped '3D float16 NCHW train failed on CUDA and ROCm due to Native batchnorm accuracy issue SWDEV-541024' OK (skipped=1) ```
Collaborator
|
Created branch autogenerated/release/2.6_cherry-pick_pr-2370 and #2391 |
jithunnair-amd
pushed a commit
that referenced
this pull request
Jul 24, 2025
… Native accuracy issue (#2391) Cherry-pick of #2370 Co-authored-by: Dmitry Nikolaev <[email protected]>
jithunnair-amd
pushed a commit
that referenced
this pull request
Jul 24, 2025
… Native accuracy issue (#2390) Cherry-pick of #2370 Co-authored-by: Dmitry Nikolaev <[email protected]>
pruthvistony
pushed a commit
that referenced
this pull request
Aug 2, 2025
#2440) This PR has fixes for P1 Jira https://ontrack-internal.amd.com/browse/SWDEV-542659. In this Jira, there are 3 test files with failing tests. 1) distributed.test_distributed_spawn 2) test_binary_ufuncs 3) test_nn The test files **distributed.test_distributed_spawn** & **test_binary_ufuncs** are passing with latest mainline build- **registry-sc-harbor.amd.com/framework/compute-rocm-dkms-no-npi-hipclang:16426_ubuntu22.04_py3.10_pytorch_lw_release-2.7_fe3d37a9**. The test file **test_nn** has 2 failing tests- **test_batchnorm_3D_train_NCHW_vs_native_mixed_float16** & **test_RNN_dropout_state**. The **test_batchnorm_3D_train_NCHW_vs_native_mixed_float16** test is skipped from PR #2370. The **test_RNN_dropout_state** is fixed by cherry picking upstream commit 1aa971a. Tested on MI200 with docker image- **registry-sc-harbor.amd.com/framework/compute-rocm-dkms-no-npi-hipclang:16426_ubuntu22.04_py3.10_pytorch_lw_release-2.7_fe3d37a9**. --------- Co-authored-by: Iurii Paikov <[email protected]> Co-authored-by: Jeff Daily <[email protected]> Co-authored-by: Nikita Shulga <[email protected]>
dhonnappa-amd
pushed a commit
that referenced
this pull request
Aug 13, 2025
#2440) This PR has fixes for P1 Jira https://ontrack-internal.amd.com/browse/SWDEV-542659. In this Jira, there are 3 test files with failing tests. 1) distributed.test_distributed_spawn 2) test_binary_ufuncs 3) test_nn The test files **distributed.test_distributed_spawn** & **test_binary_ufuncs** are passing with latest mainline build- **registry-sc-harbor.amd.com/framework/compute-rocm-dkms-no-npi-hipclang:16426_ubuntu22.04_py3.10_pytorch_lw_release-2.7_fe3d37a9**. The test file **test_nn** has 2 failing tests- **test_batchnorm_3D_train_NCHW_vs_native_mixed_float16** & **test_RNN_dropout_state**. The **test_batchnorm_3D_train_NCHW_vs_native_mixed_float16** test is skipped from PR #2370. The **test_RNN_dropout_state** is fixed by cherry picking upstream commit 1aa971a. Tested on MI200 with docker image- **registry-sc-harbor.amd.com/framework/compute-rocm-dkms-no-npi-hipclang:16426_ubuntu22.04_py3.10_pytorch_lw_release-2.7_fe3d37a9**. --------- Co-authored-by: Iurii Paikov <[email protected]> Co-authored-by: Jeff Daily <[email protected]> Co-authored-by: Nikita Shulga <[email protected]>
dhonnappa-amd
pushed a commit
that referenced
this pull request
Aug 13, 2025
#2440) This PR has fixes for P1 Jira https://ontrack-internal.amd.com/browse/SWDEV-542659. In this Jira, there are 3 test files with failing tests. 1) distributed.test_distributed_spawn 2) test_binary_ufuncs 3) test_nn The test files **distributed.test_distributed_spawn** & **test_binary_ufuncs** are passing with latest mainline build- **registry-sc-harbor.amd.com/framework/compute-rocm-dkms-no-npi-hipclang:16426_ubuntu22.04_py3.10_pytorch_lw_release-2.7_fe3d37a9**. The test file **test_nn** has 2 failing tests- **test_batchnorm_3D_train_NCHW_vs_native_mixed_float16** & **test_RNN_dropout_state**. The **test_batchnorm_3D_train_NCHW_vs_native_mixed_float16** test is skipped from PR #2370. The **test_RNN_dropout_state** is fixed by cherry picking upstream commit 1aa971a. Tested on MI200 with docker image- **registry-sc-harbor.amd.com/framework/compute-rocm-dkms-no-npi-hipclang:16426_ubuntu22.04_py3.10_pytorch_lw_release-2.7_fe3d37a9**. --------- Co-authored-by: Iurii Paikov <[email protected]> Co-authored-by: Jeff Daily <[email protected]> Co-authored-by: Nikita Shulga <[email protected]>
dhonnappa-amd
pushed a commit
that referenced
this pull request
Aug 13, 2025
#2440) This PR has fixes for P1 Jira https://ontrack-internal.amd.com/browse/SWDEV-542659. In this Jira, there are 3 test files with failing tests. 1) distributed.test_distributed_spawn 2) test_binary_ufuncs 3) test_nn The test files **distributed.test_distributed_spawn** & **test_binary_ufuncs** are passing with latest mainline build- **registry-sc-harbor.amd.com/framework/compute-rocm-dkms-no-npi-hipclang:16426_ubuntu22.04_py3.10_pytorch_lw_release-2.7_fe3d37a9**. The test file **test_nn** has 2 failing tests- **test_batchnorm_3D_train_NCHW_vs_native_mixed_float16** & **test_RNN_dropout_state**. The **test_batchnorm_3D_train_NCHW_vs_native_mixed_float16** test is skipped from PR #2370. The **test_RNN_dropout_state** is fixed by cherry picking upstream commit 1aa971a. Tested on MI200 with docker image- **registry-sc-harbor.amd.com/framework/compute-rocm-dkms-no-npi-hipclang:16426_ubuntu22.04_py3.10_pytorch_lw_release-2.7_fe3d37a9**. --------- Co-authored-by: Iurii Paikov <[email protected]> Co-authored-by: Jeff Daily <[email protected]> Co-authored-by: Nikita Shulga <[email protected]>
Author
|
! cherry-pick --onto release/2.8 |
dhonnappa-amd
pushed a commit
that referenced
this pull request
Sep 18, 2025
…tive accuracy issue (#2370) Skip for `test_nn.py::TestNN.test_batchnorm_3D_train_NCHW_vs_native_mixed_float16` Test failed on `weight gradient` comparison MIOpen/CuDNN vs Native batchnorm. But CPU test `test_batchnorm_3D_train_NCHW_vs_cpu_mixed_float16` passed. It looks like FP16 Native batchnorm issue. Failed on MI200/MI300 and V100 It passed somehow on Navi (with enabled MIOpen) Fixes SWDEV-541024, SWDEV-539171 ``` python test_nn.py -v -k test_batchnorm_3D_train_NCHW_vs_native_mixed_float16 test_batchnorm_3D_train_NCHW_vs_native_mixed_float16 (__main__.TestNN) ... skipped '3D float16 NCHW train failed on CUDA and ROCm due to Native batchnorm accuracy issue SWDEV-541024' OK (skipped=1) ```
|
Created branch autogenerated/release/2.8_cherry-pick_pr-2370 and #2652 Comment processed by Build |
dhonnappa-amd
added a commit
that referenced
this pull request
Sep 18, 2025
Cherry-pick of #2370 Co-authored-by: Dmitry Nikolaev <[email protected]>
Author
|
! cherry-pick --onto release/2.9 |
|
Created branch autogenerated/release/2.9_cherry-pick_pr-2370 and #2788. It contains a merge conflict. Please resolve it Comment processed by Build |
jeffdaily
pushed a commit
that referenced
this pull request
Nov 5, 2025
… Native accuracy issue (#2788) Skip for `test_batchnorm_3D_train_NCHW_vs_native_mixed_float16` Cherry-pick of #2370 ~Need to resolve conflicts~ - resolved --------- Co-authored-by: Dmitry Nikolaev <[email protected]>
jeffdaily
pushed a commit
that referenced
this pull request
Nov 17, 2025
… Native accuracy issue (#2788) Skip for `test_batchnorm_3D_train_NCHW_vs_native_mixed_float16` Cherry-pick of #2370 ~Need to resolve conflicts~ - resolved --------- Co-authored-by: Dmitry Nikolaev <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Skip for
test_nn.py::TestNN.test_batchnorm_3D_train_NCHW_vs_native_mixed_float16Test failed on
weight gradientcomparison MIOpen/CuDNN vs Native batchnorm.But CPU test
test_batchnorm_3D_train_NCHW_vs_cpu_mixed_float16passed.It looks like FP16 Native batchnorm issue.
Failed on MI200/MI300 and V100
It passed somehow on Navi (with enabled MIOpen)
Fixes SWDEV-541024, SWDEV-539171
Cherry-picked to release/2.7 branch via #2390
Cherry-picked to release/2.6 branch via #2391
Cherry-picked to release/2.8 branch via #2652
Cherry-picked to release/2.9 branch via #2788