09-persistent-matmul hangs on B580

[pbchekin]

The step "Run tutorials" is running for 9h in the last nightly test on B580:
https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/14352243070/job/40233474592.

[pbchekin]

@etiotto could you please take a look?

[etiotto]

In this B580 run (https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/14328476769/job/40158793146) the tutorial took about 30mins. The latest commit in that run was 25f5666, which did contains the PR #3858 which enabled tutorial 09-persistent-matmul. Am I reading this correctly @pbchekin ?

The B580 run after that last good one, contains the following additional commits. So one of these commits could be the culprit (or something changed on the machine)?

commit 3bcb78a (origin/amyachev/merge)
Author: Anatoly Myachev anatoly.myachev@intel.com
Date: Tue Apr 8 18:09:55 2025 +0000

[intel] Update LTS skiplist

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>

commit 7cf4142
Merge: 6f743d1 6f0ae97
Author: Anatoly Myachev anatoly.myachev@intel.com
Date: Tue Apr 8 10:17:58 2025 +0000

Merge commit '6f0ae970635ef9d2dd3acc88375fc2c70a8a89cb'

commit 6f743d1
Author: Anatoly Myachev anatoliimyachev@mail.com
Date: Tue Apr 8 18:52:24 2025 +0200

Fix `test_matmul.py::test_lhs_in_tmem` (#3866)

Signed-off-by: Anatoly Myachev <anatoly.myachev@intel.com>
Co-authored-by: Whitney Tsang <whitney.tsang@intel.com>

commit 91ef068
Author: Whitney Tsang whitney.tsang@intel.com
Date: Tue Apr 8 09:07:59 2025 -0400

[FA] Port remaining performance features from advanced path (#3848)

`set_fast_math` didn't work on default path, as lowering pass generates
LLVM operations. This PR changed setting of fastmath flag on LLVM IR.
This PR guards the setting of fastmath flag under an env var to avoid
accuracy failures.
![Screenshot 2025-04-06
220833](https://github.com/user-attachments/assets/6d53900a-28f2-4294-8e9a-722284b4ac4b)
Observations:
1. performance of advanced path without setting fastmath flag is the
same as default path
2. performance of default path with setting fastmath flag is faster than
advanced path

As default path (with env var) is able to achieve performance no worse
than advanced path, this PR stops running FA with advanced path.

Closes #3286

---------

Signed-off-by: Whitney Tsang <whitney.tsang@intel.com>

[pbchekin]

In this B580 run (https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/14328476769/job/40158793146) the tutorial took about 30mins. The latest commit in that run was 25f5666, which did contains the PR #3858 which enabled tutorial 09-persistent-matmul. Am I reading this correctly @pbchekin ?

Correct, this run contains 09-persistent-matmul, but it was aborted:

2025-04-08T09:44:18.3121177Z terminate called after throwing an instance of 'sycl::_V1::exception'
2025-04-08T09:44:18.3123503Z   what():  UR backend failed. UR backend returns:40 (UR_RESULT_ERROR_OUT_OF_RESOURCES)
2025-04-08T09:44:19.0363594Z /runner/_work/intel-xpu-backend-for-triton/intel-xpu-backend-for-triton/scripts/pytest-utils.sh: line 56: 134868 Aborted                 (core dumped) $RUN_TUTORIAL

[etiotto]

In this B580 run (https://github.com/intel/intel-xpu-backend-for-triton/actions/runs/14328476769/job/40158793146) the tutorial took about 30mins. The latest commit in that run was 25f5666, which did contains the PR #3858 which enabled tutorial 09-persistent-matmul. Am I reading this correctly @pbchekin ?

Correct, this run contains 09-persistent-matmul, but it was aborted:

2025-04-08T09:44:18.3121177Z terminate called after throwing an instance of 'sycl::_V1::exception'
2025-04-08T09:44:18.3123503Z   what():  UR backend failed. UR backend returns:40 (UR_RESULT_ERROR_OUT_OF_RESOURCES)
2025-04-08T09:44:19.0363594Z /runner/_work/intel-xpu-backend-for-triton/intel-xpu-backend-for-triton/scripts/pytest-utils.sh: line 56: 134868 Aborted                 (core dumped) $RUN_TUTORIAL

I see. So we have a problem in CI as well because when a benchmark aborts we should fail the tutorial step @pbchekin. I'll need to investigate the reason for the exception.

[pbchekin]

So we have a problem in CI as well because when a benchmark aborts we should fail the tutorial step

Yes, we have a problem is CI because we have to run a workflow to ignore errors to get a pass rate for client GPUs: most of them still have failing tests. I will reconfigure recurrent jobs for BMG, Linux to fail on errors. However, this will not solve the existing issue with failing/hanging 09-persistent-matmul. I will make a PR to skip 09-persistent-matmul on BMG.