failing tests on Arm for SciPy-bundle v2025.07 w/ `lfbf/2025b` toolchain

[boegel]

follow-up for #24482

[boegel]

@Thyre Can you perhaps looks into making a summary of the issues we've observed in #24482, including error messages, details on failing tests, and relevant upstream issues, so this issue is easy to find?

[Thyre]

In #24482, we did testing with quite a few architectures to battle-test the new LLVM toolchain.
While testing with Neoverse V2, I observed test failures which changed based on the BLAS implementation used in FlexiBLAS.

With OpenBLAS (our default in EasyBuild), tests were failing with NumPy (see #24482 (comment)):

=================================================================================================================== short test summary info ===================================================================================================================
FAILED numpy/linalg/tests/test_linalg.py::TestDet::test_types[complex64] - RuntimeWarning: divide by zero encountered in det
FAILED numpy/linalg/tests/test_linalg.py::TestDet::test_types[complex128] - RuntimeWarning: divide by zero encountered in det
============================================================================= 2 failed, 43510 passed, 329 skipped, 2819 deselected, 33 xfailed, 1539 xpassed in 251.66s (0:04:11) =============================================================================

These failures look very similar to failures they have seen with macOS before (see numpy/numpy#20023 (comment)). Changing the BLAS backend to BLIS lets one work around this failure, so this may be an OpenBLAS issue.

Testing with BLIS, we fail a SciPy test (see #24482 (comment)):

=================================================================================================================== short test summary info ===================================================================================================================
FAILED scipy/sparse/linalg/_isolve/tests/test_iterative.py::test_x0_working[tfqmr] - assert np.float64(9.05172644450491e-06) <= (6e-06 * np.float64(1.240298651028214))
============================================================================== 1 failed, 78509 passed, 5188 skipped, 305 xfailed, 15 xpassed, 2943 warnings in 550.22s (0:09:10) ==============================================================================

This test is known to exceed the tolerance on some systems. That's why we already use a patch to increase it (see 798dabc). Looks like BLIS still exceeds this tolerance.

We should probably report these issues to the BLIS & OpenBLAS developers. I think NumPy and SciPy are less likely to be the culprit here.

[Thyre]

Worth noting that I haven't tested NVPL or other implementations so far, though this is easy to do once NumPy / SciPy are installed, since they ship their entire tests in their installation.

[boegel]

For the BLIS part, let's fix #24765 first, and then re-evaluate...

[boegel]

Same numpy tests are also failing on Neoverse V1 (Graviton 3):

FAILED numpy/linalg/tests/test_linalg.py::TestDet::test_types[complex64] - RuntimeWarning: divide by zero encountered in det
FAILED numpy/linalg/tests/test_linalg.py::TestDet::test_types[complex128] - RuntimeWarning: divide by zero encountered in det

[boegel]

I'm not seeing any failing tests for scipy when using FLEXIBLAS=BLIS eb ..., so the failing scipy/sparse/linalg/_isolve/tests/test_iterative.py seems specific to Neoverse V2 (or NVIDIA Grace)?

[boegel]

see also numpy/numpy#30260

[Thyre]

Here's some more testing on Neoverse V2 with existing installations (installed with --ignore-test-failure):

Overview

	OpenBLAS	BLIS	NVPL
NumPy	0 failed, 43487 passed	0 failed, 43487 passed	0 failed, 43487 passed
SciPy	0 failed, 63527 passed	1 failed, 63526 passed	0 failed, 63527 passed

The tests can be ran with:

$ module load llvm-compilers SciPy-bundle hypothesis Meson
$ FLEXIBLAS=<backend> python -c 'import scipy; scipy.test()'
$ CC=clang CXX=clang++ FC=flang F77=flang F90=flang FLEXIBLAS=<backend> python -c 'import numpy; numpy.test()'

For some reason, NumPy does not fail in this test scenario.
Maybe the failure is caused by some combination of environment variables we use during the build? Those are not used in testing here.

---

Failures

SciPy (BLIS)

Click to open

scipy/sparse/linalg/_isolve/tests/test_iterative.py:508: in test_x0_working
    assert norm(A @ x - b) <= 6e-6*norm(b)
E   assert np.float64(9.05172644450491e-06) <= (6e-06 * np.float64(1.240298651028214))
E    +  where np.float64(9.05172644450491e-06) = norm(((array([[3.68514493, 2.22433683, 2.80424149, 2.06524296, 2.82772901,\n        1.70134349, 2.98806253, 2.88379072, 1.9578..., 2.29515724, 2.77591422, 1.55001596, 1.93969785,\n        1.88091968, 2.24894423, 2.49791763, 1.66944017, 2.12310297]]) @ array([ -1.22143905, -23.52882067,  -4.72398375, -21.36850263,\n       -12.68971783,  -0.9674085 ,  32.85878805, -11.4028599 ,\n        10.99362138,  30.93084508])) - array([0.55127821, 0.22828434, 0.09135873, 0.07249963, 0.11693987,\n       0.74246847, 0.72216603, 0.1607452 , 0.15471589, 0.18022921])))
E    +  and   np.float64(1.240298651028214) = norm(array([0.55127821, 0.22828434, 0.09135873, 0.07249963, 0.11693987,\n       0.74246847, 0.72216603, 0.1607452 , 0.15471589, 0.18022921]))
        A          = array([[3.68514493, 2.22433683, 2.80424149, 2.06524296, 2.82772901,
        1.70134349, 2.98806253, 2.88379072, 1.9578..., 2.29515724, 2.77591422, 1.55001596, 1.93969785,
        1.88091968, 2.24894423, 2.49791763, 1.66944017, 2.12310297]])
        b          = array([0.55127821, 0.22828434, 0.09135873, 0.07249963, 0.11693987,
       0.74246847, 0.72216603, 0.1607452 , 0.15471589, 0.18022921])
        info       = 0
        kw         = {'atol': 0, 'rtol': 1e-06}
        n          = 10
        rng        = Generator(PCG64) at 0x4000DFD0F840
        solver     = <function tfqmr at 0x4000467cb240>
        x          = array([ -1.22143905, -23.52882067,  -4.72398375, -21.36850263,
       -12.68971783,  -0.9674085 ,  32.85878805, -11.4028599 ,
        10.99362138,  30.93084508])
        x0         = array([0.63685093, 0.46836296, 0.0677924 , 0.52832101, 0.49467215,
       0.68333247, 0.3392967 , 0.57867322, 0.25729156, 0.60952728])
[...]
FAILED sparse/linalg/_isolve/tests/test_iterative.py::test_x0_working[tfqmr] - assert np.float64(9.05172644450491e-06) <= (6e-06 * np.float64(1.240298651028214))

[boegel]

On Neoverse V1 (AWS Graviton 3), post-installation:

• scipy w/ FLEXBLAS=OpenBLAS

63527 passed, 4044 skipped, 16244 deselected, 191 xfailed, 12 xpassed, 2782 warnings in 472.33s (0:07:52)

• scipy w/ FLEXBLAS=BLIS

63527 passed, 4044 skipped, 16244 deselected, 191 xfailed, 12 xpassed, 2774 warnings in 476.76s (0:07:56)

• scipy w/ FLEXBLAS=NVPL

FAILED sparse/linalg/_isolve/tests/test_iterative.py::test_x0_equals_Mb[nonsymposdef-F-bicgstab] - assert -11 == 0
=========================================== 1 failed, 63526 passed, 4044 skipped, 16244 deselected, 192 xfailed, 11 xpassed, 2782 warnings in 469.21s (0:07:49) ============================================

details on failing test:

Details

________________________________________________________________________________ test_x0_equals_Mb[nonsymposdef-F-bicgstab] ________________________________________________________________________________
/tmp/boegel/software/SciPy-bundle/2025.07-lfbf-2025b/lib/python3.13/site-packages/scipy/sparse/linalg/_isolve/tests/test_iterative.py:525: in test_x0_equals_Mb
    assert info == 0
E   assert -11 == 0
        A          = <Compressed Sparse Row sparse array of dtype 'complex64'
        with 19 stored elements and shape (10, 10)>
        b          = array([0., 1., 2., 3., 4., 5., 6., 7., 8., 9.])
        case       = <nonsymposdef-F-bicgstab>
        info       = -11
        rtol       = 1e-08
        x          = array([0.        +0.j, 0.52361589+0.j, 1.32356787+0.j, 2.15281624+0.j,
       3.        +0.j, 4.        +0.j, 5.        +0.j, 6.        +0.j,
       7.        +0.j, 8.        +0.j])
        x0         = 'Mb'

• numpy w/ FLEXBLAS=OpenBLAS

NumPy version 2.3.2
NumPy CPU features:  NEON NEON_FP16 NEON_VFPV4 ASIMD ASIMDHP* ASIMDFHM* SVE*
...
43487 passed, 354 skipped, 2819 deselected, 33 xfailed, 1539 xpassed, 9 warnings in 215.72s (0:03:35)

• numpy w/ FLEXBLAS=BLIS

NumPy version 2.3.2
NumPy CPU features:  NEON NEON_FP16 NEON_VFPV4 ASIMD ASIMDHP* ASIMDFHM* SVE*
...
43487 passed, 354 skipped, 2819 deselected, 33 xfailed, 1539 xpassed in 211.57s (0:03:31)

• numpy w/ FLEXBLAS=NVPL

NumPy version 2.3.2
NumPy CPU features:  NEON NEON_FP16 NEON_VFPV4 ASIMD ASIMDHP* ASIMDFHM* SVE*
...
43487 passed, 354 skipped, 2819 deselected, 33 xfailed, 1539 xpassed, 1 warning in 209.71s (0:03:29)

Details

For the latter run, there was a hard crash in the middle (well, at 85%):

Fatal Python error: Aborted

Current thread 0x0000400001e88190 (most recent call first):
  File "/tmp/boegel/software/SciPy-bundle/2025.07-lfbf-2025b/lib/python3.13/site-packages/numpy/linalg/tests/test_linalg.py", line 2006 in test_xerbla_override
  File "/home/boegel/easybuild/Rocky8/neoverse-v1/software/Python-bundle-PyPI/2025.07-GCCcore-14.3.0/lib/python3.13/site-packages/_pytest/python.py", line 157 in pytest_pyfunc_call
  File "/home/boegel/easybuild/Rocky8/neoverse-v1/software/Python-bundle-PyPI/2025.07-GCCcore-14.3.0/lib/python3.13/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/home/boegel/easybuild/Rocky8/neoverse-v1/software/Python-bundle-PyPI/2025.07-GCCcore-14.3.0/lib/python3.13/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/boegel/easybuild/Rocky8/neoverse-v1/software/Python-bundle-PyPI/2025.07-GCCcore-14.3.0/lib/python3.13/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/home/boegel/easybuild/Rocky8/neoverse-v1/software/Python-bundle-PyPI/2025.07-GCCcore-14.3.0/lib/python3.13/site-packages/_pytest/python.py", line 1671 in runtest
  File "/home/boegel/easybuild/Rocky8/neoverse-v1/software/Python-bundle-PyPI/2025.07-GCCcore-14.3.0/lib/python3.13/site-packages/_pytest/runner.py", line 178 in pytest_runtest_call
  File "/home/boegel/easybuild/Rocky8/neoverse-v1/software/Python-bundle-PyPI/2025.07-GCCcore-14.3.0/lib/python3.13/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/home/boegel/easybuild/Rocky8/neoverse-v1/software/Python-bundle-PyPI/2025.07-GCCcore-14.3.0/lib/python3.13/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/boegel/easybuild/Rocky8/neoverse-v1/software/Python-bundle-PyPI/2025.07-GCCcore-14.3.0/lib/python3.13/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/home/boegel/easybuild/Rocky8/neoverse-v1/software/Python-bundle-PyPI/2025.07-GCCcore-14.3.0/lib/python3.13/site-packages/_pytest/runner.py", line 246 in <lambda>
  File "/home/boegel/easybuild/Rocky8/neoverse-v1/software/Python-bundle-PyPI/2025.07-GCCcore-14.3.0/lib/python3.13/site-packages/_pytest/runner.py", line 344 in from_call
  File "/home/boegel/easybuild/Rocky8/neoverse-v1/software/Python-bundle-PyPI/2025.07-GCCcore-14.3.0/lib/python3.13/site-packages/_pytest/runner.py", line 245 in call_and_report
  File "/home/boegel/easybuild/Rocky8/neoverse-v1/software/Python-bundle-PyPI/2025.07-GCCcore-14.3.0/lib/python3.13/site-packages/_pytest/runner.py", line 136 in runtestprotocol
  File "/home/boegel/easybuild/Rocky8/neoverse-v1/software/Python-bundle-PyPI/2025.07-GCCcore-14.3.0/lib/python3.13/site-packages/_pytest/runner.py", line 117 in pytest_runtest_protocol
  File "/home/boegel/easybuild/Rocky8/neoverse-v1/software/Python-bundle-PyPI/2025.07-GCCcore-14.3.0/lib/python3.13/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/home/boegel/easybuild/Rocky8/neoverse-v1/software/Python-bundle-PyPI/2025.07-GCCcore-14.3.0/lib/python3.13/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/boegel/easybuild/Rocky8/neoverse-v1/software/Python-bundle-PyPI/2025.07-GCCcore-14.3.0/lib/python3.13/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/home/boegel/easybuild/Rocky8/neoverse-v1/software/Python-bundle-PyPI/2025.07-GCCcore-14.3.0/lib/python3.13/site-packages/_pytest/main.py", line 367 in pytest_runtestloop
  File "/home/boegel/easybuild/Rocky8/neoverse-v1/software/Python-bundle-PyPI/2025.07-GCCcore-14.3.0/lib/python3.13/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/home/boegel/easybuild/Rocky8/neoverse-v1/software/Python-bundle-PyPI/2025.07-GCCcore-14.3.0/lib/python3.13/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/boegel/easybuild/Rocky8/neoverse-v1/software/Python-bundle-PyPI/2025.07-GCCcore-14.3.0/lib/python3.13/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/home/boegel/easybuild/Rocky8/neoverse-v1/software/Python-bundle-PyPI/2025.07-GCCcore-14.3.0/lib/python3.13/site-packages/_pytest/main.py", line 343 in _main
  File "/home/boegel/easybuild/Rocky8/neoverse-v1/software/Python-bundle-PyPI/2025.07-GCCcore-14.3.0/lib/python3.13/site-packages/_pytest/main.py", line 289 in wrap_session
  File "/home/boegel/easybuild/Rocky8/neoverse-v1/software/Python-bundle-PyPI/2025.07-GCCcore-14.3.0/lib/python3.13/site-packages/_pytest/main.py", line 336 in pytest_cmdline_main
  File "/home/boegel/easybuild/Rocky8/neoverse-v1/software/Python-bundle-PyPI/2025.07-GCCcore-14.3.0/lib/python3.13/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/home/boegel/easybuild/Rocky8/neoverse-v1/software/Python-bundle-PyPI/2025.07-GCCcore-14.3.0/lib/python3.13/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/boegel/easybuild/Rocky8/neoverse-v1/software/Python-bundle-PyPI/2025.07-GCCcore-14.3.0/lib/python3.13/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/home/boegel/easybuild/Rocky8/neoverse-v1/software/Python-bundle-PyPI/2025.07-GCCcore-14.3.0/lib/python3.13/site-packages/_pytest/config/__init__.py", line 175 in main
  File "/tmp/boegel/software/SciPy-bundle/2025.07-lfbf-2025b/lib/python3.13/site-packages/numpy/_pytesttester.py", line 197 in __call__
  File "<string>", line 1 in <module>

So, the numpy test failures that happened when the test suite was being run during the installation did not occur when doing a post-installation run of the test suite.

The divide by zero issue did pop up during the installation though, as it was recorded in the EasyBuild log file:

Details

================================================================================================= FAILURES =================================================================================================
______________________________________________________________________________________ TestDet.test_types[complex64] _______________________________________________________________________________________

self = <numpy.linalg.tests.test_linalg.TestDet object at 0x40009c8f3490>, dtype = <class 'numpy.complex64'>

    @pytest.mark.parametrize('dtype', [single, double, csingle, cdouble])
    def test_types(self, dtype):
        x = np.array([[1, 0.5], [0.5, 1]], dtype=dtype)
>       assert_equal(np.linalg.det(x).dtype, dtype)
                     ^^^^^^^^^^^^^^^^

dtype      = <class 'numpy.complex64'>
self       = <numpy.linalg.tests.test_linalg.TestDet object at 0x40009c8f3490>
x          = array([[1. +0.j, 0.5+0.j],
       [0.5+0.j, 1. +0.j]], dtype=complex64)

numpy/linalg/tests/test_linalg.py:945:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

a = array([[1. +0.j, 0.5+0.j],
       [0.5+0.j, 1. +0.j]], dtype=complex64)

    @array_function_dispatch(_unary_dispatcher)
    def det(a):
        """
        Compute the determinant of an array.

        Parameters
        ----------
        a : (..., M, M) array_like
            Input array to compute determinants for.

        Returns
        -------
        det : (...) array_like
            Determinant of `a`.

        See Also
        --------
        slogdet : Another way to represent the determinant, more suitable
          for large matrices where underflow/overflow may occur.
        scipy.linalg.det : Similar function in SciPy.

        Notes
        -----
       Broadcasting rules apply, see the `numpy.linalg` documentation for
        details.

        The determinant is computed via LU factorization using the LAPACK
        routine ``z/dgetrf``.

        Examples
        --------
        The determinant of a 2-D array [[a, b], [c, d]] is ad - bc:

        >>> import numpy as np
        >>> a = np.array([[1, 2], [3, 4]])
        >>> np.linalg.det(a)
        -2.0 # may vary

        Computing determinants for a stack of matrices:

        >>> a = np.array([ [[1, 2], [3, 4]], [[1, 2], [2, 1]], [[1, 3], [3, 1]] ])
        >>> a.shape
        (3, 2, 2)
        >>> np.linalg.det(a)
        array([-2., -3., -8.])

        """
        a = asarray(a)
        _assert_stacked_square(a)
        t, result_t = _commonType(a)
        signature = 'D->D' if isComplexType(t) else 'd->d'
>       r = _umath_linalg.det(a, signature=signature)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E       RuntimeWarning: divide by zero encountered in det

a          = array([[1. +0.j, 0.5+0.j],
       [0.5+0.j, 1. +0.j]], dtype=complex64)
result_t   = <class 'numpy.complex64'>
signature  = 'D->D'
t          = <class 'numpy.complex128'>

numpy/linalg/_linalg.py:2430: RuntimeWarning
______________________________________________________________________________________ TestDet.test_types[complex128] ______________________________________________________________________________________

self = <numpy.linalg.tests.test_linalg.TestDet object at 0x40009d439370>, dtype = <class 'numpy.complex128'>

    @pytest.mark.parametrize('dtype', [single, double, csingle, cdouble])
    def test_types(self, dtype):
        x = np.array([[1, 0.5], [0.5, 1]], dtype=dtype)
>       assert_equal(np.linalg.det(x).dtype, dtype)
                     ^^^^^^^^^^^^^^^^

dtype      = <class 'numpy.complex128'>
self       = <numpy.linalg.tests.test_linalg.TestDet object at 0x40009d439370>
x          = array([[1. +0.j, 0.5+0.j],
       [0.5+0.j, 1. +0.j]])

numpy/linalg/tests/test_linalg.py:945:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

a = array([[1. +0.j, 0.5+0.j],
       [0.5+0.j, 1. +0.j]])

    @array_function_dispatch(_unary_dispatcher)
    def det(a):
        """
        Compute the determinant of an array.

        Parameters
        ----------
        a : (..., M, M) array_like
            Input array to compute determinants for.

        Returns
        -------
        det : (...) array_like
            Determinant of `a`.

        See Also
        --------
        slogdet : Another way to represent the determinant, more suitable
          for large matrices where underflow/overflow may occur.
        scipy.linalg.det : Similar function in SciPy.

        Notes
        -----
        Broadcasting rules apply, see the `numpy.linalg` documentation for
        details.

        The determinant is computed via LU factorization using the LAPACK
        routine ``z/dgetrf``.
       Examples
        --------
        The determinant of a 2-D array [[a, b], [c, d]] is ad - bc:

        >>> import numpy as np
        >>> a = np.array([[1, 2], [3, 4]])
        >>> np.linalg.det(a)
        -2.0 # may vary

        Computing determinants for a stack of matrices:

        >>> a = np.array([ [[1, 2], [3, 4]], [[1, 2], [2, 1]], [[1, 3], [3, 1]] ])
        >>> a.shape
        (3, 2, 2)
        >>> np.linalg.det(a)
        array([-2., -3., -8.])

        """
        a = asarray(a)
        _assert_stacked_square(a)
        t, result_t = _commonType(a)
        signature = 'D->D' if isComplexType(t) else 'd->d'
>       r = _umath_linalg.det(a, signature=signature)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E       RuntimeWarning: divide by zero encountered in det

a          = array([[1. +0.j, 0.5+0.j],
       [0.5+0.j, 1. +0.j]])
result_t   = <class 'numpy.complex128'>
signature  = 'D->D'
t          = <class 'numpy.complex128'>

numpy/linalg/_linalg.py:2430: RuntimeWarning
========================================================================================= short test summary info ==========================================================================================
FAILED numpy/linalg/tests/test_linalg.py::TestDet::test_types[complex64] - RuntimeWarning: divide by zero encountered in det
FAILED numpy/linalg/tests/test_linalg.py::TestDet::test_types[complex128] - RuntimeWarning: divide by zero encountered in det
=================================================== 2 failed, 43511 passed, 328 skipped, 2819 deselected, 33 xfailed, 1539 xpassed in 384.67s (0:06:24) ====================================================

[boegel]

@Thyre W.r.t. numpy, how the test suite is being run seems to be a factor too...

During the installation, EasyBuild runs spin test --no-build --verbose (A).
You're running python -c 'import numpy; numpy.test()' (B)

With A in an interactive shell after the numpy test suite failed on Neoverse V1, I can reproduce the failing tests by running the spin test command (or by running "spin test --no-build --verbose -- -k test_types", which runs a small subsuite of tests, including the failing ones).
With B, I can't reproduce the failing tests (so maybe those tests aren't actually being run then?)

Test result with spin test (without filtering):

2 failed, 43511 passed, 328 skipped, 2819 deselected, 33 xfailed, 1539 xpassed in 356.93s (0:05:56)

Note the slightly higher total test count (after fewer skipped tests)...