BLD/DOC: Verify MKL works with Meson build, and document BLAS/LAPACK selection better

[rgommers]

After gh-16187 is complete, I think solid MKL support is a last blocker for shipping 1.9.0 with Meson. Building against MKL should work with pkg-config, however conda-forge .pc files are missing. And we'd like more robust support beyond pkg-config, in particular for MKL but also for other libraries. Relevant issues:

• Feature: MKL dependency mesonbuild/meson#2835
• pkg-config files for MKL are missing conda-forge/intel_repack-feedstock#34

[rgommers]

Here is an overview of pkg-config support for MKL and other libraries in a few packaging systems:

For conda-forge:

• package openblas installs: <conda-root>/envs/<env-name>/lib/pkgconfig/openblas.pc
• package blis installs: <conda-root>/envs/<env-name>/share/pkgconfig/blis.pc
• package blas (note, for Netlib BLAS only installable with conda install "libblas=*=*netlib", see Switching BLAS implementation) installs: <conda-root>/envs/<env-name>/lib/pkgconfig/blas.pc
• package lapack (same note, conda install "libblas=*=*netlib") installs: <conda-root>/envs/<env-name>/lib/pkgconfig/lapack.pc
• package blas-devel installs (additionally to blas.pc and lapack.pc):
  • <conda-root>/envs/<env-name>/lib/pkgconfig/cblas.pc
  • <conda-root>/envs/<env-name>/lib/pkgconfig/lapacke.pc

To check that they can be found and used:

$ which pkg-config
/home/rgommers/anaconda3/envs/testpkgconfig/bin/pkg-config
$ pkg-config --libs blas
-L/home/rgommers/anaconda3/envs/testpkgconfig/lib -lblas
$ pkg-config --libs lapack
-L/home/rgommers/anaconda3/envs/testpkgconfig/lib -llapack

MKL does not have any pkgconfig files in conda-forge. They do exist however, see Intel Math Kernel Library (Intel MKL) and pkg-config tool . Debian seems to ship all flavors (from https://packages.debian.org/bullseye/amd64/libmkl-dev/filelist)

/usr/lib/x86_64-linux-gnu/pkgconfig/mkl-dynamic-ilp64-iomp.pc
/usr/lib/x86_64-linux-gnu/pkgconfig/mkl-dynamic-ilp64-seq.pc
/usr/lib/x86_64-linux-gnu/pkgconfig/mkl-dynamic-lp64-iomp.pc
/usr/lib/x86_64-linux-gnu/pkgconfig/mkl-dynamic-lp64-seq.pc
/usr/lib/x86_64-linux-gnu/pkgconfig/mkl-sdl-ilp64.pc
/usr/lib/x86_64-linux-gnu/pkgconfig/mkl-sdl-lp64.pc
/usr/lib/x86_64-linux-gnu/pkgconfig/mkl-static-ilp64-iomp.pc
/usr/lib/x86_64-linux-gnu/pkgconfig/mkl-static-ilp64-seq.pc
/usr/lib/x86_64-linux-gnu/pkgconfig/mkl-static-lp64-iomp.pc
/usr/lib/x86_64-linux-gnu/pkgconfig/mkl-static-lp64-seq.pc

Arch Linux ships only the dynamic versions (from https://archlinux.org/packages/community/x86_64/intel-mkl/):

usr/lib/pkgconfig/mkl-dynamic-ilp64-iomp.pc
usr/lib/pkgconfig/mkl-dynamic-ilp64-seq.pc
usr/lib/pkgconfig/mkl-dynamic-lp64-iomp.pc
usr/lib/pkgconfig/mkl-dynamic-lp64-seq.pc

Spack seems to ship the dynamic flavors as well.

The mkl and mkl-devel packages in conda-forge come from https://github.com/conda-forge/intel_repack-feedstock. I've opened conda-forge/intel_repack-feedstock#34 to request .pc files for MKL.

For ATLAS, naming may vary. Debian packages things like:

/usr/lib/x86_64-linux-gnu/liblapack_atlas.a
/usr/lib/x86_64-linux-gnu/liblapack_atlas.so
/usr/lib/x86_64-linux-gnu/pkgconfig/blas-atlas.pc
/usr/lib/x86_64-linux-gnu/pkgconfig/lapack-atlas.pc

[rgommers]

xref gh-11812 for notes on some of the issues around building against MKL.

[rgommers]

Getting back to this, and figuring out what we actually want. Let's start with stating that NumPy and SciPy should both be built against the same BLAS/LAPACK libraries - MKL in this case. So we don't have to deal with "pip install numpy (vendored OpenBLAS) and then build SciPy against MKL". What we do need to support:

• MKL being installed directly with the Intel installers (OneAPI or standalone), or with conda, spack, or a Linux package manager (or any package manager really; Homebrew and Chocolatey don't seem to package MKL though).
• Detection through either pkg-config (preferred) or cmake (fallback). No manual setting of paths through a site.cfg file - if folks want something like that, they can manually point pkg-config to a self-constructed or copied .pc file.

Scenarios to support and test:

1. Locally building NumPy and SciPy against MKL in a virtualenv or conda env.
   • NumPy built with numpy.distutils (status as of now)
   • NumPy built with meson (in the future)
2. Building a conda-forge package for SciPy against Netlib BLAS/LAPACK with -Duse-g77-abi=true (see meson_options.txt)
3. Building a Spack environment with NumPy and SciPy linked against MKL in one go (see py-scipy: add v1.9 spack/spack#31810)

There's a number of ways to link against MKL, see https://www.intel.com/content/www/us/en/developer/tools/oneapi/onemkl-link-line-advisor.html. I think what we want is:

• default to mkl-dynamic-lp64-seq (or to whatever is the regular MKL default?),
• allow users to choose another option, via something like dependency('mkl', static: true, modules: ['openmp', 'ILP64']) (TODO: check how CMake does this)
• handle the assortment of environment variables for MKL (e.g. MKLROOT, MKL_INTERFACE_LAYER=LP64,GNU) in a sensible way

In case NumPy is built with numpy.distutils, the Intel link line advisor isn't followed. Unless a custom site.cfg is used, NumPy just blindly links against libmkl_rt. Meson's behavior should not depend on how NumPy was built, so probably it's best to support that via meson setup build -Dblas=mkl_rt -Dlapack=mkl_rt. We have already seen that that works in the Spack PR linked above.

For how NumPy does things now:

• https://github.com/numpy/numpy/blob/71155e9/site.cfg.example#L194-L217
• https://github.com/numpy/numpy/blob/71155e9331cd30dbb37732a74fca5f23ab734aa4/numpy/distutils/system_info.py#L1251
• mklroot is typically None in that system_info.py code, so the end result is -lmkl_rt on Windows and -lmkl_rt -lpthread on other platforms.

[rgommers]

A first attempt, after preparing a conda environment on Linux with:

$ mamba env create -f environment.yml
$ mamba activate scipy-dev
$ mamba install cmake mkl mkl-include

Then build NumPy in-place with numpy.distutils and check it:

$ cd ../numpy
$ python setup.py build_ext -i
$ python -c "import numpy as np; np.test('full')"  # tests pass
$ ldd lapack_lite.cpython-310-x86_64-linux-gnu.so
        linux-vdso.so.1 (0x00007ffff3f35000)
        libmkl_rt.so.2 => /home/rgommers/anaconda3/envs/scipy-dev/lib/libmkl_rt.so.2 (0x00007f5e1af4d000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007f5e1ae3d000)
        libgcc_s.so.1 => /home/rgommers/anaconda3/envs/scipy-dev/lib/libgcc_s.so.1 (0x00007f5e1ae24000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f5e1ac1a000)
        libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f5e1ac15000)
        /usr/lib64/ld-linux-x86-64.so.2 (0x00007f5e1b847000)

Then build SciPy (a few minor fixes, see gh-17013) against MKL (using CMake here) - this does not work:

$ cd ../scipy
$ meson setup build -Dblas=MKL -Dlapack=MKL --prefix=$PWD/build-install
$ python dev.py build
$ python dev.py test -s linalg  # crashes, because of:
$ ldd build/scipy/linalg/_fblas.cpython-310-x86_64-linux-gnu.so
        linux-vdso.so.1 (0x00007fff23dbb000)
        libomp.so => /home/rgommers/anaconda3/envs/scipy-dev/lib/libomp.so (0x00007fabaa62e000)
        libmkl_core.so.2 => /home/rgommers/anaconda3/envs/scipy-dev/lib/libmkl_core.so.2 (0x00007faba6258000)
        libmkl_gf_ilp64.so.2 => /home/rgommers/anaconda3/envs/scipy-dev/lib/libmkl_gf_ilp64.so.2 (0x00007faba5717000)
        libmkl_intel_thread.so.2 => /home/rgommers/anaconda3/envs/scipy-dev/lib/libmkl_intel_thread.so.2 (0x00007faba1fa3000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007faba1e93000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007faba1c87000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007faba1c82000)
        librt.so.1 => /usr/lib/librt.so.1 (0x00007faba1c7d000)
        libdl.so.2 => /usr/lib/libdl.so.2 (0x00007faba1c78000)
        /usr/lib64/ld-linux-x86-64.so.2 (0x00007fabaa7ba000)

Now try to link against the same library (libmkl_rt) as numpy.distutils does:

$ meson setup build -Dblas=mkl_rt -Dlapack=mkl_rt --prefix=$PWD/build-install
...
Run-time dependency mkl_rt found: NO (tried pkgconfig and cmake)

No pkg-config files, no luck. There won't be any .pc files nor CMake support for mkl_rt I guess. So we need to do this by hand probably. The Intel link line advisor says that for using gfortran, Intel64 architecture, and 32-bit Fortran interface, we need:

 -L${MKLROOT}/lib/intel64 -Wl,--no-as-needed -lmkl_gf_lp64 -lmkl_sequential -lmkl_core -lpthread -lm -ldl

There is no way to get it to say we ever need -lmkl_rt, so I guess numpy.distutils does something old/funky.

Back to regular -Dblas=MKL, and trying the MKL environment variables:

$ git clean -xdf
$ export MKL_INTERFACE_LAYER=GNU,LP64
$ export MKL_THREADING_LAYER=SEQUENTIAL
$ meson setup build -Dblas=MKL -Dlapack=MKL --prefix=$PWD/build-instal
$ python dev.py build
...
  User defined options
    prefix: /home/rgommers/code/scipy/build-install
    blas  : MKL
    lapack: MKL
...
$ ldd build/scipy/linalg/_fblas.cpython-310-x86_64-linux-gnu.so | grep mkl
        libmkl_core.so.2 => /home/rgommers/anaconda3/envs/scipy-dev/lib/libmkl_core.so.2 (0x00007f08be64d000)
        libmkl_gf_ilp64.so.2 => /home/rgommers/anaconda3/envs/scipy-dev/lib/libmkl_gf_ilp64.so.2 (0x00007f08bdb0c000)
        libmkl_intel_thread.so.2 => /home/rgommers/anaconda3/envs/scipy-dev/lib/libmkl_intel_thread.so.2 (0x00007f08ba398000)

$ python dev.py test -s linalg
...
=================== 3626 passed, 3 skipped, 7 deselected, 3 xfailed, 2 xpassed in 11.90s ===================

$ python dev.py test
...
FAILED scipy/sparse/linalg/_isolve/tests/test_iterative.py::test_x0_equals_Mb[bicgstab] - AssertionError: 
FAILED scipy/sparse/tests/test_base.py::TestCOO::test_reshape_copy - AssertionError
FAILED scipy/sparse/tests/test_base.py::TestCOONonCanonical::test_reshape_copy - AssertionError
FAILED scipy/sparse/tests/test_base.py::Test64Bit::test_resiliency_limit_10[TestCOO-test_reshape_copy] - ...
FAILED scipy/sparse/tests/test_base.py::Test64Bit::test_resiliency_random[TestCOO-test_reshape_copy] - As...
FAILED scipy/sparse/tests/test_base.py::Test64Bit::test_resiliency_all_32[TestCOO-test_reshape_copy] - As...
FAILED scipy/sparse/tests/test_base.py::Test64Bit::test_resiliency_all_64[TestCOO-test_reshape_copy] - As...
FAILED scipy/sparse/tests/test_base.py::Test64Bit::test_no_64[TestCOO-test_reshape_copy] - AssertionError
=== 8 failed, 37226 passed, 2171 skipped, 12230 deselected, 138 xfailed, 8 xpassed in 438.02s (0:07:18) ====

8 errors, which look unrelated. Let's call that a first success:)

EDIT: linalg tests pass with every setting of MKL_THREADING_LAYER (or after unset-ing it). The critical variable is MKL_INTERFACE_LAYER - it must be GNU,LP64 or LP64 for things to work, when it is unset or is ILP64 or GNU,ILP64 there are crashes.

[ilayn]

@rgommers This looks like we are done. Should we close it?

[rgommers]

This issue is still useful to me and not 100% finished; MKL selection works with the above details, but a simple -Dblas=mkl or auto-selection does not yet.

[ilayn]

Ah ok