Issue with IDAKLU installation from source

[brosaplanella]

I am doing a fresh install on Python 3.11 (in WSL). The comman nox -s pybamm-requires worked fine, but when running nox -s dev I got the error below, related to IDAKLU.

Has anyone else encountered this?

      copying pybamm/../CMakeBuild.py -> build/lib.linux-x86_64-cpython-311/pybamm/..
      running build_ext
      ---------- Running CMake for idaklu solver ----------------------------------------
      -- The C compiler identification is GNU 9.4.0
      -- The CXX compiler identification is GNU 9.4.0
      -- Detecting C compiler ABI info
      -- Detecting C compiler ABI info - done
      -- Check for working C compiler: /usr/bin/cc - skipped
      -- Detecting C compile features
      -- Detecting C compile features - done
      -- Detecting CXX compiler ABI info
      -- Detecting CXX compiler ABI info - done
      -- Check for working CXX compiler: /usr/bin/c++ - skipped
      -- Detecting CXX compile features
      -- Detecting CXX compile features - done
      -- pybind11 v2.12.0 dev1
      CMake Warning (dev) at pybind11/tools/FindPythonLibsNew.cmake:101 (message):
        Policy CMP0148 is not set: The FindPythonInterp and FindPythonLibs modules
        are removed.  Run "cmake --help-policy CMP0148" for policy details.  Use
        the cmake_policy command to set the policy and suppress this warning, or
        preferably upgrade to using FindPython, either by calling it explicitly
        before pybind11, or by setting PYBIND11_FINDPYTHON ON before pybind11.
      Call Stack (most recent call first):
        pybind11/tools/pybind11Tools.cmake:50 (find_package)
        pybind11/tools/pybind11Common.cmake:188 (include)
        pybind11/CMakeLists.txt:222 (include)
      This warning is for project developers.  Use -Wno-dev to suppress it.

      -- Found PythonInterp: /home/brosaplanella/PyBaMM/venv/bin/python (found suitable version "3.11.6", minimum required is "3.6")
      -- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.11.so
      -- Performing Test HAS_FLTO
      -- Performing Test HAS_FLTO - Success
      Traceback (most recent call last):
        File "<string>", line 1, in <module>
      ModuleNotFoundError: No module named 'casadi'
      Found python casadi path:
      Trying to link against python casadi package
      CMake Error at CMakeLists.txt:80 (find_package):
        Could not find a package configuration file provided by "casadi" with any
        of the following names:

          casadiConfig.cmake
          casadi-config.cmake

        Add the installation prefix of "casadi" to CMAKE_PREFIX_PATH or set
        "casadi_DIR" to a directory containing one of the above files.  If "casadi"
        provides a separate development package or SDK, be sure it has been
        installed.


      -- Configuring incomplete, errors occurred!
      ---------- Building idaklu module ----------------------------------------
      make: Makefile: No such file or directory
      make: *** No rule to make target 'Makefile'.  Stop.
      error: can't copy '/home/brosaplanella/PyBaMM/build/temp.linux-x86_64-cpython-311/idaklu.cpython-311-x86_64-linux-gnu.so': doesn't exist or not a regular file
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pybamm
  Running setup.py clean for pybamm
Failed to build pybamm
ERROR: Could not build wheels for pybamm, which is required to install pyproject.toml-based projects
nox > Command /home/brosaplanella/PyBaMM/venv/bin/python -m pip install '.[all,dev,jax,odes]' failed with exit code 1
nox > Session dev failed.

[agriyakhetarpal]

I have encountered this issue earlier (a lot, actually! 😃), but it should be nothing to worry. Could you uninstall or --force-reinstall casadi and try installing PyBaMM with updated setuptools and wheel?

The gist is that CMake is unable to find casadi in the site-packages directory inside your venv, following which it tries to find casadi in alternative locations.

[brosaplanella]

Is that with the venv activated or deactivated?

[agriyakhetarpal]

Deactivated, since the nox -s dev session now creates a folder venv/ for you and installs all the dependencies inside it

[brosaplanella]

Converting it into a discussion as it might be useful to other people.

[agriyakhetarpal]

Yes, a discussion makes sense. For more information, a general overview of an installation of PyBaMM (from source) as of 16 November 2023 (23.9rc1) is as follows.

1. PyBaMM has a C++ extension module pybamm.solvers.idaklu which constitutes the optional IDAKLU solver, interfacing IDAS and KLU. This requires SUNDIALS and SuiteSparse to be installed on the source machine beforehand, which are then linked with casadi and pybind11 to build the shared module that Python's importlib uses to call the solvers' defined methods and classes through their C++ code. This shared module is found in the build/ folder on non-editable installations, i.e., with pip install . and is found in the pybamm/solvers/ folder on editable installations, i.e., with pip install -e .. The name of the shared object file is usually like idaklu.cpython-311-darwin.so. Here, idaklu is the name of the CMake target used to link all these libraries, cpython is the Python implementation and compatible ABI for the solver, 311 is the Python version it was built for, and darwin is the platform it was built for. The file extension is .so on Linux and macOS. The current installation instructions in the documentation mention that the IDAKLU solver is unavailable on Windows when installing PyBaMM from source: the (simple) reason is that the build procedure is complicated, requires Visual Studio, and the SUNDIALS needs to be enabled to use the KLU component from SuiteSparse. We have a custom vcpkg registry for this purpose when building wheels and pushing them to PyPI. The file extension for the IDAKLU solver in the Windows wheels is pyd, which is the same as a DLL (dynamic link library) that a Python executable can readily import.
2. When one runs nox -s pybamm-requires, the nox session runs scripts/install_KLU_Sundials.py under the hood, which downloads and extracts SUNDIALS and SuiteSparse tarballs from their GitHub releases pages in the install_KLU_Sundials/ folder. Then, CMake (with custom build arguments to enable required features such as OpenMP, KLU, and more) and subsequently Make are invoked to build the SUNDIALS and SuiteSparse .so (shared object files, Linux) or .dylib (dynamic libraries, macOS) files in a default installation location – which is a .local/ folder in the HOME directory. The session then clones pybind11 from GitHub in the PyBaMM root directory.
3. When pip install -e . is invoked, setup.py decides if the IDAKLU extension is to be compiled or not based on if the mentioned build-time requirements are compiled. It skips the compilation if the setup is running on Windows or if either of CMake or pybind11 are not found. PyBaMM is essentially a pure-Python package without the IDAKLU solver/extension. If the IDAKLU extension is then decided to be compiled, the setup runs CMake in a subprocess, which uses the CMakeLists.txt file for its configuration. This file uses the Python executable and tries to import casadi in order to find the path where it is installed and tries to link the idaklu target against it. It fails if casadi cannot be imported or is not found to be installed for some reason during the dependency installation phase in setup.py and tries to link to a different casadi if available, such as on Windows where it comes from another custom vcpkg registry. The IDAKLU extension is then linked against SUNDIALS and SuiteSparse which were compiled earlier, and the installation process gets completed or proceeds based on whether any extras have been requested to be installed.
4. The error in the log: idaklu.cpython-311-x86_64-linux-gnu.so': doesn't exist or not a regular file is a very generic trace that comes from prior failed steps in the build process and is seldom the actual error.

The best practice is to install nox globally as a --user (or use pipx run nox to run it in an isolated environment): then run nox -s pybamm-requires, followed by nox -s dev. The latter command automatically creates a virtual environment with virtualenv in the root directory in a folder named venv/ and installs PyBaMM in editable mode with its dependencies (such as casadi, numpy, scipy, xarray, etc.) into that folder. The virtual environment can be activated afterwards and used. If nox -s pybamm-requires is not run, the IDAKLU solver will not be available and PyBaMM can be installed as a pure-Python package, i.e., pybamm.have_idaklu() will return False. Note that this overview is strictly only for the source installation and the installation procedure differs if a wheel (from PyPI) is being used; where SuiteSparse, SUNDIALS, and the IDAKLU extension pre-compiled binaries are statically linked inside the wheel which allows a faster and more reliable installation procedure.

[chmabaur]

Here more details about the linking issue. The libsuitesparseconfig.so.6 was compiled correctly and is in the ~/.local/lib folder

/mnt/data-share/Repositories/PyBaMM/venv/lib/python3.10/site-packages/pybamm/solvers$ ldd idaklu.cpython-310-x86_64-linux-gnu.so
        linux-vdso.so.1 (0x00007ffffc873000)
        libsundials_idas.so.5 => /home/user/.local/lib/libsundials_idas.so.5 (0x00007f01b2a9b000)
        libsundials_sunlinsolklu.so.4.5.0 => /home/user/.local/lib/libsundials_sunlinsolklu.so.4.5.0 (0x00007f01b2a80000)
        libsundials_nvecopenmp.so.6 => /home/user/.local/lib/libsundials_nvecopenmp.so.6 (0x00007f01b2a5b000)
        libcasadi.so.3.7 => /mnt/data-share/Repositories/PyBaMM/venv/lib/python3.10/site-packages/casadi/libcasadi.so.3.7 (0x00007f01b2000000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f01b1c00000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f01b2a22000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f01b1800000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f01b2b7c000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f01b293b000)
        libsundials_sunmatrixsparse.so.4 => /home/user/.local/lib/libsundials_sunmatrixsparse.so.4 (0x00007f01b291d000)
        libklu.so.2 => /home/user/.local/lib/libklu.so.2 (0x00007f01b1fcc000)
        libgomp.so.1 => /lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f01b1f82000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f01b2916000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f01b2911000)
        libsuitesparseconfig.so.6 => not found
        libamd.so.3 => not found
        libcolamd.so.3 => not found
        libbtf.so.2 => not found

ls -als  ~/.local/lib/ | grep libsuite*
   8 -rw-r--r-- 1 user user    7308 Nov 21 08:42 libsuitesparseconfig.a
   0 lrwxrwxrwx 1 user user      25 Nov 20 11:12 libsuitesparseconfig.so -> libsuitesparseconfig.so.6
   0 lrwxrwxrwx 1 user user      29 Nov 20 11:12 libsuitesparseconfig.so.6 -> libsuitesparseconfig.so.6.0.3
  20 -rw-r--r-- 1 user user   16912 Nov 21 08:42 libsuitesparseconfig.so.6.0.3

[agriyakhetarpal]

Thanks, this helps a lot and might be a new bug if it is reproduced. It looks like the suitesparseconfig shared object file was compiled with the absence of a search path for an unexplained reason. Does removing the contents of the .local/ folder and then running pybamm-requires again help?

[agriyakhetarpal]

Also, while I am not familiar with how Python API interacts with shared extensions I know that it uses dlopen for the import mechanism: could you share the output from strace -e trace=open idaklu.cpython-310-x86_64-linux-gnu.so as well, besides ldd?

[chmabaur]

Deleting ./local folder and rerunning pybamm-requires did not help, still same linking error.

strace outputs the following:

/mnt/data-share/Repositories/PyBaMM/venv/lib/python3.10/site-packages/pybamm/solvers$ strace -e trace=open /mnt/data-share/Repositories/PyBaMM/venv/lib/python3.10/site-packages/pybamm/solvers/idaklu.cpython-310-x86_64-linux-gnu.so
--- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_ACCERR, si_addr=0x7fa5201d5000} ---
+++ killed by SIGSEGV (core dumped) +++
Segmentation fault (core dumped)

[agriyakhetarpal]

Thanks for the help @chmabaur – I'm not sure how to debug that further, tagging @martinjrobins. Also, @arjxn-py, could you try this and let me know if you face the same issue on a fresh install on WSL?