Optimizing Direct Solvers

Introduction

One of the core philosophies of EMerge is that is should be easy to use and install without complicated package dependencies. Sadly, due to differences in compatibility of high performance libraries, this is not always as easy as we hope. Different Python versions on different operating systems with different package managers may see different results.

As far as we know, SuperLU and UMFPACK will work on all operating systems. However, the OpenBLAS backend of Scipy that contains SuperLU is about 2 to 3 times faster than other backends. The level of optimization with parallel threads also varies. Getting SuperLU to work with OpenBLAS is not always easy.

We are working on writing the solvers such that EMerge works "out of the box" as well as possible. To assist with optimization we have the following instructions.

Windows

For Intel machines, Windows seems to pick the right backends for standard Scipy installation through Pip or Conda. With AMD processors we currently don't know the performance.

MacOS (Intel and ARM)

On older MacOS Intel systems it is known that Scipy doesn't always install Scipy with the right backend. For that reason we propose the following recipe courtesy of reddit user /u/Swipecat. It is verified that this also works for ARM systems. The improvement over Accelerate is marginal but present (about +25% faster).

• Python version <=3.11 Uninstall existing scipy and numpy builds.

pip uninstall scipy
pip uninstall numpy

Then install the anaconda version of scipy and numpy

pip install --pre -i https://pypi.anaconda.org/scipy-wheels-nightly/simple scipy
pip install --pre -i https://pypi.anaconda.org/scipy-wheels-nightly/simple numpy

If this doesn't work you might have to add `python -m ...':

python -m pip install --pre -i https://pypi.anaconda.org/scipy-wheels-nightly/simple scipy
python -m pip install --pre -i https://pypi.anaconda.org/scipy-wheels-nightly/simple numpy

It is known that this installation may output the following line in your terminal when importing emerge.

/bin/sh: lscpu: command not found

As far as its known, this is harmless.

Linux

EMerge is known to work on Linux machines with PyPardiso on an intel machine. The SuperLU openBLAS performance needs to be verified.

Compatibility table

SuperLU runs on all operating systems. The following table summarises the known compatibility of the direct solver libraries with different operating systems and CPU architectures.

OS	Arch.	PyPARDISO	SuperLU	UMFPACK	SuperLU (OpenBLAS)
Windows	Intel	✅	✅	❓	✅
Windows	AMD	✅	✅	❓	✅
MacOS	Intel	✅	✅	❓	Python <=3.11
MacOS	ARM	❌	✅	✅	Python <=3.11
Linux	Intel	✅	✅	❓	✅
Linux	AMD	❓	✅	❓	❓