Default installation leads to sub-optimal performance on Ubuntu 20.04 and 22.04

[cpjordan]

Details

Following on from some discussion on Thetis Issue #391, I have found that the default options for installing PETSc lead to slower run times, whereas when using a Docker image or building my own PETSc first, I get much faster run times. When I build my own PETSc I use the same configuration flags as the Firedrake Dockerfile.env (see the linked Issue for my exact installation steps). I have been experimenting with this test on Thetis as I have also been comparing a slow down in performance since October 2024. My results are as follows:

Dell Precision 5820 Tower X-Series (Ubuntu 20.04.6 LTS)

Local build - default: ~390s (serial_latest-default_petsc_log.txt)
Local build - own PETSc build: 100s (serial_latest_separate_PETSc_build_petsc_log.txt)
Docker image: 100s

HP Z6 G5 Workstation Desktop (Ubuntu 22.04.5 LTS)

Local build - default: 351s
Docker image: 76s
We haven't tested building our own PETSc here, but I anticipate the same results. I can do this if helpful.

The difference in times between default installation and the latest image/separate PETSc build appears to be due to the choice of blas and lapack used in the default installation (thanks @stephankramer for figuring this out for me!). This appears to be a result of the installation options here: https://github.com/firedrakeproject/firedrake/blob/master/scripts/firedrake-install#L755. Is this behaviour expected? I can't see any other complaints noting a slow down in performance from the default build. Note that this seems to have only been an issue I've personally been aware of since October last year, but I initially thought the slow down I was experiencing could have been for other reasons.

[connorjward]

Instead of those arguments might it be to do with the CFLAGS which we set in the Dockerfile? We don't appear to set them inside firedrake-install so perhaps the default compiler flags have gotten worse. If so then this issue will be addressed in #4011

[cpjordan]

Stephan also suggested that, but when I installed with:

./configure --COPTFLAGS="-O3 -march=native -mtune=native" --CXXOPTFLAGS=-O3 -march=native -mtune=native --FOPTFLAGS="-O3 -march=native -mtune=native"

I didn't see an improvement in my run times. The log (L254) appears to have the correct C flags but the time is still really slow: serial_cflags_configured_petsc_log.txt
It was only building PETSc beforehand that then improved my results, despite using those C flags in both cases.

I can build PETSc myself, it's just an extra few steps - it makes a massive difference to my run times however. I'll test when #4011 is implemented and let you know the results!

[connorjward]

Interesting. I added those extra options because I believe pastix wasn't building without them, though it appears to be now... Again those are currently going away with #4011, though it would probably be fine to remove them earlier than that.

[cpjordan]

The pip install helped a lot on the local Ubuntu 22.04 install I had working, though it was still slightly slower than the Docker image. Given the difficulty I've found trying to install PETSc on either Ubuntu 20.04 & 22.04, it's easier to just upgrade to 24.04 if possible and use Docker if not.

Ubuntu 24.04 LTS Docker image and local builds run the test in the same time.