Installing Firedrake with alternate PETSc version

[cpjordan]

I've been having issues with checkpointing (on Thetis) with my latest Firedrake install and I think the latest PETSc version is probably the issue. My latest working version had PETSc 3.20.2 and I had no issues, my newest Firedrake install had PETSc 3.20.5. How can I install with PETSc 3.20.2 to verify this is the issue?

I had tried python3 firedrake-install --package-branch petsc4py 3.20.2 but this doesn't work as I'm guessing it's dealt with internally during the install - make.log shows it install 3.20.5 anyway.

firedrake-install.log
configure.log
make.log

[dham]

The package-branch option modifies what git branch is installed, which is petsc, rather than petsc4py. Now, the next issue is that there isn't a tag in our PETSc mirror for release 3.20.2 (or any other release). So your options are:

1. Work out which commit you want to build petsc at and pass that to --package-branch
2. Build upstream PETSc yourself, set the PETSC_DIR and PETSC_ARCH variables, and pass --honour-petsc-dir to the install script. If you adopt this approach (which is the preferable one), you should run: python firedrake-install --show-petsc-configure-options in order to see the list of configure options you need to pass in to PETSc.

[cpjordan]

Thanks for your quick response @dham! I have tried both options and still have issues with both...

1. Works initially, but then I later have issues with cmake and so have to do firedrake-update - which tries to update PETSc, which defeats the purpose of what I am trying to do. Is there a flag to pass to prevent updating of specified packages?
2. I have followed these instructions, tested that PETSc passes tests, and then built cmake to solve issue 3374 and updated using:

cd $VIRTUAL_ENV/src/libsupermesh/build
/home/s1626662/petsc/arch-linux-c-op/bin/cmake .. \
  -DBUILD_SHARED_LIBS=ON \
  -DCMAKE_INSTALL_PREFIX=/home/s1626662/petsc/arch-linux-c-op \
  -DMPI_C_COMPILER=/home/s1626662/petsc/arch-linux-c-op/bin/mpicc \
  -DMPI_CXX_COMPILER=/home/s1626662/petsc/arch-linux-c-op/bin/mpicxx \
  -DMPI_Fortran_COMPILER=/home/s1626662/petsc/arch-linux-c-op/bin/mpif90 \
  -DCMAKE_Fortran_COMPILER=/home/s1626662/petsc/arch-linux-c-op/bin/mpif90 \
  -DMPIEXEC_EXECUTABLE=/home/s1626662/petsc/arch-linux-c-op/bin/mpiexec
make
make install
firedrake-update --honour-petsc-dir

Then when I run the tests, they all fail for various PETSc reasons e.g. AttributeError: 'petsc4py.PETSc.DMPlex' object has no attribute 'getCoordinatesLocalized'. Here are the logs:

firedrake-install.log
configure.log
make.log
firedrake-update.log

[dham]

OK so this is telling you that the PETSc you are trying to use is too old for the current Firedrake, because current Firedrake is accessing features (such as that attribute) that were added to PETSc later.

[cpjordan]

So if I checkout a version of Firedrake from Jan. 15th (last working install date) and then run the tests, I firstly found that I hadn't exported my PETSC_DIR and PETSC_ARCH variables post-install:

INTERNALERROR> ImportError: Firedrake was installed with --honour-petsc-dir, but PETSC_DIR is not set.
INTERNALERROR> Please set PETSC_DIR (and PETSC_ARCH) before using Firedrake.

So once done, I am still getting major issues:

pytest tests/regression/ -k "poisson_strong or stokes_mini or dg_advection"
=========================================================================================== test session starts ===========================================================================================
platform linux -- Python 3.8.10, pytest-8.1.1, pluggy-1.4.0
rootdir: /home/s1626662/firedrake/src/firedrake
configfile: setup.cfg
plugins: anyio-4.3.0, xdist-3.5.0, mpi-0.1, nbval-0.11.0
collected 5 items / 164 errors                                                                                                                                                                            
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/s1626662/firedrake/lib/python3.8/site-packages/_pytest/main.py", line 285, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/home/s1626662/firedrake/lib/python3.8/site-packages/_pytest/main.py", line 338, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "/home/s1626662/firedrake/lib/python3.8/site-packages/pluggy/_hooks.py", line 501, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>   File "/home/s1626662/firedrake/lib/python3.8/site-packages/pluggy/_manager.py", line 119, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/home/s1626662/firedrake/lib/python3.8/site-packages/pluggy/_callers.py", line 138, in _multicall
INTERNALERROR>     raise exception.with_traceback(exception.__traceback__)
INTERNALERROR>   File "/home/s1626662/firedrake/lib/python3.8/site-packages/pluggy/_callers.py", line 121, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>   File "/home/s1626662/firedrake/lib/python3.8/site-packages/_pytest/logging.py", line 793, in pytest_collection
INTERNALERROR>     return (yield)
INTERNALERROR>   File "/home/s1626662/firedrake/lib/python3.8/site-packages/pluggy/_callers.py", line 121, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>   File "/home/s1626662/firedrake/lib/python3.8/site-packages/_pytest/warnings.py", line 120, in pytest_collection
INTERNALERROR>     return (yield)
INTERNALERROR>   File "/home/s1626662/firedrake/lib/python3.8/site-packages/pluggy/_callers.py", line 121, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>   File "/home/s1626662/firedrake/lib/python3.8/site-packages/_pytest/config/__init__.py", line 1415, in pytest_collection
INTERNALERROR>     return (yield)
INTERNALERROR>   File "/home/s1626662/firedrake/lib/python3.8/site-packages/pluggy/_callers.py", line 102, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/s1626662/firedrake/lib/python3.8/site-packages/_pytest/main.py", line 349, in pytest_collection
INTERNALERROR>     session.perform_collect()
INTERNALERROR>   File "/home/s1626662/firedrake/lib/python3.8/site-packages/_pytest/main.py", line 816, in perform_collect
INTERNALERROR>     hook.pytest_collection_modifyitems(
INTERNALERROR>   File "/home/s1626662/firedrake/lib/python3.8/site-packages/pluggy/_hooks.py", line 501, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>   File "/home/s1626662/firedrake/lib/python3.8/site-packages/pluggy/_manager.py", line 119, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/home/s1626662/firedrake/lib/python3.8/site-packages/pluggy/_callers.py", line 138, in _multicall
INTERNALERROR>     raise exception.with_traceback(exception.__traceback__)
INTERNALERROR>   File "/home/s1626662/firedrake/lib/python3.8/site-packages/pluggy/_callers.py", line 121, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>   File "/home/s1626662/firedrake/lib/python3.8/site-packages/_pytest/cacheprovider.py", line 414, in pytest_collection_modifyitems
INTERNALERROR>     res = yield
INTERNALERROR>   File "/home/s1626662/firedrake/lib/python3.8/site-packages/pluggy/_callers.py", line 121, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>   File "/home/s1626662/firedrake/lib/python3.8/site-packages/_pytest/cacheprovider.py", line 342, in pytest_collection_modifyitems
INTERNALERROR>     res = yield
INTERNALERROR>   File "/home/s1626662/firedrake/lib/python3.8/site-packages/pluggy/_callers.py", line 102, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/s1626662/firedrake/src/firedrake/tests/conftest.py", line 26, in pytest_collection_modifyitems
INTERNALERROR>     from firedrake.utils import complex_mode, SLATE_SUPPORTS_COMPLEX
INTERNALERROR>   File "/home/s1626662/firedrake/src/firedrake/firedrake/__init__.py", line 19, in <module>
INTERNALERROR>     import firedrake.petsc as petsc
INTERNALERROR>   File "/home/s1626662/firedrake/src/firedrake/firedrake/petsc.py", line 19, in <module>
INTERNALERROR>     petsc4py.init(sys.argv)
INTERNALERROR>   File "/home/s1626662/firedrake/lib/python3.8/site-packages/petsc4py/__init__.py", line 41, in init
INTERNALERROR>     PETSc = petsc4py.lib.ImportPETSc(arch)
INTERNALERROR>   File "/home/s1626662/firedrake/lib/python3.8/site-packages/petsc4py/lib/__init__.py", line 29, in ImportPETSc
INTERNALERROR>     return Import('petsc4py', 'PETSc', path, arch)
INTERNALERROR>   File "/home/s1626662/firedrake/lib/python3.8/site-packages/petsc4py/lib/__init__.py", line 95, in Import
INTERNALERROR>     module = import_module(pkg, name, path, arch)
INTERNALERROR>   File "/home/s1626662/firedrake/lib/python3.8/site-packages/petsc4py/lib/__init__.py", line 71, in import_module
INTERNALERROR>     module = importlib.util.module_from_spec(spec)
INTERNALERROR>   File "<frozen importlib._bootstrap>", line 553, in module_from_spec
INTERNALERROR> AttributeError: 'NoneType' object has no attribute 'loader'

=========================================================================================== 164 errors in 3.27s ===========================================================================================

I'm not sure if at this stage, other dependencies also had moved on to the later PETSc version perhaps? I have the list of installs on my other machine that all currently work, but it seems that it might be quite a lot of work to build everything from scratch at the correct version/commit?

[dham]

That's PETSc4py failing initialisation, which is probably indicative that your PETSc4py didn't build cleanly.

This does, though, sound like a painful way to approach the problem.