Install not working on Mac

[AlbanBloorRiley]

I have been trying to install Firedrake using homebrew on my Mac (Arm 13.5.2) but it hasn't been working. I have gone through all the common steps trying to fix the problem - I have uninstalled anaconda, and installed python using homebrew also, but it still doesn't work.

The print out of the terminal is:
% python3 firedrake-install
/Users/user/firedrake-install:20: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
from pkg_resources.extern.packaging.version import Version, InvalidVersion
Running firedrake-install
Installing command line tools...
Checking brew doctor...
Installing required packages via homebrew...
Required package 'gcc' is already installed via Homebrew.
Required package 'autoconf' is already installed via Homebrew.
Required package 'make' is already installed via Homebrew.
Required package 'automake' is already installed via Homebrew.
Required package 'cmake' is already installed via Homebrew.
Required package 'libtool' is already installed via Homebrew.
Required package 'boost' is already installed via Homebrew.
Required package 'openblas' is already installed via Homebrew.
Creating firedrake venv in '/Users/user/firedrake'.
Traceback (most recent call last):
File "/Users/user/firedrake-install", line 1719, in
venv.EnvBuilder(with_pip=with_pip).create(firedrake_env)
File "/opt/homebrew/Cellar/[email protected]/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/venv/init.py", line 76, in create
self._setup_pip(context)
File "/opt/homebrew/Cellar/[email protected]/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/venv/init.py", line 358, in _setup_pip
self._call_new_python(context, '-m', 'ensurepip', '--upgrade',
File "/opt/homebrew/Cellar/[email protected]/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/venv/init.py", line 354, in _call_new_python
subprocess.check_output(args, **kwargs)
File "/opt/homebrew/Cellar/[email protected]/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/subprocess.py", line 466, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Cellar/[email protected]/3.11.5/Frameworks/Python.framework/Versions/3.11/lib/python3.11/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/Users/user/firedrake/bin/python3.11', '-m', 'ensurepip', '--upgrade', '--default-pip']' died with <Signals.SIGABRT: 6>.

and the log is:
firedrake-install.log

Interestingly using the Mac python the installation gets further - to building petsc but still breaks.

[Ig-dolci]

Perhaps this install way can help you.

[dthillaithevan]

I am also using a Mac and I am getting this exact same error message when trying to install firedrake

[connorjward]

There appears to be an issue with your compilers. Can you please make sure that your command line tools and homebrew are up to date and then see if you can compile and run a simple hello world example in C.

You should also definitely continue to use pyenv. Homebrew and MacOS Pythons are both broken in different ways.

[AlbanBloorRiley]

They are all up to date, and I can compile and run a basic hello world C file. Whats confusing is using pyenv still leads to ssh failing, but I thought that was only a problem for the system python?

On the install website it says that using homebrew python is supported and tested - is that no longer the case?

[dthillaithevan]

@connorjward - I've updated homebrew and commandline tools are up to date. I have also been able to compile and run simple hello world examples in C and C++ without any errors. Do you know if there is anything else I can try?

[connorjward]

@AlbanBloorRiley please can you try running firedrake-install passing the flag --package-manager petsc ksagiyam/merge_upstream. Make sure that you are using a very recently downloaded installation script.

I am not sure what you mean by "leads to ssh failing". If you do not have the right SSH keys configured you will see lots of "cloning with SSH failed, falling back to HTTPS" messages. These are nothing to worry about.

The latest version of Python from Homebrew has an issue causing it to break firedrake-install, hence the pyenv recommendation. I intend to update the website ASAP.

@dthillaithevan we can continue the discussion of your issue on #3123.

[Ig-dolci]

I had the same Petsc error here. Then I tried to Petsc install with the merge_upstream, which sort it out.

[AlbanBloorRiley]

Whats more surprising is that I already uninstalled the homebrew PETSc

[connorjward]

Can you try deleting the petsc4py directory inside $VIRTUAL_ENV/lib/python3.11/site-packages? Also delete $PETSC_DIR/src/binding/petsc4py/build.

[AlbanBloorRiley]

So that has fixed the petsc4py problem, but now PETSc seems to have the same issue.

[connorjward]

I wonder if a fresh install, now that the homebrew packages have been expunged, would work. Clearly homebrew has some invasive path magic that makes things link against it even with LDFLAGS set.

[AlbanBloorRiley]

Yeah that's sort of what I was thinking... I will try that now. Thanks

[AlbanBloorRiley]

So I am sorry to say that the test still isn't running... its fixed the previous problems, but now I get this:

(firedrake) p97026ab@E-LOSX0GQ23 ~ % cd $VIRTUAL_ENV/src/firedrake
pytest tests/regression/ -k "poisson_strong or stokes_mini or dg_advection"
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
============================= test session starts ==============================
platform darwin -- Python 3.11.5, pytest-7.4.2, pluggy-1.3.0
rootdir: /Users/user/firedrake/src/firedrake
configfile: setup.cfg
plugins: anyio-4.0.0, nbval-0.10.0, mpi-0.1, xdist-3.3.1
collecting ... Fatal Python error: Aborted

Current thread 0x00000001e977e080 (most recent call first):
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1241 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1232 in _handle_fromlist
  File "/Users/user/firedrake/lib/python3.11/site-packages/h5py/__init__.py", line 56 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 1241 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1232 in _handle_fromlist
  File "/Users/user/firedrake/src/firedrake/firedrake/checkpointing.py", line 7 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "/Users/user/firedrake/src/firedrake/firedrake/__init__.py", line 74 in <module>
  File "<frozen importlib._bootstrap>", line 241 in _call_with_frames_removed
  File "<frozen importlib._bootstrap_external>", line 940 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "/Users/user/firedrake/src/firedrake/tests/regression/test_2dcohomology.py", line 15 in <module>
  File "/Users/user/firedrake/lib/python3.11/site-packages/_pytest/assertion/rewrite.py", line 178 in exec_module
  File "<frozen importlib._bootstrap>", line 690 in _load_unlocked
  File "<frozen importlib._bootstrap>", line 1147 in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 1176 in _find_and_load
  File "<frozen importlib._bootstrap>", line 1204 in _gcd_import
  File "/Users/user/.pyenv/versions/3.11.5/lib/python3.11/importlib/__init__.py", line 126 in import_module
  File "/Users/user/firedrake/lib/python3.11/site-packages/_pytest/pathlib.py", line 567 in import_path
  File "/Users/user/firedrake/lib/python3.11/site-packages/_pytest/python.py", line 617 in _importtestmodule
  File "/Users/user/firedrake/lib/python3.11/site-packages/_pytest/python.py", line 528 in _getobj
  File "/Users/user/firedrake/lib/python3.11/site-packages/_pytest/python.py", line 310 in obj
  File "/Users/user/firedrake/lib/python3.11/site-packages/_pytest/python.py", line 545 in _inject_setup_module_fixture
  File "/Users/user/firedrake/lib/python3.11/site-packages/_pytest/python.py", line 531 in collect
  File "/Users/user/firedrake/lib/python3.11/site-packages/_pytest/runner.py", line 372 in <lambda>
  File "/Users/user/firedrake/lib/python3.11/site-packages/_pytest/runner.py", line 341 in from_call
  File "/Users/user/firedrake/lib/python3.11/site-packages/_pytest/runner.py", line 372 in pytest_make_collect_report
  File "/Users/user/firedrake/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/Users/user/firedrake/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/Users/user/firedrake/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/Users/user/firedrake/lib/python3.11/site-packages/_pytest/runner.py", line 547 in collect_one_node
  File "/Users/user/firedrake/lib/python3.11/site-packages/_pytest/main.py", line 842 in genitems
  File "/Users/user/firedrake/lib/python3.11/site-packages/_pytest/main.py", line 675 in perform_collect
  File "/Users/user/firedrake/lib/python3.11/site-packages/_pytest/main.py", line 335 in pytest_collection
  File "/Users/user/firedrake/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/Users/user/firedrake/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/Users/user/firedrake/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/Users/user/firedrake/lib/python3.11/site-packages/_pytest/main.py", line 324 in _main
  File "/Users/user/firedrake/lib/python3.11/site-packages/_pytest/main.py", line 271 in wrap_session
  File "/Users/user/firedrake/lib/python3.11/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main
  File "/Users/user/firedrake/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/Users/user/firedrake/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/Users/user/firedrake/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/Users/user/firedrake/lib/python3.11/site-packages/_pytest/config/__init__.py", line 169 in main
  File "/Users/user/firedrake/lib/python3.11/site-packages/_pytest/config/__init__.py", line 192 in console_main
  File "/Users/user/firedrake/bin/pytest", line 8 in <module>

Extension modules: zmq.backend.cython.context, zmq.backend.cython.message, zmq.backend.cython.socket, zmq.backend.cython._device, zmq.backend.cython._poll, zmq.backend.cython._proxy_steerable, zmq.backend.cython._version, zmq.backend.cython.error, zmq.backend.cython.utils, tornado.speedups, _pydevd_bundle.pydevd_cython, mpi4py.MPI, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, petsc4py.PETSc, pvectorc, pyop2.sparsity, symengine.lib.symengine_wrapper, firedrake.cython.dmcommon, firedrake.cython.extrusion_numbering, firedrake.cython.spatialindex, firedrake.cython.patchimpl, firedrake.cython.hdf5interface, h5py._errors, h5py.defs, h5py._objects, h5py.h5, h5py.h5r, h5py.utils, h5py.h5s, h5py.h5ac, h5py.h5p, h5py.h5t, h5py._conv, h5py.h5z, h5py._proxy, h5py.h5a, h5py.h5d, h5py.h5ds, h5py.h5f, h5py.h5g, h5py.h5i (total: 53)
zsh: abort      pytest tests/regression/ -k "poisson_strong or stokes_mini or dg_advection"

[AlbanBloorRiley]

By the way do I still have to install firedrake using firedrake-install --package-branch petsc ksagiyam/merge_upstream --with-blas=download ?

[connorjward]

Yes please. And continue to use pyenv.

[AlbanBloorRiley]

Ok, well I am trying a clean install now. But if that doesn't work I think I will have to try the VS Code Docker approach.

[AlbanBloorRiley]

Unfortunately reinstalling homebrew doesn't seem to have helped, still fails those three tests.

Was the PETSc configure log useful - or was it just wrong?

If you don't think there is anything else to do then I guess I will just go with the docker route..

[connorjward]

I think Docker is the best approach here.

[JDBetteridge]

Just wondering if you have tried reinstalling recently? As of this merged PR I believe that the MacOS install issues have been fixed.