Failing the installation tests on Mac

[hwoo749]

Hi there,

I'm using Homebrew Python 3.11.3 on macOS 12.15.1.

After installing firedrake and running the test installation command: pytest tests/regression/ -k "poisson_strong or stokes_mini or dg_advection", I get the following errors.

'''
(firedrake) hw@wifi-452-91-423-678 firedrake % pytest tests/regression/ -k "poisson_strong or stokes_mini or dg_advection"

========================================================== test session starts ==========================================================
platform darwin -- Python 3.11.3, pytest-7.3.1, pluggy-1.0.0
benchmark: 4.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /Users/hw/firedrake/src/firedrake
configfile: setup.cfg
plugins: anyio-3.6.2, benchmark-4.0.0
collected 5 items / 154 errors
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR> File "/Users/hw/firedrake/lib/python3.11/site-packages/_pytest/main.py", line 269, in wrap_session
INTERNALERROR> session.exitstatus = doit(config, session) or 0
INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/Users/hw/firedrake/lib/python3.11/site-packages/_pytest/main.py", line 322, in _main
INTERNALERROR> config.hook.pytest_collection(session=session)
INTERNALERROR> File "/Users/hw/firedrake/lib/python3.11/site-packages/pluggy/_hooks.py", line 265, in call
INTERNALERROR> return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/Users/hw/firedrake/lib/python3.11/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/Users/hw/firedrake/lib/python3.11/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR> return outcome.get_result()
INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/Users/hw/firedrake/lib/python3.11/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR> raise ex[1].with_traceback(ex[2])
INTERNALERROR> File "/Users/hw/firedrake/lib/python3.11/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR> res = hook_impl.function(*args)
INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/Users/hw/firedrake/lib/python3.11/site-packages/_pytest/main.py", line 333, in pytest_collection
INTERNALERROR> session.perform_collect()
INTERNALERROR> File "/Users/hw/firedrake/lib/python3.11/site-packages/_pytest/main.py", line 668, in perform_collect
INTERNALERROR> hook.pytest_collection_modifyitems(
INTERNALERROR> File "/Users/hw/firedrake/lib/python3.11/site-packages/pluggy/_hooks.py", line 265, in call
INTERNALERROR> return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/Users/hw/firedrake/lib/python3.11/site-packages/pluggy/_manager.py", line 80, in _hookexec
INTERNALERROR> return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/Users/hw/firedrake/lib/python3.11/site-packages/pluggy/_callers.py", line 60, in _multicall
INTERNALERROR> return outcome.get_result()
INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/Users/hw/firedrake/lib/python3.11/site-packages/pluggy/_result.py", line 60, in get_result
INTERNALERROR> raise ex[1].with_traceback(ex[2])
INTERNALERROR> File "/Users/hw/firedrake/lib/python3.11/site-packages/pluggy/_callers.py", line 39, in _multicall
INTERNALERROR> res = hook_impl.function(*args)
INTERNALERROR> ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> File "/Users/hw/firedrake/src/firedrake/tests/conftest.py", line 20, in pytest_collection_modifyitems
INTERNALERROR> from firedrake.utils import complex_mode, SLATE_SUPPORTS_COMPLEX
INTERNALERROR> ModuleNotFoundError: No module named 'firedrake'

'''

After trying this command: python -m pytest tests/regression/ -k "poisson_strong or stokes_mini or dg_advection"

The final error message says module not found for petsc4py rather than firedrake : INTERNALERROR> import petsc4py
INTERNALERROR> ModuleNotFoundError: No module named 'petsc4py'

I have installed PETSc and uninstalled anaconda - how do I go about passing the tests and fixing this issue?

Thanks in advance.

[dham]

We should first check that your Firedrake install succeeded. Please post firedrake-install.log, which you will find next to the top firedrake folder on your computer. I.e. /Users/hw/firedrake-install.log

[dham]

Firedrake will build its own PETSc unless you tell it not to (and you probably don't want to do that). It won't be affected by any other PETSc you have installed.

This time your Firedrake install failed because you used the Python that ships with your Mac rather than installing python using homebrew. This can be seen from paths like this one: /Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/urllib/request.py which start with /Library/Frameworks/Python.framework rather than something like /opt/homebrew/bin/python3. This fails because the Mac Python has a broken ssl and hence can't download files over the network. You should ensure you have homebrew Python installed and are using that.

[hwoo749]

Hi there, I removed and then reinstalled Python through homebrew, then tried the firedrake install again. This time the PETSc install took a day to install which seemed promising, but I still ended up with the same errors. I will post the errors and install log.

I got these errors at the end of the PETSc install:

'''
Installing pip dependencies for firedrake
Traceback (most recent call last):
File "/Users/harrietwoodward/firedrake-install", line 1898, in
pip_requirements(p, compiler_env)
File "/Users/harrietwoodward/firedrake-install", line 1001, in pip_requirements
run_pip_install_wrap(reqs, parallel_compiler_env)
File "/Users/harrietwoodward/firedrake-install", line 994, in run_pip_install_wrap
run_pip_install([req])
File "/Users/harrietwoodward/firedrake-install", line 975, in run_pip_install
check_call(pipinstall + pipargs)
File "/Users/harrietwoodward/firedrake-install", line 663, in check_call
log.debug(subprocess.check_output(arguments, stderr=subprocess.STDOUT, env=os.environ).decode())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Cellar/[email protected]/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/subprocess.py", line 466, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/Cellar/[email protected]/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/Users/harrietwoodward/firedrake/bin/python', '-m', 'pip', 'install', '--no-build-isolation', '--no-binary', 'mpi4py,randomgen,islpy,numpy', '-vvv', 'pytest-xdist']' returned non-zero exit status 2.

'''

Then before the testing I got this message which I haven't gotten before:
'''
0.00s - Debugger warning: It seems that frozen modules are being used, which may
0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off
0.00s - to python to disable frozen modules.
0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation.
'''

Here is the install log:

firedrake-install.log

[connorjward]

The log is showing errors like:

pip._vendor.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='files.pythonhosted.org', port=443): Read timed out.

and

TimeoutError: The read operation timed out

It seems like you were installing with a flaky internet connection. Trying again may do the trick.

I also notice that brew doctor is raising a number of warnings. It would be good to resolve these too.

[hwoo749]

Hello, thank you for your help, I followed your instructions, and this install was definitely more successful! The only error I'm left with is this one:

========================================== ERRORS ==========================================
_____________________ ERROR collecting tests/regression/test_netgen.py _____________________
ImportError while importing test module '/Users/harrietwoodward/firedrake/src/firedrake/tests/regression/test_netgen.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/local/Cellar/[email protected]/3.11.3/Frameworks/Python.framework/Versions/3.11/lib/python3.11/importlib/init.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/regression/test_netgen.py:2: in
from netgen.geom2d import SplineGeometry
E ModuleNotFoundError: No module named 'netgen'
================================= short test summary info ==================================
ERROR tests/regression/test_netgen.py
!!!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!!!
============================ 3121 deselected, 1 error in 55.72s ============================
(firedrake) harrietwoodward@Harriets-MacBook-Air firedrake %

How would I go about fixing this?

[connorjward]

Netgen is an optional dependency of Firedrake. Unless you passed --netgen to firedrake-install this is expected behaviour.