BLD: A fresh clone without submodules fails to build inscrutably

[ngoldbaum]

It would be nice if the error message said something about initializing submodules and installing meson-python:

If meson-python is installed, we get this:

$  git clone git@github.com:numpy/numpy.git numpy-new
Cloning into 'numpy-new'...
remote: Enumerating objects: 270287, done.
remote: Counting objects: 100% (118/118), done.
remote: Compressing objects: 100% (84/84), done.
remote: Total 270287 (delta 63), reused 65 (delta 34), pack-reused 270169
Receiving objects: 100% (270287/270287), 137.08 MiB | 4.74 MiB/s, done.
Resolving deltas: 100% (213353/213353), done.

$ cd numpy-new

± python -m pip install -v . --no-build-isolation -Cbuilddir=build -C'compile-args=-v' -C'setup-args=-Dbuildtype=debug'
Using pip 24.0 from /Users/goldbaum/.pyenv/versions/3.13-dev-debug/lib/python3.13/site-packages/pip (python 3.13)
/Users/goldbaum/.pyenv/versions/3.13-dev-debug/lib/python3.13/site-packages/pip/_internal/metadata/importlib/_envs.py:111: DeprecationWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html
  from pip._vendor.pkg_resources import find_distributions
Processing /Users/goldbaum/Documents/numpy-new
  Running command Preparing metadata (pyproject.toml)

  meson-python: error: Could not find meson version 0.63.3 or newer, found .
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.

  note: This error originates from a subprocess, and is likely not a problem with pip.

If it isn't, we get an error about mesonpy being missing:

ERROR: Exception:
Traceback (most recent call last):
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-debug/lib/python3.13/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
    status = run_func(*args)
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-debug/lib/python3.13/site-packages/pip/_internal/cli/req_command.py", line 245, in wrapper
    return func(self, options, args)
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-debug/lib/python3.13/site-packages/pip/_internal/commands/install.py", line 377, in run
    requirement_set = resolver.resolve(
        reqs, check_supported_wheels=not options.target_dir
    )
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-debug/lib/python3.13/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 76, in resolve
    collected = self.factory.collect_root_requirements(root_reqs)
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-debug/lib/python3.13/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 534, in collect_root_requirements
    reqs = list(
        self._make_requirements_from_install_req(
    ...<2 lines>...
        )
    )
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-debug/lib/python3.13/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 490, in _make_requirements_from_install_req
    cand = self._make_base_candidate_from_link(
        ireq.link,
    ...<2 lines>...
        version=None,
    )
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-debug/lib/python3.13/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 228, in _make_base_candidate_from_link
    self._link_candidate_cache[link] = LinkCandidate(
                                       ~~~~~~~~~~~~~^
        link,
        ^^^^^
    ...<3 lines>...
        version=version,
        ^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-debug/lib/python3.13/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 290, in __init__
    super().__init__(
    ~~~~~~~~~~~~~~~~^
        link=link,
        ^^^^^^^^^^
    ...<4 lines>...
        version=version,
        ^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-debug/lib/python3.13/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
    self.dist = self._prepare()
                ~~~~~~~~~~~~~^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-debug/lib/python3.13/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 222, in _prepare
    dist = self._prepare_distribution()
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-debug/lib/python3.13/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 301, in _prepare_distribution
    return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-debug/lib/python3.13/site-packages/pip/_internal/operations/prepare.py", line 525, in prepare_linked_requirement
    return self._prepare_linked_requirement(req, parallel_builds)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-debug/lib/python3.13/site-packages/pip/_internal/operations/prepare.py", line 640, in _prepare_linked_requirement
    dist = _get_prepared_distribution(
        req,
    ...<3 lines>...
        self.check_build_deps,
    )
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-debug/lib/python3.13/site-packages/pip/_internal/operations/prepare.py", line 71, in _get_prepared_distribution
    abstract_dist.prepare_distribution_metadata(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        finder, build_isolation, check_build_deps
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-debug/lib/python3.13/site-packages/pip/_internal/distributions/sdist.py", line 67, in prepare_distribution_metadata
    self.req.prepare_metadata()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-debug/lib/python3.13/site-packages/pip/_internal/req/req_install.py", line 579, in prepare_metadata
    self.metadata_directory = generate_metadata(
                              ~~~~~~~~~~~~~~~~~^
        build_env=self.build_env,
        ^^^^^^^^^^^^^^^^^^^^^^^^^
        backend=self.pep517_backend,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        details=details,
        ^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-debug/lib/python3.13/site-packages/pip/_internal/operations/build/metadata.py", line 35, in generate_metadata
    distinfo_dir = backend.prepare_metadata_for_build_wheel(metadata_dir)
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-debug/lib/python3.13/site-packages/pip/_internal/utils/misc.py", line 766, in prepare_metadata_for_build_wheel
    return super().prepare_metadata_for_build_wheel(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        metadata_directory=metadata_directory,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        config_settings=cs,
        ^^^^^^^^^^^^^^^^^^^
        _allow_fallback=_allow_fallback,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-debug/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 186, in prepare_metadata_for_build_wheel
    return self._call_hook('prepare_metadata_for_build_wheel', {
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        'metadata_directory': abspath(metadata_directory),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        'config_settings': config_settings,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        '_allow_fallback': _allow_fallback,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    })
    ^^
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-debug/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_impl.py", line 321, in _call_hook
    raise BackendUnavailable(data.get('traceback', ''))
pip._vendor.pyproject_hooks._impl.BackendUnavailable: Traceback (most recent call last):
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-debug/lib/python3.13/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend
    obj = import_module(mod_path)
  File "/Users/goldbaum/.pyenv/versions/3.13-dev-debug/lib/python3.13/importlib/__init__.py", line 88, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1324, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'mesonpy'

[andyfaff]

Is there a reason why we don't have meson in build-requires?

[ngoldbaum]

Because it's vendored, so pip can't handle it for us. Unless there's a way to do that with a vendored submodule.

[rgommers]

Is there a reason why we don't have meson in build-requires?
Because it's vendored, so pip can't handle it for us

It still wouldn't be in build-requires if we didn't vendor it. It's only a transitive dependency; meson-python is our direct dependency and that depends on meson.

meson-python: error: Could not find meson version 0.63.3 or newer, found .

I can make this one more informative - needs to be done in meson-python.

ModuleNotFoundError: No module named 'mesonpy'

This cannot be improved in either numpy or meson-python, and is user error that you only see if you invoke pip with --no-build-isolation and are missing build requirements. It'd be nice for pip to say something more informative here like "build-backend specified in pyproject.toml is not importable" - but contributing to piptends to be a little painful, so I'll pass on that one.

It does already raise BackendUnavailable, so it's not that bad.

[ngoldbaum]

It does already raise BackendUnavailable, so it's not that bad.

Agreed, I was on the bubble of including this error but decided to. On second thought there's probably enough information, and you're right, using --no-build-isolation means I should be responsible for setting up my build environment.

The really inscrutable errors happens when you don't have meson-python installed, or if you try to force meson-python to use a version of meson that's not the vendored one. One of the CPython developers hit this error while trying to build scikit-learn to understand a segfault in the scikit-learn tests and got very confused.

[rgommers]

Fixed by mesonbuild/meson-python#631, will be available in meson-python 0.17.0