mesonpy editable installs

[crusaderky]

Problem description

Related: #5268

versioned-hdf5 is a Cython package that compiles against hdf5 (not just h5py) and uses mesonpy to manage the building process.

mesonpy's editable installs do not integrate nicely with pixi's suggested pattern to write

# Editable installs
[tool.pixi.pypi-dependencies]
pixi-py = { path = ".", editable = true }

There are multiple problems:

1. mesonpy creates a build directory, upon which the dummy wheel depends, named build/cp314 by default (replace with your python version). If I delete the temporary build directory, then my pixi environment becomes broken and I also have to clean the pixi env as well as the rattler-build cache.
2. This does not allow for variant builds. In my specific case, when you change minor versions of libhdf5 (1.10, 1.12, 1.14, 2.0) you need to recompile everything from source; however this won't happen until you manually delete build/cp314. TSAN and ASAN python variants is another example.
3. It's impossible to pass any parameters to python -m pip / python -m build. Namely, there is a mesonpy issue that causes an editable install to crash silently in case of any errors; one needs to pass the parameter "-Ceditable-verbose=true" as a workaround.

Workaround

Use a variant of workaround n.2 described in #5268. Change the pip command line in ci/editable_install.py to

    project_root = pathlib.Path(__file__).parent.parent
    build_dir = pathlib.Path(os.environ["CONDA_PREFIX"]) / "build"
    args = [
        sys.executable,
        "-m",
        "pip",
        "install",
        "--no-build-isolation",
        "--no-dependencies",
        "--editable",
        str(project_root),
        f"-Cbuild-dir={build_dir}",
        "-Ceditable-verbose=true",
    ]

This will create the meson build dir in $CONDA_PREFIX/build, which tightly couples it to the pixi environment, and also let you pass whatever other command line arguments you need.

Suggested design

Allow passing parameters to python -m pip or to python -m build:

# Editable installs
[tool.pixi.pypi-dependencies]
pixi-py = { path = ".", editable = true, build_args = ["-Cbuild_dir=$CONDA_PREFIX/build", "-Ceditable-verbose=true"] }

Note the environment variable resolution of $CONDA_PREFIX, which needs to be specific to the pixi env you're installing the package into; care must also be taken not to create a generic rattler-build cache just keyed to 'cp314'.

[Hofer-Julian]

Thanks for opening the issue @crusaderky!

We are currently hard at work to make use cases like yours work well with pixi-build. Could you try to adapt your setup similar to this maturin example: https://github.com/prefix-dev/pixi-build-testsuite/tree/main/tests/data/pixi_build/maturin

We'd be happy to fix problems that you encounter on that way.

[lucascolley]

Yeah I recommend following what I've done in data-apis/array-api-extra#583 Guido, seems to be working well

[crusaderky]

Sorry I don't underestand. I can't find anything in either example about passing command-line parameters to mesonpy?

[lucascolley]

https://prefix-dev.github.io/pixi-build-backends/backends/pixi-build-python/#extra-args

[lucascolley]

(maybe we should have that in array-api-extra too? please send a PR if so!)

[crusaderky]

crusaderky/versioned-hdf5#1

1. Can't make setuptools-scm work with pixi-build, please advise. I had to change

[project]
dynamic = ["version"]

to

[project]
version = "0.0.1"

2. Can't make mesonpy editable installs with cython compilation work. In the PR linked above, if I run pixi r smoke I can see that slicetools.pyx is being compiled, but then runtime crashes with

  File "/my-git-root/versioned-hdf5/versioned_hdf5/hashtable.py", line 10, in <module>
    from versioned_hdf5.slicetools import spaceid_to_slice
ModuleNotFoundError: No module named 'versioned_hdf5.slicetools'

/my-git-root is where I ran git clone for the project. With the previous version which the PR merges into, I was getting

>>> import versioned_hdf5
>>> versioned_hdf5.__file__
'/my-git-root/versioned-hdf5/versioned_hdf5/__init__.py'
>>> versioned_hdf5.slicetools.__file__
'/my-git-root/versioned-hdf5/.pixi/envs/default/build/versioned_hdf5/slicetools.cpython-314-x86_64-linux-gnu.so'

the above path was thanks to the -Cbuild-dir= parameter described in the opening post; without I would have gotten instead

>>> import versioned_hdf5
>>> versioned_hdf5.__file__
'/my-git-root/versioned-hdf5/versioned_hdf5/__init__.py'
>>> versioned_hdf5.slicetools.__file__
'/my-git-root/versioned-hdf5/build/cp314/versioned_hdf5/slicetools.cpython-314-x86_64-linux-gnu.so'

The linked PR doesn't produce /my-git-root/versioned-hdf5/versioned_hdf5/slicetools.*.so either, like it used to happen back in the day with python setup.py build_ext --inplace. I don't know where the .so file went; please advise?

3. I need the h5py runtime version to match the build version. You see for example I have environment hdf5-112 and py310, which both compile and run against hdf5=1.12 and hdf5=1.14 respectively. I guess I need to write a recipe.yaml and variants.yaml and incorporate the hdf5 version in the ABI tag? 😕

[lucascolley]

Can't make setuptools-scm work with pixi-build

tracked in gh-2946