Using poetry and meson together for a Python/Cython/C++ package

[adam2392]

I am trying to use poetry and meson as its backend. I am wondering if anyone has any working examples of this? +1 if they also use Cython and C++ builds.

What installation instructions would I need to use? Is it possible to do it with just poetry install, or do I need to do meson build; meson install?

SO post: https://stackoverflow.com/questions/75036296/poetry-meson-how-to-use-meson-as-a-build-backend-with-cython-files-to-install-a

[eli-schwartz]

Poetry is two things:

• a virtual environments manager, competitor to e.g. requirements.txt + venv, competitor to pipenv
• a build backend, competitor to setuptools/flit/meson-python

Which one are you trying to use it for? If you pip install . does that work, and produce a working install? Because if that does work, then IIRC poetry's virtual environments manager just uses pip internally and that should work as well.

[adam2392]

It works, but the Cython files are not registered/built.

(sktree) adam2392@Adams-MBP scikit-tree % pip install .
Processing /Users/adam2392/Documents/scikit-tree
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: scikit_tree
  Building wheel for scikit_tree (pyproject.toml) ... done
  Created wheel for scikit_tree: filename=scikit_tree-0.0.0.dev0-cp39-cp39-macosx_13_0_arm64.whl size=191667 sha256=21c8f1ea2230f74a71f9dc842cbc0bef5a640c43216e79507ea1f0cb98c131e0
  Stored in directory: /Users/adam2392/Library/Caches/pip/wheels/04/0b/eb/5bf45d99f396bab0ff88a00a64fb349013242494ef712c3ddd
Successfully built scikit_tree
Installing collected packages: scikit_tree
Successfully installed scikit_tree-0.0.0.dev0
(sktree) adam2392@Adams-MBP scikit-tree % python -c "from sktree import tree"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/Users/adam2392/Documents/scikit-tree/sktree/__init__.py", line 37, in <module>
    from . import tree
  File "/Users/adam2392/Documents/scikit-tree/sktree/tree/__init__.py", line 1, in <module>
    from ._classes import UnsupervisedDecisionTree
  File "/Users/adam2392/Documents/scikit-tree/sktree/tree/_classes.py", line 3, in <module>
    from ._unsup_criterion import UnsupervisedCriterion
ModuleNotFoundError: No module named 'sktree.tree._unsup_criterion'

I am trying to use poetry as the virtual env manager. I want to use meson to replace the build backend because traditionally I used something like numpy.distutils and followed scikit-learn's usage of setuptools/distutils, which IIUC is being phased out.

For reference, the code I'm trying to get work is on this branch: neurodata/treeple#8

[eli-schwartz]

  File "/Users/adam2392/Documents/scikit-tree/sktree/tree/_classes.py", line 3, in <module>
    from ._unsup_criterion import UnsupervisedCriterion

Since you ran python -c inside the root directory of the scikit-tree repository, and python by default adds the current working directory to the sys.path for an interactive interpreter or when using -c or -m, this attempted to import the source code instead of the installed code.

Some projects choose to have all the project code live inside a src/ directory for this reason. The other solution is to simply cd .. before trying to import it.

Installing meson-python from git adds support for editable installs. pip install -e . -- this actually injects an import hook that has higher priority than the version in the source tree, so that "just works" anyway.

[eli-schwartz]

SciPy is clever about this: https://github.com/scipy/scipy/blob/fdc32b8bb95d4c5e101c3d1beb321663cba0b85e/scipy/__init__.py#L116-L127

if __SCIPY_SETUP__:
    import sys
    sys.stderr.write('Running from SciPy source directory.\n')
    del sys
else:
    try:
        from scipy.__config__ import show as show_config
    except ImportError as e:
        msg = """Error importing SciPy: you cannot import SciPy while
        being in scipy source directory; please exit the SciPy source
        tree first and relaunch your Python interpreter."""
        raise ImportError(msg) from e