Best way to make a venv path visible to cython

[orlitzky]

Let's jump right in to an XY problem: I think I want to be able to override cython with python /path/to/cython.

When mixing venv and system packages, we often call the system cython to build an extension module inside a venv. But this will fail if the extension module needs pxd files from some other package that was installed to the venv: cython gets its search path from python, but the shebang on cython points to the system python rather than the venv python, so when running "cython", we get the usual system search path and not one that includes the venv.

This is the wrong way to go about all this, but in a project with one pyx file, I'd like to just change the pyx compiler from cython to python /path/to/cython, reducing the problem to that of ensuring that "python" is the venv python.

[eli-schwartz]

Meson detects cython from $PATH, so if you have activated the venv rather than calling it via path/to/venv/bin/python this should already work out of the box.

Your other option is using https://mesonbuild.com/Machine-files.html to define:

[binaries]
cython = '/path/to/cython'

There is no need to specify python anywhere there. You could also do this:

[constants]
venv = '/path/to/venv/bin'

[binaries]
python = venv / 'python'
cython = venv / 'cython'

Thereby forcing meson to respect both python and cython (while running meson as a system tool using some unidentified different version of python).

[orlitzky]

I'm not sure what you want then.

You CANNOT run path/to/venv/python /usr/bin/cython as it makes no logical sense. It will not run cython using the venv python. It will fail to run at all, because cython is not installed for the venv.

It's more like path/to/venv/python $(which cython). If system-site-packages are disabled in the venv, then cython will be pulled into the venv (even if cython is installed on the system), and which will pick it up from the PATH. If system-site-packages are enabled, and if cython is not installed in the venv, then which will pick up the /usr/bin copy... but the /usr/bin copy can be executed with the venv python in that case.

you also have to have cython installed into the venv. At that point, you can just detect the cython installed in the venv.

It's all stupid if you ask me, but the python tooling makes it easy for users to create a situation where the system cython is used to build a package in the venv but cython itself bypasses the venv. I am trying my best to support whatever the degenerates who don't use system packages may do.

[eli-schwartz]

OK, that's... definitely an interesting one. :)

Either way, the machine file documentation points out you can specify binaries as an array instead of a string, so you can e.g. say that cython = ['path/to/venv/bin/python', '/usr/bin/cython']. And meson will use that command whenever it is asked to invoke cython.

Whether it works or not is up to the person setting up said environment -- meson is entirely indifferent to whether system-site-packages are enabled or not, it won't check, it will simply attempt to run the command you specified to see if it works.

Meson doesn't allow literally using $(command -v cython) in a cross file, you'd need to do that when generating the cross file.

(Aside: "command -v" is POSIX and guaranteed to exist verywhere. "which" happens to be in @system on Gentoo, but there's a ticket to remove it and depending on your distro it may not be installed at all on other distros. Reducing the amount of reliance on it is always a good thing.)

[orlitzky]

Oh, yeah, I know. I just use which when I'm out in the real world because otherwise no one will know what I'm talking about.

It's funny, all I'm trying to do is build one cython extension against another one, but I feel like I'm blazing a path through the uncharted wilderness. Meson didn't create any new problems of course but I was hoping it would be easy to just hack the cython command on-the-fly.

Easiest solution is probably to update the README and tell people to install cython in the venv if they want to use a venv.

[eli-schwartz]

It should perhaps be noted that meson also respects $CYTHON the same way it respects $CC.

[orlitzky]

I am already abusing this in sage to solve the problem 🙃

The same thing could work in the upstream package, but I'm pretty sure I don't want to ship this script with every cython package I write and tell my users to set it as CYTHON:

import os
import shutil
import sys

python = sys.executable
cython = shutil.which("cython")

args = sys.argv[1:]
args.insert(0, python)
args.insert(1, cython)
os.execv(python, args)

[eli-schwartz]

Despite all the talk about overriding the cython command, I wonder why cython --include-dir doesn't work?

[orlitzky]

IIRC that's for C/C++ headers.

The problem in this case is the pxd files needed for from foo cimport bar. Cython looks for foo/bar.pxd in the usual python search path, and that's OK, because that's where it's installed. It's all quite analogous to installing py files and then doing from foo import bar. Where cython faceplants is in a venv with system site packages enabled: running python -c 'from foo import bar' in a venv will use the venv search path, so it will pick up foo/bar.py inside the venv, no problem. Running cython however on a file that contains from foo cimport bar in a venv will not use the venv search path if cython is the system cython because it gets launched using the system python, with the system search path, and not the venv python/path. As a result, cython can't see anything installed in the venv.

It's only a problem if (a) your venv allows system site packages and (b) your cython package cimports cython dependencies. I guess no one else has tried this.

[eli-schwartz]

IIRC that's for C/C++ headers.

cythonize has "include_dirs" and "cython_include_dirs" attributes for setup.py which are distinctive from each other -- one for C/C++ headers, one for cython cimported pxds.

cython doesn't handle C/C++ at all, it only generates the transpiled pyx -> .c files. C/C++ headers would simply be added as #include rather than inlined, so there is no use for specifying the directory of C/C++ headers. The command line option here, is only for pxd imports.

[orlitzky]

Yeah it looks like you're right. I could swear I tried that 🤷

New plan: I'll change meson.build to search for "python" on PATH, and then run a crazy python -c command to output the contents of sys.path each prefixed by --include-dir.

[eli-schwartz]

the find_installation() call that is used for extension_module() and install_sources() should already be set to the python for the environment you're intending to install into (this is e.g. enforced by meson-python as it generates a machine file defining a python override pointing at the python command for the possibly-isolated build env).

You can directly use that in commands, e.g.

py = py_mod.find_installation()

# have it print --include-dir for convenience too
path_args = run_command(py, '-c', '.......').split('\n')

py.extension_module('foo'
    'foo.pyx',
    cython_args: path_args,
)

[orlitzky]

I'm pretty sure my first bug report was because find_installation() grabs the system python when meson-python comes from the system. In any case, I think I've got it working, the meson compile -v output looks right anyhow. Now I just have to test it in ten awkward scenarios.

[orlitzky]

One final hurdle in that meson adds its own quoting around arguments with spaces in them (even if they are already quoted), but I think we have a winner:

cython_include_args = run_command(
  py,
  [
    '-c',
    '''
import sys
for p in sys.path:
    if p:
        print("--include-dir")
        print(p)
    '''.strip()
  ],
  check: true
).stdout().splitlines()

py.extension_module(
  ...,
  cython_args : cython_include_args
)

Thanks for the assist.