Building and testing Python packages out of tree for Python package with C++ code produces error saying a required module (numpy) is not found.

[aya49]

🐛 Bug

I am in WSL2, trying to build a Python package (FlowUtils) out of tree for Pyodide because it has C++ scripts in the package: https://pyodide.org/en/stable/development/building-and-testing-packages.html#building-and-testing-packages-out-of-tree

After following the instructions in a virtual environment, I get into the Package directory and run pyodide build, it gives me an error saying that I don't have numpy, which I do.

I also tried building the package normally, and it works. The website said if that's the case, then to open an issue, so here it is! Am I doing something wrong?

Note: I also tried building the package in the tree but it also fails when I try to build pyodide build-recipes flowutils --install, failing at the first package... that's out of scope here, but this is why I tried building out of tree: https://pyodide.org/en/stable/development/new-packages.html

To Reproduce

Pre-requisite: I've already installed Python packages numpy and pyodide-dev. I also have python3-dev installed.

In Windows 11 WSL2 Ubuntu 22.02 terminal:

git clone https://github.com/emscripten-core/emsdk.git
cd emsdk  
PYODIDE_EMSCRIPTEN_VERSION=$(pyodide config get emscripten_version)
./emsdk install ${PYODIDE_EMSCRIPTEN_VERSION}
./emsdk activate ${PYODIDE_EMSCRIPTEN_VERSION}
source emsdk_env.sh
cd ..
git clone https://github.com/whitews/FlowUtils.git
cd FlowUtils
pyodide build

Expected behavior

From the website:

"pyodide build invokes a slightly modified version of the pypa/build build frontend so the behavior should be similar to what happens if you do:"

pip install build
python -m build

"If you run into problems, make sure that building a native wheel with pypa/build works."

The packages builds normally when using pypa/build. But it shows and error when I use Pyodide's build (output below). I think the latter is suppose to build normally too but it doesn't.

Environment

• Pyodide Version: 0.26.3
• Any other relevant information: Python 3.10.12

• Commit hash of Pyodide git repository: Add Narwhals to package list pyodide#5121
• Build environment<!--(e.g. Ubuntu 22.04)- ->: Windows 11, WSL2 Ubuntu 22.04

Additional context

Error:

(.venv) user@user:/mnt/d/schoolwork/projects/FlowUtils$ pyodide build
xbuild environment already exists, skipping download
Installing xbuild environment
Traceback (most recent call last):
  File "/mnt/d/schoolwork/projects/test-app/.venv/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 389, in <module>
    main()
  File "/mnt/d/schoolwork/projects/test-app/.venv/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 373, in main
    json_out["return_val"] = hook(**hook_input["kwargs"])
  File "/mnt/d/schoolwork/projects/test-app/.venv/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 143, in get_requires_for_build_wheel
    return hook(config_settings)
  File "/tmp/build-env-6rk2k36w/lib/python3.10/site-packages/setuptools/build_meta.py", line 332, in get_requires_for_build_wheel
    return self._get_build_requires(config_settings, requirements=[])
  File "/tmp/build-env-6rk2k36w/lib/python3.10/site-packages/setuptools/build_meta.py", line 302, in _get_build_requires
    self.run_setup()
  File "/tmp/build-env-6rk2k36w/lib/python3.10/site-packages/setuptools/build_meta.py", line 318, in run_setup
    exec(code, locals())
  File "<string>", line 5, in <module>
ModuleNotFoundError: No module named 'numpy'
Traceback (most recent call last):
  File "/mnt/d/schoolwork/projects/test-app/.venv/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 389, in <module>
    main()
  File "/mnt/d/schoolwork/projects/test-app/.venv/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 373, in main
    json_out["return_val"] = hook(**hook_input["kwargs"])
  File "/mnt/d/schoolwork/projects/test-app/.venv/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 143, in get_requires_for_build_wheel
    return hook(config_settings)
  File "/tmp/build-env-6rk2k36w/lib/python3.10/site-packages/setuptools/build_meta.py", line 332, in get_requires_for_build_wheel
    return self._get_build_requires(config_settings, requirements=[])
  File "/tmp/build-env-6rk2k36w/lib/python3.10/site-packages/setuptools/build_meta.py", line 302, in _get_build_requires
    self.run_setup()
  File "/tmp/build-env-6rk2k36w/lib/python3.10/site-packages/setuptools/build_meta.py", line 318, in run_setup
    exec(code, locals())
  File "<string>", line 5, in <module>
ModuleNotFoundError: No module named 'numpy'

ERROR Backend subprocess exited when trying to invoke get_requires_for_build_wheel

(.venv) user@user:/mnt/d/schoolwork/projects/FlowUtils$ python --version
Python 3.10.12
(.venv) user@user:/mnt/d/schoolwork/projects/FlowUtils$ pip install numpy
Requirement already satisfied: numpy in /mnt/d/schoolwork/projects/test-app/.venv/lib/python3.10/site-packages (2.1.2)

When building normally:

(.venv) user@user:/mnt/d/schoolwork/projects/FlowUtils$ pip install build
Requirement already satisfied: build in /mnt/d/schoolwork/projects/test-app/.venv/lib/python3.10/site-packages (1.2.2.post1)
Requirement already satisfied: packaging>=19.1 in /mnt/d/schoolwork/projects/test-app/.venv/lib/python3.10/site-packages (from build) (24.1)
Requirement already satisfied: pyproject_hooks in /mnt/d/schoolwork/projects/test-app/.venv/lib/python3.10/site-packages (from build) (1.2.0)
Requirement already satisfied: tomli>=1.1.0 in /mnt/d/schoolwork/projects/test-app/.venv/lib/python3.10/site-packages (from build) (2.0.2)
(.venv) user@user:/mnt/d/schoolwork/projects/FlowUtils$ python -m build
* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
  - oldest-supported-numpy
  - setuptools>=61.0
* Getting build dependencies for sdist...
running egg_info
writing src/FlowUtils.egg-info/PKG-INFO
writing dependency_links to src/FlowUtils.egg-info/dependency_links.txt
writing requirements to src/FlowUtils.egg-info/requires.txt
writing top-level names to src/FlowUtils.egg-info/top_level.txt
reading manifest file 'src/FlowUtils.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file 'src/FlowUtils.egg-info/SOURCES.txt'
* Building sdist...
running sdist
running egg_info
writing src/FlowUtils.egg-info/PKG-INFO
writing dependency_links to src/FlowUtils.egg-info/dependency_links.txt
writing requirements to src/FlowUtils.egg-info/requires.txt
writing top-level names to src/FlowUtils.egg-info/top_level.txt
reading manifest file 'src/FlowUtils.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file 'src/FlowUtils.egg-info/SOURCES.txt'
running check
creating flowutils-1.1.0
creating flowutils-1.1.0/src/FlowUtils.egg-info
creating flowutils-1.1.0/src/flowutils
creating flowutils-1.1.0/src/flowutils/gating_c_ext
creating flowutils-1.1.0/src/flowutils/logicle_c_ext
copying files to flowutils-1.1.0...
copying LICENSE -> flowutils-1.1.0
copying MANIFEST.in -> flowutils-1.1.0
copying README.md -> flowutils-1.1.0
copying pyproject.toml -> flowutils-1.1.0
copying setup.py -> flowutils-1.1.0
copying src/FlowUtils.egg-info/PKG-INFO -> flowutils-1.1.0/src/FlowUtils.egg-info
copying src/FlowUtils.egg-info/SOURCES.txt -> flowutils-1.1.0/src/FlowUtils.egg-info
copying src/FlowUtils.egg-info/dependency_links.txt -> flowutils-1.1.0/src/FlowUtils.egg-info
copying src/FlowUtils.egg-info/requires.txt -> flowutils-1.1.0/src/FlowUtils.egg-info
copying src/FlowUtils.egg-info/top_level.txt -> flowutils-1.1.0/src/FlowUtils.egg-info
copying src/flowutils/__init__.py -> flowutils-1.1.0/src/flowutils
copying src/flowutils/_version.py -> flowutils-1.1.0/src/flowutils
copying src/flowutils/compensate.py -> flowutils-1.1.0/src/flowutils
copying src/flowutils/gating.py -> flowutils-1.1.0/src/flowutils
copying src/flowutils/transforms.py -> flowutils-1.1.0/src/flowutils
copying src/flowutils/gating_c_ext/_gate_helpers.c -> flowutils-1.1.0/src/flowutils/gating_c_ext
copying src/flowutils/gating_c_ext/gate_helpers.c -> flowutils-1.1.0/src/flowutils/gating_c_ext
copying src/flowutils/gating_c_ext/gate_helpers.h -> flowutils-1.1.0/src/flowutils/gating_c_ext
copying src/flowutils/logicle_c_ext/_logicle.c -> flowutils-1.1.0/src/flowutils/logicle_c_ext
copying src/flowutils/logicle_c_ext/logicle.c -> flowutils-1.1.0/src/flowutils/logicle_c_ext
copying src/flowutils/logicle_c_ext/logicle.h -> flowutils-1.1.0/src/flowutils/logicle_c_ext
copying src/FlowUtils.egg-info/SOURCES.txt -> flowutils-1.1.0/src/FlowUtils.egg-info
Writing flowutils-1.1.0/setup.cfg
Creating tar archive
removing 'flowutils-1.1.0' (and everything under it)
* Building wheel from sdist
* Creating isolated environment: venv+pip...
* Installing packages in isolated environment:
  - oldest-supported-numpy
  - setuptools>=61.0
* Getting build dependencies for wheel...
running egg_info
writing src/FlowUtils.egg-info/PKG-INFO
writing dependency_links to src/FlowUtils.egg-info/dependency_links.txt
writing requirements to src/FlowUtils.egg-info/requires.txt
writing top-level names to src/FlowUtils.egg-info/top_level.txt
reading manifest file 'src/FlowUtils.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file 'src/FlowUtils.egg-info/SOURCES.txt'
* Building wheel...
running bdist_wheel
running build
running build_py
creating build/lib.linux-x86_64-cpython-310/flowutils
copying src/flowutils/_version.py -> build/lib.linux-x86_64-cpython-310/flowutils
copying src/flowutils/compensate.py -> build/lib.linux-x86_64-cpython-310/flowutils
copying src/flowutils/gating.py -> build/lib.linux-x86_64-cpython-310/flowutils
copying src/flowutils/__init__.py -> build/lib.linux-x86_64-cpython-310/flowutils
copying src/flowutils/transforms.py -> build/lib.linux-x86_64-cpython-310/flowutils
running build_ext
building 'flowutils.logicle_c' extension
creating build/temp.linux-x86_64-cpython-310/src/flowutils/logicle_c_ext
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/tmp/build-env-alaox7iy/lib/python3.10/site-packages/numpy/core/include -Isrc/flowutils/logicle_c_ext -I/tmp/build-env-alaox7iy/include -I/usr/include/python3.10 -c src/flowutils/logicle_c_ext/_logicle.c -o build/temp.linux-x86_64-cpython-310/src/flowutils/logicle_c_ext/_logicle.o -std=c99
In file included from /tmp/build-env-alaox7iy/lib/python3.10/site-packages/numpy/core/include/numpy/ndarraytypes.h:1969,
                 from /tmp/build-env-alaox7iy/lib/python3.10/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /tmp/build-env-alaox7iy/lib/python3.10/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from src/flowutils/logicle_c_ext/_logicle.c:2:
/tmp/build-env-alaox7iy/lib/python3.10/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
   17 | #warning "Using deprecated NumPy API, disable it with " \
      |  ^~~~~~~
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/tmp/build-env-alaox7iy/lib/python3.10/site-packages/numpy/core/include -Isrc/flowutils/logicle_c_ext -I/tmp/build-env-alaox7iy/include -I/usr/include/python3.10 -c src/flowutils/logicle_c_ext/logicle.c -o build/temp.linux-x86_64-cpython-310/src/flowutils/logicle_c_ext/logicle.o -std=c99
x86_64-linux-gnu-gcc -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -g -fwrapv -O2 build/temp.linux-x86_64-cpython-310/src/flowutils/logicle_c_ext/_logicle.o build/temp.linux-x86_64-cpython-310/src/flowutils/logicle_c_ext/logicle.o -L/usr/lib/x86_64-linux-gnu -o build/lib.linux-x86_64-cpython-310/flowutils/logicle_c.cpython-310-x86_64-linux-gnu.so
building 'flowutils.gating_c' extension
creating build/temp.linux-x86_64-cpython-310/src/flowutils/gating_c_ext
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/tmp/build-env-alaox7iy/lib/python3.10/site-packages/numpy/core/include -Isrc/flowutils/gating_c_ext -I/tmp/build-env-alaox7iy/include -I/usr/include/python3.10 -c src/flowutils/gating_c_ext/_gate_helpers.c -o build/temp.linux-x86_64-cpython-310/src/flowutils/gating_c_ext/_gate_helpers.o -std=c99
In file included from /tmp/build-env-alaox7iy/lib/python3.10/site-packages/numpy/core/include/numpy/ndarraytypes.h:1969,
                 from /tmp/build-env-alaox7iy/lib/python3.10/site-packages/numpy/core/include/numpy/ndarrayobject.h:12,
                 from /tmp/build-env-alaox7iy/lib/python3.10/site-packages/numpy/core/include/numpy/arrayobject.h:4,
                 from src/flowutils/gating_c_ext/_gate_helpers.c:2:
/tmp/build-env-alaox7iy/lib/python3.10/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:17:2: warning: #warning "Using deprecated NumPy API, disable it with " "#define NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
   17 | #warning "Using deprecated NumPy API, disable it with " \
      |  ^~~~~~~
x86_64-linux-gnu-gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -fPIC -I/tmp/build-env-alaox7iy/lib/python3.10/site-packages/numpy/core/include -Isrc/flowutils/gating_c_ext -I/tmp/build-env-alaox7iy/include -I/usr/include/python3.10 -c src/flowutils/gating_c_ext/gate_helpers.c -o build/temp.linux-x86_64-cpython-310/src/flowutils/gating_c_ext/gate_helpers.o -std=c99
x86_64-linux-gnu-gcc -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-Bsymbolic-functions -g -fwrapv -O2 build/temp.linux-x86_64-cpython-310/src/flowutils/gating_c_ext/_gate_helpers.o build/temp.linux-x86_64-cpython-310/src/flowutils/gating_c_ext/gate_helpers.o -L/usr/lib/x86_64-linux-gnu -o build/lib.linux-x86_64-cpython-310/flowutils/gating_c.cpython-310-x86_64-linux-gnu.so
installing to build/bdist.linux-x86_64/wheel
running install
running install_lib
creating build/bdist.linux-x86_64/wheel
creating build/bdist.linux-x86_64/wheel/flowutils
copying build/lib.linux-x86_64-cpython-310/flowutils/_version.py -> build/bdist.linux-x86_64/wheel/./flowutils
copying build/lib.linux-x86_64-cpython-310/flowutils/compensate.py -> build/bdist.linux-x86_64/wheel/./flowutils
copying build/lib.linux-x86_64-cpython-310/flowutils/gating.py -> build/bdist.linux-x86_64/wheel/./flowutils
copying build/lib.linux-x86_64-cpython-310/flowutils/__init__.py -> build/bdist.linux-x86_64/wheel/./flowutils
copying build/lib.linux-x86_64-cpython-310/flowutils/transforms.py -> build/bdist.linux-x86_64/wheel/./flowutils
copying build/lib.linux-x86_64-cpython-310/flowutils/gating_c.cpython-310-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/wheel/./flowutils
copying build/lib.linux-x86_64-cpython-310/flowutils/logicle_c.cpython-310-x86_64-linux-gnu.so -> build/bdist.linux-x86_64/wheel/./flowutils
running install_egg_info
running egg_info
writing src/FlowUtils.egg-info/PKG-INFO
writing dependency_links to src/FlowUtils.egg-info/dependency_links.txt
writing requirements to src/FlowUtils.egg-info/requires.txt
writing top-level names to src/FlowUtils.egg-info/top_level.txt
reading manifest file 'src/FlowUtils.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
adding license file 'LICENSE'
writing manifest file 'src/FlowUtils.egg-info/SOURCES.txt'
Copying src/FlowUtils.egg-info to build/bdist.linux-x86_64/wheel/./FlowUtils-1.1.0-py3.10.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/FlowUtils-1.1.0.dist-info/WHEEL
creating '/mnt/d/schoolwork/projects/FlowUtils/dist/.tmp-6q9ebrg6/FlowUtils-1.1.0-cp310-cp310-linux_x86_64.whl' and adding 'build/bdist.linux-x86_64/wheel' to it
adding 'flowutils/__init__.py'
adding 'flowutils/_version.py'
adding 'flowutils/compensate.py'
adding 'flowutils/gating.py'
adding 'flowutils/gating_c.cpython-310-x86_64-linux-gnu.so'
adding 'flowutils/logicle_c.cpython-310-x86_64-linux-gnu.so'
adding 'flowutils/transforms.py'
adding 'FlowUtils-1.1.0.dist-info/LICENSE'
adding 'FlowUtils-1.1.0.dist-info/METADATA'
adding 'FlowUtils-1.1.0.dist-info/WHEEL'
adding 'FlowUtils-1.1.0.dist-info/top_level.txt'
adding 'FlowUtils-1.1.0.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
Successfully built flowutils-1.1.0.tar.gz and FlowUtils-1.1.0-cp310-cp310-linux_x86_64.whl

[hoodmane]

Thanks for the report @aya49! @ryanking13 is this potentially related to #21?

[hoodmane]

From the paths it looks like you are using python3.10. Can you try again with Python 3.12? If that's the problem, we should add a version check with a sensible error message.

[ryanking13]

I am guessing that it is more related to the Python version mismatch.

[hoodmane]

We definitely should add a test that we return a good error when there is a version mismatch.

[ryanking13]

We do check the version when installing the xbuildenv, but looking at the build log, for some reason it is already installed so there was no additional verification.

[hoodmane]

Probably pyodide build should always do a version check. It's much cheaper than installing the xbuildenv and we do that all the time even if it isn't needed.

[ryanking13]

Yeah, I think we should

1. Store the metadata including local Python and (optionally?) pyodide-build version when installing xbuildenv
2. When running pyodide build check this metadata and raise if the local Python or pyodide-build version has changed.

[aya49]

Thank you for the fast reply @hoodmane! I will try it out and post the results! Edit: It worked!!