Where did my Cython extensions go?

[dionhaefner]

Since setuptools 0.64.0, doing pip install -e . does not put compiled Cython extensions into the source tree anymore. Trying to import them fails. I can see that they are being build though:

Building wheels for collected packages: veros
  Created temporary directory: /private/var/folders/ld/ylk19q555951g5k6t0vhz6740000gp/T/pip-wheel-ic3ndhp5
  Destination directory: /private/var/folders/ld/ylk19q555951g5k6t0vhz6740000gp/T/pip-wheel-ic3ndhp5
  Running command Building editable for veros (pyproject.toml)
  /private/var/folders/ld/ylk19q555951g5k6t0vhz6740000gp/T/pip-build-env-w64klq58/overlay/lib/python3.9/site-packages/setuptools/dist.py:771: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
    warnings.warn(
  running editable_wheel
  creating /private/var/folders/ld/ylk19q555951g5k6t0vhz6740000gp/T/pip-wheel-ic3ndhp5/tmpsd9xqaz8/veros.egg-info
  writing /private/var/folders/ld/ylk19q555951g5k6t0vhz6740000gp/T/pip-wheel-ic3ndhp5/tmpsd9xqaz8/veros.egg-info/PKG-INFO
  writing dependency_links to /private/var/folders/ld/ylk19q555951g5k6t0vhz6740000gp/T/pip-wheel-ic3ndhp5/tmpsd9xqaz8/veros.egg-info/dependency_links.txt
  writing entry points to /private/var/folders/ld/ylk19q555951g5k6t0vhz6740000gp/T/pip-wheel-ic3ndhp5/tmpsd9xqaz8/veros.egg-info/entry_points.txt
  writing requirements to /private/var/folders/ld/ylk19q555951g5k6t0vhz6740000gp/T/pip-wheel-ic3ndhp5/tmpsd9xqaz8/veros.egg-info/requires.txt
  writing top-level names to /private/var/folders/ld/ylk19q555951g5k6t0vhz6740000gp/T/pip-wheel-ic3ndhp5/tmpsd9xqaz8/veros.egg-info/top_level.txt
  writing manifest file '/private/var/folders/ld/ylk19q555951g5k6t0vhz6740000gp/T/pip-wheel-ic3ndhp5/tmpsd9xqaz8/veros.egg-info/SOURCES.txt'
  reading manifest file '/private/var/folders/ld/ylk19q555951g5k6t0vhz6740000gp/T/pip-wheel-ic3ndhp5/tmpsd9xqaz8/veros.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  warning: no files found matching '*.pxd' anywhere in distribution
  adding license file 'LICENSE'
  writing manifest file '/private/var/folders/ld/ylk19q555951g5k6t0vhz6740000gp/T/pip-wheel-ic3ndhp5/tmpsd9xqaz8/veros.egg-info/SOURCES.txt'
  creating '/private/var/folders/ld/ylk19q555951g5k6t0vhz6740000gp/T/pip-wheel-ic3ndhp5/tmpsd9xqaz8/veros-1.4.4+3.ge171196.dist-info'
  adding license file "LICENSE" (matched pattern "LICEN[CS]E*")
  creating /private/var/folders/ld/ylk19q555951g5k6t0vhz6740000gp/T/pip-wheel-ic3ndhp5/tmpsd9xqaz8/veros-1.4.4+3.ge171196.dist-info/WHEEL
  /private/var/folders/ld/ylk19q555951g5k6t0vhz6740000gp/T/pip-build-env-w64klq58/overlay/lib/python3.9/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
    warnings.warn(
  running build
  running build_py
  running build_ext
  building 'veros.core.special.tdma_cython_' extension
  creating /var/folders/ld/ylk19q555951g5k6t0vhz6740000gp/T/tmpnl7kf864.build-temp/veros
  creating /var/folders/ld/ylk19q555951g5k6t0vhz6740000gp/T/tmpnl7kf864.build-temp/veros/core
  creating /var/folders/ld/ylk19q555951g5k6t0vhz6740000gp/T/tmpnl7kf864.build-temp/veros/core/special
  clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk -I/Users/dion/.virtualenvs/tempenv-1abe253492c84/include -I/usr/local/Cellar/[email protected]/3.9.12/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c veros/core/special/tdma_cython_.c -o /var/folders/ld/ylk19q555951g5k6t0vhz6740000gp/T/tmpnl7kf864.build-temp/veros/core/special/tdma_cython_.o
  creating /var/folders/ld/ylk19q555951g5k6t0vhz6740000gp/T/tmp3jv_fuuz.build-lib/veros
  creating /var/folders/ld/ylk19q555951g5k6t0vhz6740000gp/T/tmp3jv_fuuz.build-lib/veros/core
  creating /var/folders/ld/ylk19q555951g5k6t0vhz6740000gp/T/tmp3jv_fuuz.build-lib/veros/core/special
  clang -bundle -undefined dynamic_lookup -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX12.sdk /var/folders/ld/ylk19q555951g5k6t0vhz6740000gp/T/tmpnl7kf864.build-temp/veros/core/special/tdma_cython_.o -o /var/folders/ld/ylk19q555951g5k6t0vhz6740000gp/T/tmp3jv_fuuz.build-lib/veros/core/special/tdma_cython_.cpython-39-darwin.so
  Editable install will be performed using a meta path finder.

  Options like `package-data`, `include/exclude-package-data` or
  `packages.find.exclude/include` may have no effect.

  adding '__editable___veros_1_4_4_3_ge171196_dirty_finder.py'
  adding '__editable__.veros-1.4.4+3.ge171196.dirty.pth'
  creating '/private/var/folders/ld/ylk19q555951g5k6t0vhz6740000gp/T/pip-wheel-ucsuc43n/tmpkpgl_smq/veros-1.4.4+3.ge171196.dirty-0.editable-cp39-cp39-macosx_12_x86_64.whl' and adding '/var/folders/ld/ylk19q555951g5k6t0vhz6740000gp/T/tmp023s6ta4veros-1.4.4+3.ge171196.dirty-0.editable-cp39-cp39-macosx_12_x86_64.whl' to it
  adding 'veros-1.4.4+3.ge171196.dirty.dist-info/LICENSE'
  adding 'veros-1.4.4+3.ge171196.dirty.dist-info/METADATA'
  adding 'veros-1.4.4+3.ge171196.dirty.dist-info/WHEEL'
  adding 'veros-1.4.4+3.ge171196.dirty.dist-info/entry_points.txt'
  adding 'veros-1.4.4+3.ge171196.dirty.dist-info/top_level.txt'
  adding 'veros-1.4.4+3.ge171196.dirty.dist-info/RECORD'
  Building editable for veros (pyproject.toml) ... done
  Created wheel for veros: filename=veros-1.4.4+3.ge171196.dirty-0.editable-cp39-cp39-macosx_12_x86_64.whl size=7132 sha256=e6d18ed8d81664a582df6e73e99cbf23c82e995ba8436fb84a6909fa1253de71
  Stored in directory: /private/var/folders/ld/ylk19q555951g5k6t0vhz6740000gp/T/pip-ephem-wheel-cache-lodhfqj6/wheels/fe/98/52/82b2337cd6fe9cbc55497579e3f445ae8c199ba669afa7f192
Successfully built veros

It looks like the extensions are being built but just thrown away after that?

My setup is here: https://github.com/team-ocean/veros

[dionhaefner]

Aha, turns out this is caused by a combination of things:

1. A custom build_ext command class that derives from Cython.Distutils.build_ext.build_ext instead of setuptools.command.build_ext.build_ext.
2. An optional extension (with ext.optional = True) failing to build.

Problem 1 is an easy fix (just replace the base class), but I don't understand 2...

Edit: problem 2 was also present in older setuptools. Raised as #3510.