API mode / C sources does not build with default setuptools / build

[dgdguk]

When compiling a CFFI module that compiles source code (these instructions), if that source code contains headers, and those headers are used in the C code, it fails with the default invocation of python -m build. This is because CFFI is not able to tell setuptools about the header files, and by default python -m build generates an isolated enviroment containing only the files that setuptools knows about, which will not include the required headers.

A workaround is to pass something like package_data={"": ["*.h"]}, to setuptools.setup, which will grab all .h files, but it may be worth either a) documenting this behaviour or b) augmenting set_source with a headers parameter so it can mark those headers as being required.

[arigo]

I'm the author of CFFI but I'm not following the current evolution of setuptools. I don't know in which situations this is a problem and the proper way to fix it. I would be happy to accept a pull request if you or someone else feels like implementing it.

[dgdguk]

Well, I think first thing to do would be to add a comment to the documentation so that people can work around it. If you want to see the issue in action, then here's some steps for that:

1. Pull https://github.com/dgdguk/convolutionalfixedsum
2. In setup.py, comment out line 20: package_data={"": ["*.h"]},
3. Run pip install build, followed by python -m build; this will crash, with the C compiler complaining it cannot find the headers.

This is by no means a critical issue at the moment; it's specific to the build tooling, so building a package with pip should still work. However, the PyPA is already recommending using build over pip, and I suspect that it's going to start becoming the default in a lot of places.

As for patches - I've not currently got the time for it, but I might be able to contribute something over the summer, if no-one gets to it before me.

[arigo]

That code calls set_source(...include_dirs=[str(analytical_base)]), though. This argument is passed straight from CFFI to setuptools, like most keyword arguments to set_source(). Does it mean that setuptools still accepts, but now ignores, the include_dirs argument in this case? If setuptools still accepts the include_dirs argument but no longer passes it to the C compiler as an -I option, that's rather strange (but not unexpected, because changing things in incompatible ways every two years seems to be common for setuptools).

[dgdguk]

I believe the problem is the isolated build environment that build creates. Basically, build copies all of the files that setuptools has been told about into a separate directory, with a seperate venv and everything, to make sure that the build doesn't rely on random files that setuptools doesn't know about and that someone with the source code can reproduce the build.

Unfortunately, while setuptools knows about all of the .c files, it doesn't know about the .h files. So it dutifully copies over all the .c files without the .h files, and then include_dirs ends up pointing to a folder which does not contain the .h files.

[dgdguk]

I've (finally) done a bit more digging on this and can confirm a few things about how the new build tooling workflow works:

1. Python creates a "clean" source distribution. Basically the same as python setup.py sdist.
2. 'setuptools's logic for "what is a source file" for extensions is only the .c source files specified as sources. It looks like at some point someone wanted to include header files, but wasn't able to do so.
3. The project is then built against the clean distribution, and fails because the .h files have not been included.

This also means that source distributions built using python setup.py sdist or similar for CFFI based modules are broken if they require .h files, as these will not be included.

There's a few ways this could be fixed. However, I'm not sure what is in best keeping with how CFFI works.

1. An extra header_files argument is added which allows the user to specify header files to be added to the list of sources.
2. When adding a .c file, add all .h files in the same directory (or perhaps subdirectory) to sources.
3. Blanket add all .h files in the project.
4. Use pycparser to pull include filenames out of the .c source files.

(1) is probably an acceptable way of doing things, because (2)/(3) - while it would "fix" existing projects whose source distributions are broken - don't seem in keeping with the ethos of how setuptools works. (4) would be optimal, but I'm not sure if it's really possible due to C macros and the like (I can think of a way to do it, but it seems like a massive hack, not to mention slow). Or alternatively, maybe some combination of the above solutions would be best?

In any case, if there's some guidance available on what would be a CFFI-style solution, I could probably code one up and send a PR.

[arigo]

The CFFI-style solution (if my opinion is still useful, as historically the main developer, but no longer): if a developer of a non-CFFI-based Python extension module has got some .h files in addition to .c files, how is he supposed to tell setuptools about it? I think we should investigate what the answer to that question is, and if it's "add some obscure parameters to setuptools" then figure out if we can already give the same parameters to set_source(). If so, we can certainly document that but I wouldn't add extra magic to CFFI itself. (Note that set_source() accepts keyword arguments and pass them unmodified to setuptools.)

[dgdguk]

That could work.

It looks like distutils.Extension has a depends argument for files the extension depends on but are not source files that can be passed into a compiler. So doing set_source( ..., depends=list_of_header_files) works.

However, I would personally advocate for adding headers as a parameter to set_source() and adding a little magic to drop anything in headers into the depends variable. That seems like it would be substantially more readable. But I suppose it's up to the maintainers for their preference.

[arigo]

If depends is the keyword chosen by setuptools for the headers then I think the issue of "it's an obscure keyword" should be brought up to setuptools, not CFFI. They probably have reasons, like "it can also be used for files that are not headers", even though I agree that an explicit headers keyword would be clearer. It's by far not the first time that a setuptools issue ends up being discussed in CFFI and I tend to reply that it's a setuptools issue. My opinion---again, as an outsider at this point---is that we should document it but not change it in CFFI.