More permanent solution for PEP 517/660's config_settings

[abravalheri]

Request for comments / options / suggestions.

Cc @pypa/setuptools-developers, see motivation in #3896, #2491 and other related issues.
The detailed "state of the implementation" is described in #3896 (comment).

What to put in config_settings?

As discussed in #3896 (comment), I believe a more definitive solution for the config_settings dilema in setuptools, is to accept something like the following:

config_settings = {
    # "{command}.{option}": value,
    "build_ext.parallel": 8,
    "bdist_wheel.py_limited_api": "cp311",
    ...
}

The following is a list of relevant options (in my opinion) by command.
I think that for now we can focus on only allowing these options to be customised, or maybe even a smaller subset.

egg_info/dist_info	build	build_ext	bdist_wheel
tag_build1	debug	debug2	compression3
tag_date	force	define	py_limited_api4
no_date	parallel	undef	python_tag5
	compiler	include_dirs	plat_name6
	plat_name7	library_dirs
		libraries
		rpath
		link_objects
		swig_cpp
		swig_opts
		plat_name8

We can keep --global-option and --build-option for now, working as proposed in #4079 (somehow compatible with pip's legacy options).

What is the challenge in passing config_settings from setuptools.build_meta to setuptools.dist.Distribution?

In setuptools/distutils one command can not only run another command directly, but also access other command's option and even modify them.
This makes it difficult to convert config_settings into sys.argv, because (due to the design discussed above) options work in an implicit way: even if you are calling bdist_wheel, the build_ext or egg_info options will influence the build outcome.

To make it worse, the following also happens:

1. PEP 517 requires each build hook to be called in a brand new subprocess⁹
2. PEP 517 requires build hooks to "idempotent" with relation to the *.dist-info⁹
3. Because we need to be backward compatible with setup.py there is an "isolation barrier" between setuptools.build_meta and the other parts of setuptools: currently all the communication is one way (build_meta => rest) and happens via sys.argv.¹⁰

How to solve this challenge?

I think that we can use something something similar to the suggestion in #3896 (comment) (i.e. use repr(...) to serialise config_settings so it can go to sys.argv, and then use ast.parse_literal(...) on the other side to retrive it).
However I would like to expand on it and also propose a "private build hook" that would allow build_meta to acess the dist object directly (and therefore counterbalance the "isolation barier" mentioned above). All togheter it could look like the following:

# somewhere in `build_meta`
sys.argv = [
    "setup.py",
    "--xxx-private-build-hook", f"{self._reference}._finalize_build_options",
    # ^--- _reference="setuptools.build_meta:_BACKEND" or "setuptools.build_meta:__legacy__"
    "--xxx-private-build-args", repr(config_settings),
    cmd
]

Then we could modify setuptools.dist.Distribution._parse_command_opts() to call xxx_private_build_hook(dist_obj, ast.literal_eval(xxx_private_build_args)).

This way build_meta can process config_settings accordingly (and we don't have to couple setuptools.dist with the UI choices made in build_meta).

Probably setuptools.build_meta:_BACKEND._finalize_build_options can work like the following:

1. Consume some of the config_settings that are not related to command objects
   (e.g. level = config_settings.get("log_level", "warning"); logging.setLevel(gatattr(logging, level.upper())).)
2. Preprocess the the other config_settings into something that can be directly assigned to the command objects.
   • This may involve translating names (e.g. dist_info.version_tag => dist_info.tag_build) and or values.
   • It may also involve translating 1 config into multiple ones (e.g. bdist_wheel.plat_name => [bdist_wheel.plat_name, build.plat_name)

Other Open Challenges

Naming and semantics

Names are hard ... and separating some options under build, others under build_ext and others under bdist_wheel seems a bit arbitraty.

Moreover, bdist_wheel allows you to specify plat_name freely, but build only allow plat_name to be specified on Windows machines (e.g. win32 or win-amd64).

My interpretation is that the build command does not want to give "false hopes" about cross compiling for Linux users, and only allow windows users to specify plat_name because it cannot automatically choose for them.
On the other hand, I imagine that bdist_wheel allows values on linux so other plugins (e.g. setuptools-rust) can use it when cross-compiling (vanilla setuptools don't have cross-compilation capabilities, but plugins do).
So we already have a disagreement between setuptools and bdist_wheel.

Ideally I would like to have only one option for plat_name (e.g. build.plat_name - does it mean that we need to flexibilize build.plat_name?) - so we avoid the confusing situation where the user can pass 2 different values for build.plat_name and bdist_wheel.plat_name...

But then we are back to the problem of arbitrarily separating parameters under build/build_ext/bdist_wheel/...

How does this change impacts the other open issues in setuptools?

To solve #3237 and #3119,
we can consider setting egg_info.egg_base and dist_info.output_dir.
This way we can use PEP 517' metadata_directory and/or temporary directories and avoid read only project trees.

However, if we do that we stumble into a different problem: sometimes the build process is designed to have
the directory containing *.egg-info/*.dist-info to be available in sys.path.
This is required so the entry points are available to importlib.metadata
(see #3921 (comment) and the following comment).

If we want to go ahead and use "off-tree" *.egg-info, I believe we have 2 choices:

1. Always add the directory containing *.egg-info/*.dist-info to sys.path
2. Implement a new MetaPathFinder with find_distributions and add it to sys.meta_path.

Footnotes

1. Maybe rename to version_tag or local_version?
   Since .whl files can have a build tag, which is a different thing... ↩

2. Fallback to build.debug (so we can probably ommit build_ext.debug) ↩

3. E.g.: "stored", "deflated". ↩

4. E.g.: cp38. ↩

5. E.g.: py3, py2, py2.py3, py27, ... ↩

6. Fallback to bdist.plat_name, which in turn fallback to build.plat_name,
   however it behaves differently from build.plat_name, because it does accept arbitrary
   values for any operating system. See footnote⁷ ↩

7. Only valid on Windows, though... There is a validation in distutils.
   Maybe we can add an externally_crosscompiled=True/False option to bypass this validation? ↩ ↩²

8. Fallback to build.plat_name ↩

9. So if you pass options to egg_info/dist_info in prepare_metadata_for_build_wheel, then you need to pass the same options again in build_wheel. ↩ ↩²

10. As a result we have to implement all sorts of hacks when we need to somehow access the "rest" of setuptools, e.g.: the hack for extracting dist.setup_requires' value. ↩

[h-vetinari]

I'm not sure this is actually very visible as a "discussion". Might be better as an RFC issue?

[jameshilliard]

We can keep --global-option and --build-option for now, working as proposed in #4079 (somehow compatible with pip's legacy options).

We were depending on --build-option to pass build_ext options to all build commands, so we need to revert the breakage caused by #4079 in #4218 for now I think.