Additive overrides

[henryiii]

I've been meaning to tackle additive overrides for a while, and I'd like to have a consistent solution between scikit-build-core and cibuildwheel if possible (see scikit-build/scikit-build-core#564 & our recent meeting https://github.com/orgs/scikit-build/discussions/1053). Currently, overrides always completely replace the table or list they are overriding, which is painful if you just need to add an item. What about this proposal:

Appending to a list or table (overwrites existing keys):

some.thing = [1, 2]
[[overrides]]
some."thing+" = [3]
# thing is now 1, 2, 3

(this could even be extended with ++, which would merge a table of tables)

Prepending to a list or table (doesn't affect existing keys):

thing = [1, 2]
[[overrides]]
some."+thing" = [3]
# thing is now 1, 2, 3

Removing from an existing table (not necessary to support for lists, at least for now, IMO):

thing = {one=1, two=2}
[[overrides]]
some."thing-" = ["two"]
# thing is now one=1

I've had this idea for quite some time, but was annoyed by the TOML requirement of adding quotes, but after working with TOML a lot, it requires quotes so often that this doesn't seem that bad anymore. You have to use them just to define a simple entry-point in PEP 621, for example.

The goal here is to agree on the design; I don't think there's a need to implement anything other than "thing+" in cibuildwheel for now. Scikit-build-core might need a bit more added.

Thoughts?

[joerick]

👍 I like the concept, it feels punitive when users have lots of variables defined to require listing them again just to add/remove one.

How do you append to a table in your proposal? That feels like the most common use-case for this, on the environment option. I'm assuming it looks something like

thing = {one=1, two=2}
[[overrides.some-build]]
"thing+" = {three=3}
# thing is now {one=1, two=2, three=3}

I'm not wild about the quotes in the TOML syntax, here. I have an alternative proposal of a $inherit key/value that indicates that the previously defined value is inherited.

So, you'd have

thing = ["1", "2"]
[[overrides.some-build]]
thing = ["$inherit", "3"]
# thing is now "1", "2", "3"

or, for a table,

environment = {A="1",B="2"} 
[[overrides.some-build]]
environment = {"$inherit"=true, C="3"}
# environment is now {A="1", B="2", C="3"}

Removal could just be setting to an empty string, or we could say that false or "$unset" did that.

The current parsing of options takes the rich TOML values and collapses them to strings. The reason is mostly historical (I think), but it does ensure that we don't add anything in TOML that is impossible in env vars. That's interesting, because perhaps this approach also has the possible benefit of being able to be used in a environment variable, e.g. CIBW_BEFORE_BUILD="$inherit && some_extra_command.sh", which would pick up the TOML-defined option and add to it, or CIBW_REPAIR_WHEEL_COMMAND="some-lint-command {wheel} && $inherit" to extend the default.

[henryiii]

This gives me an idea that's a bit more specific to overrides, but would I think work without adding a special string ($inherit would actually expand to nothing on the command line, you'd have to escape it on many shells - the example above would not work, for example).

thing = ["1", "2"]
[[overrides.some-build]]
inherit = ["thing"]
thing = ["3"]

environment = {A="1",B="2"} 
[[overrides.some-build]]
inherit = ["environment"]
environment = {C="3"}

You'd just list the keys that will be merged instead of replaced. Maybe append or similar would be a better name, especially if there was a way to remove things later.

We can't use empty strings to remove, since some things (like the https://no-color.org spec) do not treat empty strings and undefined variables equivalently.

[henryiii]

In my original proposal, it could also be:

thing = {one=1, two=2}
[[overrides.some-build]]
"thing+".three=3
# thing is now {one=1, two=2, three=3}

Quotes in TOML are pretty hard to avoid, [project.entry-points."validate_pyproject.tool_schema"] for example.