Click future direction

[Rowlando13]

In the last few years of working on Click, despite @kdeldycke adding many hundreds of tests, there has been a lot of breakage, specifically with 8.2 and 8.3. And 8.2 and 8.3 were mostly focused around fixing long standing bugs. Growing the test suite, has helped, but at this point it feels like the API is just too open, meaning there are many methods which are public. It feels like it is time for 9.0, where we actually fix the problem. In my mind 9.0 would try to:

1. Rewrite the parser to remove the optparse baggage
2. Make a lot of currently public methods private
3. Allow for fully customizable help page (which I would do)

However, I am sensitive to how much breakage this is likely to cause. 1. and 2. are mostly for the maintainers which I does not sell the breakage at all for people upgrading. 3. Is a strong reason to upgrade, but I am not convinced it will be enough right away. I think the solution is to land 9.0 with 1,2, and 3. Plan for lots of bug fix patches(9.0.1, 9.0.2...) And then try to land some much desire features in 9.1, maybe option groups, option ordering, ? However, those features should be planned in concert with 1.

I would also like to provide a clear upgrade path to users by:

• Provide a stable release for people to use since this will take a while. Maybe that is 8.3.1?
• If we make a method private, we would document what its replacement should be.
• ?

@untitaker @davidism @kdeldycke Does this sound right, or am I off base. I am very open to feedback since I have no experience with this sort of thing.

[kdeldycke]

• Make a lot of currently public methods private

If it would be on me I would be even more bolder about that by having them explicitly prefixed by an underscore _. 😅

And then try to land some much desire features in 9.1, maybe option groups, option ordering, ?

We can even plan both a 9.x and a 10.x in a short time frame (like less than a year), to have a major release dedicated to parser rewrite, and one to data/interface contracts.

Integers are cheap and it is a strong signal for maintainers to look twice at the changelog before blindly upgrading Click. See for instance how Graphviz was stagnant for a decade then went from 2.42.4 to 14.0.4 in the space of 5 years.

Other random ideas to review/re-evaluate for a major release:

• Unhide UNSET in internals
• Make UNSET falsy (cc @edgarrmondragon)
• Split-up core.py into context.py / parameter.py / command.py
• Split-up class Option into Option, FlagOption and BoolFlagOption
• Mirrors Option.is_flag/Option.flag_value concept into Parameter.is_default/Parameter.default_value
• Remove the implicit legacy cases of default and flag_value: remove some auto-magic around defaults and make behavior more strict
• Rename all private methods/functions/variables with _underscore style
• Remove utils.py / _utils.py: code needs to have a clear purpose and domain, else everything ends up being a vague utility or helper
• Remove isolated_filesystem()
• Change the data contract: move all type conversion embedded into the parser and core to Click's types (i.e. types.py::ParamType); preserve native types (Python/strings) from the edge of the framework to the console rendering. See @sirosen comment on data separation.
• consolidate signatures and improving typing

[sirosen]

We can even plan both a 9.x and a 10.x in a short time frame (like less than a year), to have a major release dedicated to parser rewrite, and one to data/interface contracts.

As a heavy user of, occasional contributor to, click I strongly endorse this path. There's nothing "wrong" with the version numbers going 9.0.0, 9.0.1, 10.0.0, if that's the best way to communicate with users. And for larger projects, porting their usage for each of these changes could be an endeavor in its own right.

I have seen projects get "stuck" on old dependency versions because a single-step upgrade changed too much at once. If I have to migrate a project across three or four or even more significant click releases in a row (e.g., 9.0, 10.0, 10.1, 11.0), that's generally going to be a lot easier on me as a downstream maintainer than trying to handle many changes all at once.

If click 9.0 simply made a lot of names private by renaming them with _, with few other intentional¹ user-facing changes, that would be extremely valuable. 10.0 could be the version for deeper structural changes.

I have written lots of custom param types, command and group types, and even custom option and argument subclasses, and undoubtedly have used methods meant to be private unknowingly. I remember that @davidism was surprised to hear I was overriding ParamType.get_metavar (see #2929 for context), but click had no prior way to indicate that this would be surprising.
In handling the removal of methods and attributes, projects will migrate to the APIs which are intended to be public (e.g., to_info_dict), and will therefore have a better experience with subsequent upgrades because the docs will direct them towards what changed.

Provide a stable release for people to use since this will take a while. Maybe that is 8.3.1?

I wouldn't worry about this at all. Users can rely on the 8.x versions for as long as those continue to work. As soon as 9.0.0 is released, everyone's expectation should be that 99%+ of click contributions are going into the 9.x line, and that 8.x will only get updates in unusual circumstances (e.g., security bugfix).

But the message should be "stop using 8.x as soon as you can."

---

One thing I've found valuable in recent projects is to write an "upgrading" guide as part of the change documentation for major releases. We maintain one for webargs, and we imitated that in a work project, globus-sdk.

To use my previous example, this means that if ParamType.get_metavar is becoming ParamType._get_metavar, the PR author for that change also has a place to write down what users who were using that method should do instead.

Footnotes

1. "intentional" being key. Doing internal cleanup, like moving around modules, which isn't supposed to impact users is, IMO, fine for that first cut. ↩

[davidism]

A difficult part about the "private" designation is that there's more nuance sometimes. We might be fine with people calling it, but once they override it becomes much hard to evolve.

[Rowlando13]

@sirosen @kdeldycke If anything from this list appeals to you, then make an issue and attach it to #3154, and we can discuss.

[untitaker]

I would caution against pushing out a major release that contains a lot of pent up breaking changes. Frankly the merits and risks of each change should be discussed and released separately

[Rowlando13]

Would you mind clarifying? Do you mean that releasing 1,2, and 3 is too much or "Make a lot of currently public methods private" is too vague and we should just consider each as method separately or something else?

[untitaker]

@Rowlando13 sorry for not responding. basically i'm suggesting to take a small breaking change, make a release with that and anticipate a wave of feedback from that. that feedback will probably influence how you roll out the rest. IMO you can't really incentivize folks to give you "proper" feedback until the damage is done, hence i suggest a slow rollout.

[Rowlando13]

Thanks for responding. That seems to be the consensus, so we are going with that.

[Rowlando13]

Based on input so far I think the tentative plan is split 1, 2, and 3 up. Anyone from the community is welcome to weigh in if they disagree.

[davidism]

The latest stuff with defaults, most recently fixed in #3224, is revealing a huge amount of internal mess. The problem is that the processing pipeline is split up into a ton of publicly documented methods on publicly documented classes. Since everything's considered public API, it could all be called by user code or overridden. We need to do internal bookkeeping across these various methods, but their public nature makes it hard to even anticipate what could happen.

Aside from the complexity of the problem, we're all also very occupied with other stuff in our lives right now. This has meant that we haven't been positioned to respond to recent issues quickly. While fixing the spaghetti of it all won't be an easy task, ideally it will make future maintenance easier.

I'd also like to get @tiangolo in on this, I did see that he was thinking about the direction of Typer recently and I'm sure Click's recent releases have been part of that.

It would be great to schedule some meetings for us maintainers to map out what we need.

[kdeldycke]

@davidism available for a meeting (virtually) anytime.

I did not responded to your invitation last time on Discord. Is the Pallets/Click community still hanging there?

[kdeldycke]

I'm still following the direction of fixing as much as we can in the code via 8.3.x/8.x releases, and use this as an excuse to write beefier unittests and document as much as we can. If I have to arbitrage, I'd rather have ugly code but bigger tests than the reverse. Big refactors and breakage can happens in a 9.x release later, as we would have more freedom at that time to break the public contract.

[davidism]

Ping me in the Pallets Discord https://discord.gg/pallets and I'll add you to the maintainers section.