Document the supported interfaces of Setuptools

[jaraco]

[Let's] make it clear that any software that relies on undocumented functionality is unsupported and can break at any time.

Originally posted by @jaraco in #4963 (comment)

[tiran]

Please keep the this bullet point of the PSF Code of Conduct in your mind:

Focusing on what is best for the community. We're respectful of the processes set forth in the community, and we work within them.

setuptools had four releases with major breaking changes in the last six weeks. Do you think that it is in the best interesting of the Python community that a fundamental core piece of Python packaging is causing disruption?

[jaraco]

Do you think that it is in the best interest of the Python community that a fundamental core piece of Python packaging is causing disruption?

Definitely not. Stability is important, and Setuptools has a particularly higher responsibility for stability.

The instability recently is due to me making another push to disentangle old and deprecated and security-sensitive functionality that's been declared obsolete since 2009 or earlier ("pip as new and improved vs. easy_install old and busted"). I wasn't the person who devised this architecture (distutils) nor the one who decided to replace it (with replacements like pip or pep 517 or twine or build) without having a strategy for retiring/transitioning the established tooling.

I'm doing the best I can as one person working in their spare time to work through these challenges and balance stability with sustainability. The solution can't be "don't ever change it" or "any change requires a multi-year deprecation period," so I'm working with the tools I have at my disposal to do my best to make measured risks and respond quickly when things go awry. I'm open to more robust integration testing or other resources to facilitate the changes that this project needs to make to be sustainable.

[tiran]

Have you considered that setuptools is a lost cause and that your energy would be spent better differently? There is just too much software that relies on undocumented or internal features of setuptools. In my opinion it would be better to declare setuptools legacy software and to put it on indefinite life support (no removal or addition of features except for new Python versions and packaging PEPs).

Thinking out loud:
The Python community would benefit from tooling to convert old-style projects with setup.py and setup.cfg to modern pyproject.toml. You could fork setuptools as setuptools-core, rip out the old cruft, and turn it into the successor of setuptools.

[jaraco]

I have thought of that. I think about it all the time.

I feel like flit and hatch have already filled two niches that setuptools-core might, and the last thing I want to do is create yet another fork that needs to die or be merged back.

Additionally, flit is nowhere close to filling the vision I have for streamlined packaging. Hatch is pretty close (close enough that I maybe should have adopted it before embarking on building something new), but it also has its own problems (is at risk for the bootstrapping problem, still requires substantial boilerplate). But what I'm really after is something that frees individual developers and projects from the toil of per-project management and leverages tooling and re-use to allow the code to be primary once again, much as it is for packages in an enterprise monorepo environment. I'm working out the kinks on the coherent build system, and it's getting close enough to being able to replace my own usage of Setuptools while also nearly eliminating cross-project maintenance concerns.

For this (or any other) system to replace Setuptools, it'll need to support compiling extension modules, which is why as I'm working to merge distutils into Setuptools, I'm also working toward extracting the compilers behaviors to make them available to other backends.

What I struggle to imagine is a path that somehow migrates legacy Setuptools users, the ones with highly entangled setup.py scripts, to anything new without breaking their builds. The easy projects are easy. There's already converters from setup.py to setup.cfg to pyproject.toml, and once you've migrated to pyproject.toml, it's a small jump to hatch or flit (or other backends) if you aren't using setup.py at all.

I have considered abandoning setuptools, letting it grow stale and leveraging that pain to migrate users to other systems (and incentivizing the support of those systems), but in my experience, it's less work in the long run to make incremental shifts toward that vision by implementing the transition instead of asking every user to make the leap. I think the Python 2 to 3 transition was a good case-in-point. If each major breaking change (unicode as str, module renames, builtin renames, etc) had been implemented independently and each with a transition strategy in place (something like six), it would have been more of an upgrade than a port.

The main thing that keeps me from abandoning setuptools is that it's got such a large following, so I want to keep all of those use cases viable (even if that does mean some disruption as I discover the unexpected dependencies).