Proposed test suite improvements and clean ups

[ichard26]

Hello!

Now that I finally have some free time, I would like to spend some time dealing with the cruft that has acculumated in our test suite. ✨

I'm opening this issue for two reasons:

• To inform the rest of y'all what I'm up to
• To ask for permission to merge PRs without review. Given the volume of changes I would like to make, it's likely unfeasible to expect reviews for everything. Also not having to wait for reviews would make it a lot easier to iterate which is important since I only have free time for a limited period of time. I'll make sure to get feedback for any more substantive changes, but since everything is going to be test suite/internal only, the risks are pretty low IMO. @pfmoore @notatallshaw

Performance

Our CI run times have crept up again. Total run durations are solidly in the 20 minutes range now which is not ideal.

• Identify abnormally slow tests and rewrite them if possible
• Include more packages as local test data to avoid hitting PyPI
• Explore spinning up a local caching index server
• Switch to flit-core as the backend for test/data packages (this will probably need reviews¹)
  setuptools is a heavy build backend. Its import times are terrible which is doubly bad with all of the subprocesses that PEP 517 processing entails.

Quality of Life

• Make it much easier to install things without ever hitting the network. I'd like to extend the pip_install_local() helper, for one. See also PR test: Extend pip_install_local() test helper #13484.
• Merge or at least rationalise the differences between result.assert_installed() vs script.assert_installed()
• Update the development documentation to include code examples for common scenarios (so we don't just cargo cult legacy testing patterns from literally 5 year old tests)

---

I'm sure I'll think of more ideas as I go, but this is what on my mind so far.

Footnotes

1. There are probably a lot of older tests that wouldn't play nice with this, but there's no reason why modern test packages need to use setuptools. ↩

[pfmoore]

Sounds reasonable to me!

[notatallshaw]

Personally I am fine with maintainers merging low impact low risk PRs by themselves, this applies to bigger changes to the test suite due to it not directly impacting pip behavior.