Switch from requirements files to pyproject.toml and uv lock file

[agjohnson]

In #12164 we tried installing ext-theme through an easier pattern, but hit some trouble due to our
rel/relcorp pattern on all our repositories.

Overall, requirements files are an old pattern and I'd feel better about spending time working
towards a modern solution. I find the pip tools solution is mush better than what we had previously but do find
it to also be confusing to figure out what is being installed and from where.

We use uv to install dependencies, but consolidating this all to pyproject.toml and uv could be a next step.
In this, the pip compile input file becomes our dependency specification, the uv lock file becomes
what our requirements files are -- a snapshot of the exact versions/commits installed. Tox doesn't
handle uv well just yet, but there is a uv extension at least.

The main quesiton, and we hit this in #12264, is how to handle our release branches (rel/relcorp).
There seem two paths here:

Reduce complexity, stop using them on dependencies

There is a fair amount of complexity in the rel/relcorp release branches, it would be worth
exploring if we really need them for at least our own dependencies (ext, ext-theme, etc). These libraries
rarely have splits in code between rel/relcorp, the release branches are mostly to snapshot the
release and to allow for hotfixing.

We may need to keep rel/relcorp at the application at least, we should decide on this separately.

For the other repositories we could determine if:

• We want a single release branch
• We don't want a release branch at all, treat main as release and rely mostly on the lock file.

In both we rely on the lock file to snapshot what went into the release ¹.
Also in both, our pyproject.toml is easier to manage and it just specifies a single dependency.

Without a release branch, hotfixing an ext/ext-theme change might be installing the repository
commit sha and commiting the lock file change. This would be new.

Keep our release branch pattern

In this, we kick our past design down the road again and just reimplement it in pyproject.toml. We
still get a reduction in complexity between multiple requirements files and pip compile inputs, and
we still get a lock file that snapshots the exact release versions/commits.

There might be multiple options:

• We use optional dependencies to specify a rel, relcorp, dev optional dependencies
• We use conditional dependencies² to install rel, relcorp, main branches -- I believe this looks a lot like optional dependencies though.

Optional dependencies might be like:

[project.optional-dependencies]
dev = [
  "http://github.com/readthedocs/ext@main",
]
rel = [
  "http://github.com/readthedocs/ext@rel",
]
relcorp = [
  "http://github.com/readthedocs/ext@relcorp",
]

• Refs Install ext-theme from requirements.txt #12264

Footnotes

1. uv should do what we want here, it pins a commit SHA in the lock file: "uv applies similar
   logic to Git dependencies. For example, if a Git dependency references the main branch, uv will
   prefer the locked commit SHA in an existing uv.lock file over the latest commit on the main
   branch" ↩

2. https://peps.python.org/pep-0508/ ↩

[humitos]

Tox doesn't
handle uv well just yet, but there is a uv extension at least.

We tried this in the past and we got blocked. It should be pretty close, tho: #11851

hotfixing an ext/ext-theme change might be installing the repository
commit sha and commiting the lock file change. This would be new.

We will need to do this on every release as well, because the lock file will need to be updated with the latest changes merged into main from these repositories.

---

In general, I don't see how switching from pip-tools to uv will solve the problems we currently have. It seems we will be just changing the tooling but we will still have the same problems.

I do see some benefit on using optional dependencies in the pyproject.toml, tho. That would allow us to run pip install .[rel] for community and pip install .[relcorp] for corporate --but we don't need to change the underlying tool for that. We should probably try that change with the current pattern first, since it seems a very scoped task.

Also note that we are not even using uv on production currently, so the tool change is not trivial work.

[agjohnson]

In general, I don't see how switching from pip-tools to uv

Well pip-tools isn't useful if we switch to pyproject.toml, and pyproject.toml isn't as useful unless we can still lock dependencies. The tool change isn't meant to solve an issue directly, uv just gets us everything in one package -- locking dependencies, pyproject.toml support, and an install speed improvement.

We will need to do this on every release as well

Yeah, this would take the place of merging to rel/relcorp on every repository. Instead we'd update the lock file on the application repo and commit that to rel/relcorp on readthedocs.org repo. The lock file becomes the place where ext-theme/ext/etc commit used for release is defined.

but we don't need to change the underlying tool for that

We do still need dependency pinning for all of our external dependencies though. So those can't be defined in a pyproject.toml without a lock file or manually managing upgrading. While it would be possible to have internal dependencies in a pyproject.toml and keep external dependencies managed in a requirements file managed by pip tools, that's definitely a more confusing pattern than what we have now.

We could try this, but I'd also say we could wait on switching to pyproject.toml until we're ready to make the shift to uv.

[humitos]

Well pip-tools isn't useful if we switch to pyproject.toml, and pyproject.toml isn't as useful unless we can still lock dependencies.

Good point 👍🏼

We could try this, but I'd also say we could wait on switching to pyproject.toml until we're ready to make the shift to uv.

Yeah, if we are going in this direction I think we should make it in different steps so it's easier to split this work:

1. use uv in production: we need to update how we create the environment and install the dependencies
2. use pyproject.toml and optional dependencies to install ext-theme
3. migrate from requirements.txt files to uv.lock files for local development
4. use uv.lock files in production

[agjohnson]

That seems like a pretty reasonable upgrade path, breaking this up into chunks is a good idea.

I think we can do 3 steps too, and switch from pip tools to uv in one step but still use uv to output the rendered requirement(s) files. This is basically a hacky lockfile approach. It might take restructuring our requirements files a little though -- instead of grouping for inheritance, we just have one for dev, one for prod, etc. I don't know if uv can output requirements files in the inheritance hierarchy style we're using now.