Use uv to manage Python dependencies #255

Sorixelle · 2025-07-18T14:03:44Z

Using plain pip+venvs to manage dependencies for build tooling has a few downsides:

Manual activation: Developers have to remember to activate the repo's venv with source .venv/bin/activate on each new terminal session, and possibly deactivate as well when they're done
Dependency locking: The requirements.txt only specifies ranges of versions, which can potentially cause breakages due to dependency updates without warning

While the second can be solved with just pip, the first has no easy solution. Instead, to tackle both issues, we can use uv, a new Python package manager which tackles these problems.

This PR creates a pyproject.toml file with dependencies imported from both requirements.txt and docs/requirements.txt, with the intention of using uv (or possibly some other tool later down the line, if uv stops being the new hotness for whatever reason) to install them. It also sets up both waf and the docs site build to use uv run for execution, automatically running them within uv's managed venv and saving developers from having to manually activate the venv themselves.

The docs have been updated to include uv setup instructions for both Ubuntu and macOS, and remove the venv setup step (since uv handles this automatically).

A few changes have also been made to the pyproject.toml files in python_libs, to ensure uv builds and installs the libraries inside it's venv correctly. The __init__.py files have been removed, as the declare_namespace calls were emitting deprecation warnings with recent versions of setuptools, and are obsoleted by PEP 420.

Hexxeh

I'm fine with this, let's get @gmarull to weigh in from Core side though.

jwise · 2025-07-25T01:40:25Z

ping @gmarull now that he's back in the office :)

Sorixelle · 2025-07-25T14:03:54Z

Not sure what's up with that build failure, can't reproduce it locally - maybe try rerun it, if it's still broken after that I'll dig into it further.

gmarull · 2025-07-25T14:13:18Z

I'm not really a fan of fancy new Python package managers unless they become the official package manager. Who knows if uv will be the last (it seems to be venture backed, scaring enough)? Pip, despite its many problems, has been running for years. The only advantage I see in uv is the lock file (something we can still manage with experimental pip lock), the others are mostly redundant on a project like ours where we install dependencies once and rarely touch them anymore. I'd probably have another take if we were a Python project. So my vote is a -1, I prefer to keep things simple. I've yet to find any issues with the current setup.

Hexxeh · 2025-07-25T14:41:35Z

I had similar thoughts, but I think the things that convinced me were:

It integrates with waf so you don't need to remember the virtualenv
There's no lock in. If uv goes out of fashion, you can just drop back to pip with little cost.

jwise · 2025-07-26T00:09:50Z

I kicked it to rerun it. The new SDK requires uv, so it makes sense to me that the firmware should too (and maintaining venvs by hand really does suck...).

jwise · 2025-07-26T00:12:20Z

@gmarull In light of @Hexxeh 's and my comments above, do you still -1?

gmarull · 2025-07-28T09:40:23Z

docs/Makefile

 # from the environment for the first two.
 SPHINXOPTS    ?=
-SPHINXBUILD   ?= sphinx-build
+SPHINXBUILD   ?= uv run sphinx-build


I prefer not to have this managed.

Same reason as waf

gmarull · 2025-07-28T09:42:19Z

waf

@@ -1,4 +1,4 @@
-#!/usr/bin/env python3
+#!/usr/bin/env -S uv run --script


same here, this creates a hidden development flow dependency on uv, which I'd avoid. If we use uv, let's make it transparent, explicit and optional.

It's not really a hidden dependency, I've documented the requirement in the getting-started docs. I can drop this, but it means you'd have to prefix every Python command with uv run (worse for the docs - more like SPHINXBUILD="uv run sphinx-build" make which is very unintuitive). Feels like putting a fair bit of overhead on developers, which is part of what I was trying to reduce with this PR.

the opposite version of this is: if you don't want to use uv, you can still python3 waf build

By hidden dependency I mean waf now depends on uv to run, you have no other choice. Considering uv is not the official Python package manager solution, I'd avoid this. We should be able to still use pip.

I'd agree it's hidden in a sense, but as @jwise pointed out you do have another choice.

This is also to my mind the biggest benefit of uv for me, so I'd prefer to keep it as it is currently in this PR.

I'm not sold this is an advantage. If activating the venv is an issue, there are quite a few shell scripts/plugins etc that will automatically load venvs if found when entering a directory, and then, regardless of who created such venv or which tool, commands will work as usual.

It doesn't - as Joshua mentioned, you can still use python3 waf and it will be run with whatever Python is currently in PATH, without interacting with uv at all.

I know, but that's precisely what I do not want. This forces a different workflow if I do not stick with uv.

.python-version

uv.lock

gmarull · 2025-07-28T09:58:37Z

pyproject.toml

+    "svg-path>=7.0",
+]
+
+[tool.uv.sources]


isn't there any standard way to define this? it worked pre-uv...

Looks like there is. I'll do that instead.

Actually, nope, I've run into astral-sh/uv#3271 - doesn't work if you run uv outside of the repo root (eg. in docs/). Will probably have to put that one back, unless there's any better way to do it.

Sorixelle · 2025-08-01T04:20:01Z

I'm going to look into that build failure. nanopb/nanopb#599 seems relevant, some sort of race condition probably?

Sorixelle · 2025-08-01T05:08:38Z

Think I squashed it. Solution feels a little hacky, so I'm open to better suggestions.

gmarull

My only objection now is to not force uv, ie, make it part of official GSG, but keep the option to use pip if someone prefers to (of course it'll become 'officially unsupported', same as other toolchains than the ARM GNU). I think it's a more future-proof solution.

gmarull · 2025-08-01T08:18:43Z

waf

@@ -1,4 +1,4 @@
-#!/usr/bin/env python3
+#!/usr/bin/env -S uv run --script


By hidden dependency I mean waf now depends on uv to run, you have no other choice. Considering uv is not the official Python package manager solution, I'd avoid this. We should be able to still use pip.

gmarull · 2025-08-01T08:19:33Z

docs/Makefile

 # from the environment for the first two.
 SPHINXOPTS    ?=
-SPHINXBUILD   ?= sphinx-build
+SPHINXBUILD   ?= uv run sphinx-build


Same reason as waf

pyproject.toml

Dependencies imported from requirements.txt with `uv add -r requirements.txt`. Signed-off-by: Ruby Iris Juric <[email protected]>

Signed-off-by: Ruby Iris Juric <[email protected]>

Imported with `uv add -r docs/requirements.txt`. Signed-off-by: Ruby Iris Juric <[email protected]>

The packages weren't being installed into uv's venv, since they had no build system defined. The declare_namespace calls are also deprecated in recent setuptools, and are no longer necessary. Signed-off-by: Ruby Iris Juric <[email protected]>

Signed-off-by: Ruby Iris Juric <[email protected]>

nanopb_generator will attempt to use protoc to generate a Python file as part of it's operation, if that file doesn't exist yet. It seems like when waf runs those generations in parallel, there's a race to generate that file that results in build failures when the race is lost. To resolve this, we call the function that generates that file during configure so we avoid the parallel race during build. Signed-off-by: Ruby Iris Juric <[email protected]>

gmarull · 2025-08-01T17:14:22Z

So my last take on this: happy to use uv, but as soon as dev workflow is not altered if using standard venv/pip. In any case, feel free to merge whatever you think is best, but I won't ack.

github-actions · 2025-09-01T01:06:27Z

This pull request has been marked as stale because it has been open (more than) 30 days with no activity. Remove the stale label or add a comment saying that you would like to have the label removed otherwise this pull request will automatically be closed in 7 days. Note, that you can always re-open a closed pull request at any time.

Hexxeh approved these changes Jul 18, 2025

View reviewed changes

Sorixelle force-pushed the python-uv branch 5 times, most recently from 45bfb8b to 69b0d73 Compare July 20, 2025 11:37

gmarull reviewed Jul 28, 2025

View reviewed changes

uv.lock Show resolved Hide resolved

gmarull reviewed Jul 28, 2025

View reviewed changes

Sorixelle force-pushed the python-uv branch 2 times, most recently from 3c09c68 to d783c65 Compare August 1, 2025 03:51

Sorixelle force-pushed the python-uv branch 4 times, most recently from 15a32fb to ada73a9 Compare August 1, 2025 05:04

gmarull reviewed Aug 1, 2025

View reviewed changes

Sorixelle added 6 commits August 2, 2025 01:25

pyproject: create pyproject.toml with uv

6365587

Dependencies imported from requirements.txt with `uv add -r requirements.txt`. Signed-off-by: Ruby Iris Juric <[email protected]>

requirements: add uv to brew requirements

5dc794b

Signed-off-by: Ruby Iris Juric <[email protected]>

waf: change shebang to run with uv environment

62582e1

Signed-off-by: Ruby Iris Juric <[email protected]>

docs: update getting started page with uv setup instructions

1a3fa81

Signed-off-by: Ruby Iris Juric <[email protected]>

docs: add dependencies for build to root pyproject.toml

46f2936

Imported with `uv add -r docs/requirements.txt`. Signed-off-by: Ruby Iris Juric <[email protected]>

Sorixelle added 2 commits August 2, 2025 01:27

ci: remove pip install step, replace with uv install

3919f7a

Signed-off-by: Ruby Iris Juric <[email protected]>

Sorixelle force-pushed the python-uv branch from ada73a9 to 6345dbb Compare August 1, 2025 15:27

github-actions bot added the stale Stale issue or PR (will be closed soon) label Sep 1, 2025

github-actions bot closed this Sep 9, 2025

		@@ -1,4 +1,4 @@
		#!/usr/bin/env python3
		#!/usr/bin/env -S uv run --script

Use uv to manage Python dependencies #255

Use uv to manage Python dependencies #255

Uh oh!

Conversation

Sorixelle commented Jul 18, 2025

Uh oh!

Hexxeh left a comment

Choose a reason for hiding this comment

Uh oh!

jwise commented Jul 25, 2025

Uh oh!

Sorixelle commented Jul 25, 2025

Uh oh!

gmarull commented Jul 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Hexxeh commented Jul 25, 2025

Uh oh!

jwise commented Jul 26, 2025

Uh oh!

jwise commented Jul 26, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Sorixelle commented Aug 1, 2025

Uh oh!

Sorixelle commented Aug 1, 2025

Uh oh!

gmarull left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gmarull commented Aug 1, 2025

Uh oh!

github-actions bot commented Sep 1, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gmarull commented Jul 25, 2025 •

edited

Loading