Add experimental uv-based PEX builder ([python].pex_builder)

[seungwoo-ji-03]

Closes #20679

Add an experimental [python].pex_builder option that allows using
uv to install dependencies when
building PEX files via pants package.

When set to "uv", Pants:

1. Downloads the uv binary (as an ExternalTool).
2. Creates a virtual environment with uv venv.
3. Installs dependencies with uv pip install.
4. Passes the pre-populated venv to PEX via --venv-repository.

When a PEX-native lockfile is available, uv installs the exact pinned
versions with --no-deps, preserving reproducibility. Otherwise it
falls back to transitive resolution from requirement strings.

Builds that cannot use uv (internal-only, cross-platform, no local
interpreter) silently fall back to the default pip path.

Benchmark

Raw pip install vs uv pip install for 11 packages (requests, boto3,
cryptography, aiohttp, sqlalchemy, pillow, etc.) measured with
hyperfine:

Condition	pip	uv	Speedup
Cold cache	6.3s	4.1s	1.6x faster
Warm cache	6.9s	0.14s	51x faster

Within Pants, the end-to-end improvement is smaller because scheduler
and bootstrap overhead dominates, but the dependency installation step
itself is significantly faster — especially with warm caches on
repeated builds.

LLM Disclosure

Code was written by the author. Claude was used for code review, catching edge cases, and verifying test coverage.

[jsirois]

Thanks for taking this up @seungwoo-ji-03. I hope you have better luck than previous folks attempting to un-stick Pants here. I encourage you to push hard on Pants reviewers to pay attention.

I don't use / maintain Pants, but I maintain Pex and added the --venv-repository feature you're using here exactly for this purpose; so I left a note about how you could take this further if you wish in follow-ups.

[jsirois]

FYI: You can do foreign platform builds with uv + Pex. You can also do multi-platform PEX builds (You can create multiple venvs and specify --venv-repository more than once when building the PEX). I demonstrate a single foreign platform PEX build below:

# I'm on x86_64 Linux:
:; uname -om
x86_64 GNU/Linux

# Build a venv for aarch64 linux using uv:
:; rm -rf /tmp/example/
:; uv venv --python 3.13 --no-project /tmp/example
Using CPython 3.13.12 interpreter at: /home/jsirois/.pyenv/versions/3.13.12/bin/python3.13
Creating virtual environment at: /tmp/example
Activate with: source /tmp/example/bin/activate

:; uv pip install --prefix /tmp/example --python 3.13 --python-platform aarch64-unknown-linux-gnu psutil cowsay
Using CPython 3.13.12 interpreter at: /home/jsirois/.pyenv/versions/3.13.12/bin/python3.13
Resolved 2 packages in 1ms
░░░░░░░░░░░░░░░░░░░░ [0/2] Installing wheels...                                                                                                                                                warning: Failed to hardlink files; falling back to full copy. This may lead to degraded performance.
         If the cache and target directories are on different filesystems, hardlinking may not be supported.
         If this is intentional, set `export UV_LINK_MODE=copy` or use `--link-mode=copy` to suppress this warning.
Installed 2 packages in 2ms
 + cowsay==6.1
 + psutil==7.2.2

# Use that foreign platform venv to create a foreign platform PEX:
:; pex --venv-repository /tmp/example/ -o example.pex
example.pex

# Prove it works:
:; docker run --rm --platform linux/arm64 -v $PWD/example.pex:/example.pex python:3.13 python /example.pex -c 'import cowsay, psutil; cowsay.tux(str(psutil.Process()))'
  _________________________________________________
 /                                                 \
| psutil.Process(pid=1, name='python', status='runn |
| ing')                                             |
 \                                                 /
  =================================================
                                                      \
                                                       \
                                                        \
                                                         .--.
                                                        |o_o |
                                                        |:_/ |
                                                       //   \ \
                                                      (|     | )
                                                     /'\_   _/`\
                                                     \___)=(___/

:; unzip -qc example.pex PEX-INFO | jq .distributions
{
  "cowsay-6.1-py3-none-any.whl": "cb195ef66765a3fa4c4a1243417d30d04849faa0d2de8eda533b807d59da0e50",
  "psutil-7.2.2-cp36-abi3-manylinux_2_17_aarch64.manylinux2014_aarch64.manylinux_2_28_aarch64.whl": "ac787d25959dbae14884467d31b49fe20915ae22c8cc10793c93fae2243ef470"
}

[seungwoo-ji-03]

Thanks for the detailed example @jsirois and for building --venv-repository in the first place, it's exactly what makes this work!

Cross-platform support via uv pip install --python-platform and multiple --venv-repository flags looks great. If Pants reviewers think it fits in this PR's scope, I'm happy to add it here — otherwise it'd be a natural follow-up.

[benjyw]

Let's do this in a follow up.

[benjyw]

Thanks for this!

There is an existing PR for integrating uv that is currently awaiting responses by the author: #22949. There is some overlap (installing uv as an ExternalTool), but that PR is focused on using uv for resolving. The PEX building code you augment here is used widely not just in package but throughout the Python backend.

In fact, a future follow up to this PR could support uv lockfiles directly, which would obviate that other PR, but one thing at a time.

I will review this one orthogonally to that one though. Meanwhile please note our new AI disclosure policy in case it is relevant to this change. Thanks!

[seungwoo-ji-03]

Thanks for this!

There is an existing PR for integrating uv that is currently awaiting responses by the author: #22949. There is some overlap (installing uv as an ExternalTool), but that PR is focused on using uv for resolving. The PEX building code you augment here is used widely not just in package but throughout the Python backend.

In fact, a future follow up to this PR could support uv lockfiles directly, which would obviate that other PR, but one thing at a time.

I will review this one orthogonally to that one though. Meanwhile please note our new AI disclosure policy in case it is relevant to this change. Thanks!

Thanks @benjyw!

I looked through #22949 before starting — the scope is different enough so I proceeded independently. Happy to rebase whichever lands second.

LLM disclosure added to the PR description. Thanks for the pointer.

[benjyw]

Thanks again for this impactful change. See my comments in line.

[benjyw]

Let's do this in a follow up.

[benjyw]

We shouldn't need to fall back to PEX/pip in this case. If we're in the WholeLockfile situation, then we can get the entire set of input reqs from the lockfile metadata (in generated_with_requirements). And if there are truly no requirements, then that is a trivial case.

Fine to do this in a followup though.

[seungwoo-ji-03]

Thanks for the quick and thorough review @benjyw! All feedback has been addressed except the two items marked as follow-ups

And thanks for maintaining such a great project!

[benjyw]

Looks great! Just one more comment and I think this will be ready to merge.

[vrazdalovschi]

Such amazing news to see this happen! Looking forward to the next pants release to try it out / test

[benjyw]

Great! I will merge on green.