[FR]: py_pex_binary support for arbitrary cross-builds

[plobsing]

What is the current behavior?

Due to the slightly hacky way py_pex_binary disassembles the output of a py_binary and reconstitutes it into an archive, it pulls in certain components as dependencies that don't make it into the final archive. But because they are tracked as dependencies, they need to be buildable for the target platform anyways or the build fails.

Notable among these is //py/tools/venv_bin:venv, which is a runtime dep for py_binary, but is not useful in the context of PEX. But, owing to its implementation either as a pre-built binary (in released rulesets) or a Rust binary (in dev rulesets), the targets that it is available for are limited. And this limitation is viral — it also applies to the overall PEX archive that the binary is a dep of (but gets stripped out from).

For example, attempting to compile a PEX archive targetting MacOS from a Linux build host fails for the dev ruleset because the Rust toolchain does not support cross-compiling this way.

(04:17:00) ERROR: /mnt/ephemeral/workdir/aspect-build/rules_py/py/tools/venv_bin/BUILD.bazel:4:12: While resolving toolchains for target //py/tools/venv_bin:venv_macos_aarch64_build (d30093f): No matching toolchains found for types @@rules_rust//rust:toolchain_type.
To debug, rerun with --toolchain_resolution_debug='@@rules_rust//rust:toolchain_type'
If platforms or toolchains are a new concept for you, we'd encourage reading https://bazel.build/concepts/platforms-intro.

In release, this cross-compile would work because the prebuilt toolchain for MacOS exists. But we also see similar problems targetting Windows, or more exotic OSes.

Describe the feature

Cross-compilation targets for PEX archives should be not be limited artificially by rules_py; if the PEX tool can run on the exec platform, and the Python dependencies of the backing py_binary are compatible with the target platform¹, we should be able to build an archive.

Footnotes

1. Either because each dep is pure-python, or has a pre-built wheel that's been pulled from PyPI, or the user has provided some other builds that works for the target platform. ↩

[plobsing]

FYI, I believe #587 may improve the situation with the runtime dependency on the //py/tools/venv_bin:venv tool. I'm not sure if that gets this feature all the way across the line or not, but it just might.