feat(bzlmod): cross-platform builds without experimental_index_url (#2325)

With this change we finally generate the same lock file within the
legacy code `pip.parse` code path and it allows to slowly transition to
using the new code path as much as possible without user doing anything.

This moves the selection of the host-compatible lock file from the
extension evaluation to the build phase - note, we will generate extra
repositories here which might not work on the host platform, however, if
the users are consuming the `whl_library` repos through the hub repo
only, then everything should work. A known issue is that it may break
`bazel query` and in these usecases it is advisable to use `cquery`
until we have `sdist` cross-building from source fully working.

Summary:
- feat: reuse the `render_pkg_aliases` for when filename is not known
  but platform is known
- feat: support generating the extra config settings required
- feat: `get_whl_flag_versions` now generates extra args for the rules
- feat: make lock file generation the same irrespective of the host
  platform
- test: add an extra test with multiple requirements files
- feat: support cross-platform builds using `download_only = True` in
  legacy setups

Note, that users depending on the naming of the whl libraries will need
to start using `extra_hub_aliases` attribute instead to keep their
setups not relying on this implementation detail.

Fixes #2268
Work towards #260

---------

Co-authored-by: Richard Levasseur <[email protected]>

Author: aignas

CHANGELOG.md

@@ -39,6 +39,12 @@ A brief description of the categories of changes:
   [`pip.parse#extra_pip_args`](https://rules-python.readthedocs.io/en/latest/api/rules_python/python/extensions/pip.html#pip.parse.extra_pip_args)
 * (pip.parse) {attr}`pip.parse.whl_modifications` now normalizes the given whl names
   and now `pyyaml` and `PyYAML` will both work.
+* (bzlmod) `pip.parse` spoke repository naming will be changed in an upcoming
+  release in places where the users specify different package versions per
+  platform in the same hub repository. The naming of the spoke repos is considered
+  an implementation detail and we advise the users to use the `hub` repository
+  directly to avoid such breakage in the future. If `rules_python` is missing
+  features to allow one to do that, please raise tickets.
 
 {#v0-0-0-fixed}
 ### Fixed
@@ -51,6 +57,12 @@ A brief description of the categories of changes:
   pass the `extra_pip_args` value when building an `sdist`.
 * (pypi) The patched wheel filenames from now on are using local version specifiers
   which fixes usage of the said wheels using standard package managers.
+* (bzlmod) The extension evaluation has been adjusted to always generate the
+  same lock file irrespective if `experimental_index_url` is set by any module
+  or not. Fixes
+  [#2268](https://github.com/bazelbuild/rules_python/issues/2268). A known
+  issue is that it may break `bazel query` and in these use cases it is
+  advisable to use `cquery` or switch to `download_only = True`
 
 {#v0-0-0-added}
 ### Added
@@ -63,6 +75,11 @@ A brief description of the categories of changes:
 * (pip.parse) {attr}`pip.parse.extra_hub_aliases` can now be used to expose extra
   targets created by annotations in whl repositories.
   Fixes [#2187](https://github.com/bazelbuild/rules_python/issues/2187).
+* (bzlmod) `pip.parse` now supports `whl-only` setup using
+  `download_only = True` where users can specify multiple requirements files
+  and use the `pip` backend to do the downloading. This was only available for
+  users setting {bzl:obj}`pip.parse.experimental_index_url`, but now users have
+  more options whilst we continue to work on stabilizing the experimental feature.
 
 {#v0-0-0-removed}
 ### Removed

MODULE.bazel

@@ -56,6 +56,10 @@ register_toolchains("@pythons_hub//:all")
 
 pip = use_extension("//python/private/pypi:pip.bzl", "pip_internal")
 pip.parse(
+    # NOTE @aignas 2024-10-26: We have an integration test that depends on us
+    # being able to build sdists for this hub, so explicitly set this to False.
+    download_only = False,
+    experimental_index_url = "https://pypi.org/simple",
     hub_name = "rules_python_publish_deps",
     python_version = "3.11",
     requirements_by_platform = {
@@ -90,17 +94,20 @@ dev_python.override(
 )
 
 dev_pip = use_extension(
-    "//python/private/pypi:pip.bzl",
-    "pip_internal",
+    "//python/extensions:pip.bzl",
+    "pip",
     dev_dependency = True,
 )
 dev_pip.parse(
-    download_only = True,  # this will not add the `sdist` values to the transitive closures at all.
+    download_only = True,
+    experimental_index_url = "https://pypi.org/simple",
     hub_name = "dev_pip",
     python_version = "3.11",
     requirements_lock = "//docs:requirements.txt",
 )
 dev_pip.parse(
+    download_only = True,
+    experimental_index_url = "https://pypi.org/simple",
     hub_name = "pypiserver",
     python_version = "3.11",
     requirements_lock = "//examples/wheel:requirements_server.txt",

docs/pypi-dependencies.md

@@ -307,6 +307,59 @@ leg of the dependency manually. For instance by making
 perhaps `apache-airflow-providers-common-sql`.
 
 
+(bazel-downloader)=
+### Multi-platform support
+
+Multi-platform support of cross-building the wheels can be done in two ways - either
+using {bzl:attr}`experimental_index_url` for the {bzl:obj}`pip.parse` bzlmod tag class
+or by using the {bzl:attr}`pip.parse.download_only` setting. In this section we
+are going to outline quickly how one can use the latter option.
+
+Let's say you have 2 requirements files:
+```
+# requirements.linux_x86_64.txt
+--platform=manylinux_2_17_x86_64
+--python-version=39
+--implementation=cp
+--abi=cp39
+
+foo==0.0.1 --hash=sha256:deadbeef
+bar==0.0.1 --hash=sha256:deadb00f
+```
+
+```
+# requirements.osx_aarch64.txt contents
+--platform=macosx_10_9_arm64
+--python-version=39
+--implementation=cp
+--abi=cp39
+
+foo==0.0.3 --hash=sha256:deadbaaf
+```
+
+With these 2 files your {bzl:obj}`pip.parse` could look like:
+```
+pip.parse(
+    hub_name = "pip",
+    python_version = "3.9",
+    # Tell `pip` to ignore sdists
+    download_only = True,
+    requirements_by_platform = {
+        "requirements.linux_x86_64.txt": "linux_x86_64",
+        "requirements.osx_aarch64.txt": "osx_aarch64",
+    },
+)
+```
+
+With this, the `pip.parse` will create a hub repository that is going to
+support only two platforms - `cp39_osx_aarch64` and `cp39_linux_x86_64` and it
+will only use `wheels` and ignore any sdists that it may find on the PyPI
+compatible indexes.
+
+```{note}
+This is only supported on `bzlmd`.
+```
+
 (bazel-downloader)=
 ### Bazel downloader and multi-platform wheel hub repository.
 

examples/bzlmod/MODULE.bazel

@@ -221,6 +221,9 @@ pip.parse(
         "host",
     ],
     hub_name = "pip",
+    # Parse all requirements files for the same lock file on all OSes, this will
+    # become the default with 1.0 release
+    parse_all_requirements_files = True,
     python_version = "3.10",
     # The requirements files for each platform that we want to support.
     requirements_by_platform = {

examples/bzlmod/MODULE.bazel.lock

@@ -148,6 +148,13 @@ def config_settings(
             )
 
 def _dist_config_settings(*, suffix, plat_flag_values, **kwargs):
+    if kwargs.get("constraint_values"):
+        # Add python version + platform config settings
+        _dist_config_setting(
+            name = suffix.strip("_"),
+            **kwargs
+        )
+
     flag_values = {_flags.dist: ""}
 
     # First create an sdist, we will be building upon the flag values, which
@@ -277,7 +284,7 @@ def _plat_flag_values(os, cpu, osx_versions, glibc_versions, muslc_versions):
 
     return ret
 
-def _dist_config_setting(*, name, is_pip_whl, is_python, python_version, native = native, **kwargs):
+def _dist_config_setting(*, name, is_python, python_version, is_pip_whl = None, native = native, **kwargs):
     """A macro to create a target that matches is_pip_whl_auto and one more value.
 
     Args:
@@ -310,6 +317,10 @@ def _dist_config_setting(*, name, is_pip_whl, is_python, python_version, native
         # `python_version` setting.
         return
 
+    if not is_pip_whl:
+        native.config_setting(name = _name, **kwargs)
+        return
+
     config_setting_name = _name + "_setting"
     native.config_setting(name = config_setting_name, **kwargs)