feat(bzlmod)!: Calling pip multiple times allowing for multiple Python versions (#1254)

Implementing the capability to call pip bzlmod extension multiple times.
We can now call the pip extension with the same hub name multiple times.
This allows a user to have multiple different requirement files based on
the Python version.

With workspace, we need incompatible aliases because you get
@pip//tabulate or @pip_tabulate//:pkg.
The requirement() macro hides this, but changing the flag becomes a
breaking change if you've manually referenced things. With bzlmod,
though, the @pip_tabulate style isn't a realistic option (you'd have to
use_repo everything, which is a huge pain).

So we have chosen to have `@pip//tabulate`.

This commit implements that capability for pip.parse to determine the
Python version from programmatically
the provided interpreter.

See 


https://github.com/bazelbuild/rules_python/blob/76aab0f91bd614ee1b2ade310baf28c38695c522/python/extensions/pip.bzl#L88

For more in-depth implementation details.

INTERFACE CHANGE::

- pip.parse argument python_version is introduced but not required. When
possible, it is
  inferred.

BREAKING CHANGE:
* `pip.parse()` extension: the `name` attribute is renamed `hub_name`.
This is to reflect
that the user-provided name isn't unique to each `pip.parse()` call. We
now have a hub that is created, and each
   pip.parse call creates a spoke.
* `pip.parse()` extension: the `incompatible_generate_aliases` arg is
removed; behavior
     has changed to assume it is always True.
* `pip.parse()` calls are collected under the same `hub_name` to support
multiple Python versions under the same resulting repo name (the hub
name0.


Close: #1229

Author: chrislovecnm

.bazelrc

@@ -3,8 +3,8 @@
 # This lets us glob() up all the files inside the examples to make them inputs to tests
 # (Note, we cannot use `common --deleted_packages` because the bazel version command doesn't support it)
 # To update these lines, run tools/bazel_integration_test/update_deleted_packages.sh
-build --deleted_packages=examples/build_file_generation,examples/build_file_generation/random_number_generator,examples/bzlmod,examples/bzlmod_build_file_generation,examples/bzlmod_build_file_generation/other_module/other_module/pkg,examples/bzlmod_build_file_generation/runfiles,examples/bzlmod/entry_point,examples/bzlmod/other_module/other_module/pkg,examples/bzlmod/runfiles,examples/bzlmod/tests,examples/multi_python_versions/libs/my_lib,examples/multi_python_versions/requirements,examples/multi_python_versions/tests,examples/pip_install,examples/pip_parse,examples/pip_parse_vendored,examples/pip_repository_annotations,examples/py_proto_library,tests/compile_pip_requirements,tests/compile_pip_requirements_test_from_external_workspace,tests/ignore_root_user_error,tests/pip_repository_entry_points
-query --deleted_packages=examples/build_file_generation,examples/build_file_generation/random_number_generator,examples/bzlmod,examples/bzlmod_build_file_generation,examples/bzlmod_build_file_generation/other_module/other_module/pkg,examples/bzlmod_build_file_generation/runfiles,examples/bzlmod/entry_point,examples/bzlmod/other_module/other_module/pkg,examples/bzlmod/runfiles,examples/bzlmod/tests,examples/multi_python_versions/libs/my_lib,examples/multi_python_versions/requirements,examples/multi_python_versions/tests,examples/pip_install,examples/pip_parse,examples/pip_parse_vendored,examples/pip_repository_annotations,examples/py_proto_library,tests/compile_pip_requirements,tests/compile_pip_requirements_test_from_external_workspace,tests/ignore_root_user_error,tests/pip_repository_entry_points
+build --deleted_packages=examples/build_file_generation,examples/build_file_generation/random_number_generator,examples/bzlmod,examples/bzlmod/entry_point,examples/bzlmod/libs/my_lib,examples/bzlmod/other_module/other_module/pkg,examples/bzlmod/runfiles,examples/bzlmod/tests,examples/bzlmod_build_file_generation,examples/bzlmod_build_file_generation/other_module/other_module/pkg,examples/bzlmod_build_file_generation/runfiles,examples/multi_python_versions/libs/my_lib,examples/multi_python_versions/requirements,examples/multi_python_versions/tests,examples/pip_install,examples/pip_parse,examples/pip_parse_vendored,examples/pip_repository_annotations,examples/py_proto_library,tests/compile_pip_requirements,tests/compile_pip_requirements_test_from_external_workspace,tests/ignore_root_user_error,tests/pip_repository_entry_points
+query --deleted_packages=examples/build_file_generation,examples/build_file_generation/random_number_generator,examples/bzlmod,examples/bzlmod/entry_point,examples/bzlmod/libs/my_lib,examples/bzlmod/other_module/other_module/pkg,examples/bzlmod/runfiles,examples/bzlmod/tests,examples/bzlmod_build_file_generation,examples/bzlmod_build_file_generation/other_module/other_module/pkg,examples/bzlmod_build_file_generation/runfiles,examples/multi_python_versions/libs/my_lib,examples/multi_python_versions/requirements,examples/multi_python_versions/tests,examples/pip_install,examples/pip_parse,examples/pip_parse_vendored,examples/pip_repository_annotations,examples/py_proto_library,tests/compile_pip_requirements,tests/compile_pip_requirements_test_from_external_workspace,tests/ignore_root_user_error,tests/pip_repository_entry_points
 
 test --test_output=errors
 

BZLMOD_SUPPORT.md

@@ -31,8 +31,8 @@ A second example, in [examples/bzlmod_build_file_generation](examples/bzlmod_bui
 
 This rule set does not have full feature partity with the older `WORKSPACE` type configuration:
 
-1. Multiple pip extensions are not yet supported, as demonstrated in [this](examples/multi_python_versions) example.
-2. Gazelle does not support finding deps in sub-modules.  For instance we can have a dep like ` "@our_other_module//other_module/pkg:lib",` in a `py_test` definition.
+1. Gazelle does not support finding deps in sub-modules.  For instance we can have a dep like ` "@our_other_module//other_module/pkg:lib",` in a `py_test` definition.
+2. We have some features that are still not fully flushed out, and the user interface may change.
 
 Check ["issues"](/bazelbuild/rules_python/issues) for an up to date list.
 

README.md

@@ -96,8 +96,7 @@ use_repo(interpreter, "interpreter")
 
 pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
 pip.parse(
-    name = "pip",
-    incompatible_generate_aliases = True,
+    hub_name = "pip",
     python_interpreter_target = "@interpreter//:python",
     requirements_lock = "//:requirements_lock.txt",
     requirements_windows = "//:requirements_windows.txt",
@@ -201,7 +200,7 @@ central external repo and individual wheel external repos.
 
 ```python
 pip.parse(
-    name = "my_deps",
+    hub_name = "my_deps",
     requirements_lock = "//:requirements_lock.txt",
 )
 

docs/pip_repository.md

@@ -8,17 +8,28 @@
 load("@bazel_skylib//rules:build_test.bzl", "build_test")
 load("@pip//:requirements.bzl", "all_requirements", "all_whl_requirements", "requirement")
 load("@python_3_9//:defs.bzl", py_test_with_transition = "py_test")
+load("@python_aliases//3.10:defs.bzl", compile_pip_requirements_3_10 = "compile_pip_requirements")
+load("@python_aliases//3.9:defs.bzl", compile_pip_requirements_3_9 = "compile_pip_requirements")
 load("@rules_python//python:defs.bzl", "py_binary", "py_library", "py_test")
-load("@rules_python//python:pip.bzl", "compile_pip_requirements")
 
 # This stanza calls a rule that generates targets for managing pip dependencies
 # with pip-compile.
-compile_pip_requirements(
-    name = "requirements",
+compile_pip_requirements_3_9(
+    name = "requirements_3_9",
     extra_args = ["--allow-unsafe"],
     requirements_in = "requirements.in",
-    requirements_txt = "requirements_lock.txt",
-    requirements_windows = "requirements_windows.txt",
+    requirements_txt = "requirements_lock_3_9.txt",
+    requirements_windows = "requirements_windows_3_9.txt",
+)
+
+# This stanza calls a rule that generates targets for managing pip dependencies
+# with pip-compile.
+compile_pip_requirements_3_10(
+    name = "requirements_3_10",
+    extra_args = ["--allow-unsafe"],
+    requirements_in = "requirements.in",
+    requirements_txt = "requirements_lock_3_10.txt",
+    requirements_windows = "requirements_windows_3_10.txt",
 )
 
 # The rules below are language specific rules defined in

examples/bzlmod/BUILD.bazel

@@ -11,16 +11,6 @@ local_path_override(
     path = "../..",
 )
 
-# This name is generated by python.toolchain(), and is later passed
-# to use_repo() and interpreter.install().
-PYTHON_NAME_39 = "python_3_9"
-
-INTERPRETER_NAME_39 = "interpreter_39"
-
-PYTHON_NAME_310 = "python_3_10"
-
-INTERPRETER_NAME_310 = "interpreter_310"
-
 # We next initialize the python toolchain using the extension.
 # You can set different Python versions in this block.
 python = use_extension("@rules_python//python/extensions:python.bzl", "python")
@@ -41,38 +31,34 @@ python.toolchain(
     python_version = "3.10",
 )
 
-# You only need to load this repositories if you are using muiltple Python versions.
-# See the tests folder for various examples.
-use_repo(python, PYTHON_NAME_39, "python_aliases")
+# You only need to load this repositories if you are using multiple Python versions.
+# See the tests folder for various examples on using multiple Python versions.
+# The names "python_3_9" and "python_3_10" are autmatically created by the repo
+# rules based on the python_versions.
+use_repo(python, "python_3_10", "python_3_9", "python_aliases")
 
-# The interpreter extension discovers the platform specific Python binary.
-# It creates a symlink to the binary, and we pass the label to the following
-# pip.parse call.
-interpreter = use_extension("@rules_python//python/extensions:interpreter.bzl", "interpreter")
-interpreter.install(
-    name = INTERPRETER_NAME_39,
-    python_name = PYTHON_NAME_39,
-)
+pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
 
-# This second call is only needed if you are using mulitple different
-# Python versions/interpreters.
-interpreter.install(
-    name = INTERPRETER_NAME_310,
-    python_name = PYTHON_NAME_310,
+# For each pip setup we call pip.parse.  We can pass in various options
+# but typically we are passing in a requirements and an interpreter.
+# If you provide the python_version we will attempt to determine
+# the interpreter target automatically. Otherwise use python_interpreter_target
+# to override the lookup.
+pip.parse(
+    hub_name = "pip",
+    python_version = "3.9",
+    requirements_lock = "//:requirements_lock_3_9.txt",
+    requirements_windows = "//:requirements_windows_3_9.txt",
 )
-use_repo(interpreter, INTERPRETER_NAME_39, INTERPRETER_NAME_310)
-
-pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
 pip.parse(
-    name = "pip",
-    # Intentionally set it false because the "true" case is already
-    # covered by examples/bzlmod_build_file_generation
-    incompatible_generate_aliases = False,
-    python_interpreter_target = "@{}//:python".format(INTERPRETER_NAME_39),
-    requirements_lock = "//:requirements_lock.txt",
-    requirements_windows = "//:requirements_windows.txt",
+    hub_name = "pip",
+    python_version = "3.10",
+    requirements_lock = "//:requirements_lock_3_10.txt",
+    requirements_windows = "//:requirements_windows_3_10.txt",
 )
-use_repo(pip, "pip")
+
+# we use the pip_39 repo because entry points are not yet supported.
+use_repo(pip, "pip", "pip_39")
 
 bazel_dep(name = "other_module", version = "", repo_name = "our_other_module")
 local_path_override(

examples/bzlmod/MODULE.bazel

@@ -1,4 +1,4 @@
-load("@pip//:requirements.bzl", "entry_point")
+load("@pip_39//:requirements.bzl", "entry_point")
 load("@rules_python//python:defs.bzl", "py_test")
 
 alias(

examples/bzlmod/entry_point/BUILD.bazel

@@ -0,0 +1,9 @@
+load("@pip//:requirements.bzl", "requirement")
+load("@rules_python//python:defs.bzl", "py_library")
+
+py_library(
+    name = "my_lib",
+    srcs = ["__init__.py"],
+    visibility = ["@//tests:__pkg__"],
+    deps = [requirement("websockets")],
+)

examples/bzlmod/libs/my_lib/BUILD.bazel

@@ -0,0 +1,22 @@
+# Copyright 2023 The Bazel Authors. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import websockets
+
+
+def websockets_is_for_python_version(sanitized_version_check):
+    # We are checking that the name of the repository folders
+    # match the expexted generated names. If we update the folder
+    # structure or naming we will need to modify this test
+    return f"pip_{sanitized_version_check}_websockets" in websockets.__file__

examples/bzlmod/libs/my_lib/__init__.py

@@ -1,3 +1,4 @@
+websockets
 requests~=2.25.1
 s3cmd~=2.1.0
 yamllint>=1.28.0