Merge branch 'main' into feat/improve-uv-mgmt

Author: aignas

CHANGELOG.md

@@ -62,16 +62,24 @@ Unreleased changes template.
   docs but usage of the extension may result in your setup breaking without any
   notice. What is more, the URLs and SHA256 values will be retrieved from the
   GitHub releases page metadata published by the `uv` project.
+* (pypi) Downgraded versions of packages: `pip` from `24.3.2` to `24.0.0` and
+  `packaging` from `24.2` to `24.0`.
 
 {#v0-0-0-fixed}
 ### Fixed
+* (rules) `python_zip_file` output with `--bootstrap_impl=script` works again
+  ([#2596](https://github.com/bazelbuild/rules_python/issues/2596)).
+* (docs) Using `python_version` attribute for specifying python versions introduced in `v1.1.0`
 * (gazelle) Providing multiple input requirements files to `gazelle_python_manifest` now works correctly.
 * (pypi) Handle trailing slashes in pip index URLs in environment variables,
   fixes [#2554](https://github.com/bazelbuild/rules_python/issues/2554).
 * (runfiles) Runfile manifest and repository mapping files are now interpreted
   as UTF-8 on all platforms.
 * (coverage) Coverage with `--bootstrap_impl=script` is fixed
   ([#2572](https://github.com/bazelbuild/rules_python/issues/2572)).
+* (pypi) Non deterministic behaviour in requirement file usage has been fixed
+  by reverting [#2514](https://github.com/bazelbuild/rules_python/pull/2514).
+  The related issue is [#908](https://github.com/bazelbuild/rules_python/issue/908).
 * (sphinxdocs) Do not crash when `tag_class` does not have a populated `doc` value.
   Fixes ([#2579](https://github.com/bazelbuild/rules_python/issues/2579)).
 

docs/_includes/py_console_script_binary.md

@@ -12,7 +12,8 @@ py_console_script_binary(
 )
 ```
 
-Or for more advanced setups you can also specify extra dependencies and the
+#### Specifying extra dependencies 
+You can also specify extra dependencies and the
 exact script name you want to call. It is useful for tools like `flake8`, `pylint`,
 `pytest`, which have plugin discovery methods and discover dependencies from the
 PyPI packages available in the `PYTHONPATH`.
@@ -34,17 +35,26 @@ py_console_script_binary(
 )
 ```
 
-A specific Python version can be forced by using the generated version-aware
-wrappers, e.g. to force Python 3.9:
+#### Using a specific Python version
+
+A specific Python version can be forced by passing the desired Python version, e.g. to force Python 3.9:
 ```starlark
-load("@python_versions//3.9:defs.bzl", "py_console_script_binary")
+load("@rules_python//python/entry_points:py_console_script_binary.bzl", "py_console_script_binary")
 
 py_console_script_binary(
     name = "yamllint",
     pkg = "@pip//yamllint",
+    python_version = "3.9"
 )
 ```
 
+#### Using a specific Python Version directly from a Toolchain
+:::{deprecated} 1.1.0
+The toolchain specific `py_binary` and `py_test` symbols are aliases to the regular rules. 
+i.e. Deprecated `load("@python_versions//3.11:defs.bzl", "py_binary")` and `load("@python_versions//3.11:defs.bzl", "py_test")`
+
+You should instead specify the desired python version with `python_version`; see above example.
+:::
 Alternatively, the [`py_console_script_binary.binary_rule`] arg can be passed
 the version-bound `py_binary` symbol, or any other `py_binary`-compatible rule
 of your choosing:
@@ -60,5 +70,4 @@ py_console_script_binary(
 ```
 
 [specification]: https://packaging.python.org/en/latest/specifications/entry-points/
-[`py_console_script_binary.binary_rule`]: #py_console_script_binary_binary_rule
-
+[`py_console_script_binary.binary_rule`]: #py_console_script_binary_binary_rule

docs/toolchains.md

@@ -116,9 +116,9 @@ python = use_extension("@rules_python//python/extensions:python.bzl", "python")
 python.toolchain(python_version = "3.12")
 
 # BUILD.bazel
-load("@python_versions//3.12:defs.bzl", "py_binary")
+load("@rules_python//python:py_binary.bzl", "py_binary")
 
-py_binary(...)
+py_binary(..., python_version="3.12")
 ```
 
 ### Pinning to a Python version
@@ -132,21 +132,59 @@ is most useful for two cases:
    typically in a mono-repo situation.
 
 To configure a submodule with the version-aware rules, request the particular
-version you need, then use the `@python_versions` repo to use the rules that
-force specific versions:
+version you need when defining the toolchain:
 
 ```starlark
+# MODULE.bazel
 python = use_extension("@rules_python//python/extensions:python.bzl", "python")
 
 python.toolchain(
     python_version = "3.11",
 )
-use_repo(python, "python_versions")
+use_repo(python)
+```
+
+Then use the `@rules_python` repo in your BUILD file to explicity pin the Python version when calling the rule:
+
+```starlark
+# BUILD.bazel
+load("@rules_python//python:py_binary.bzl", "py_binary")
+
+py_binary(..., python_version = "3.11")
+py_test(..., python_version = "3.11")
 ```
 
-Then use e.g. `load("@python_versions//3.11:defs.bzl", "py_binary")` to use
-the rules that force that particular version. Multiple versions can be specified
-and use within a single build.
+Multiple versions can be specified and used within a single build.
+
+```starlark
+# MODULE.bazel
+python = use_extension("@rules_python//python/extensions:python.bzl", "python")
+
+python.toolchain(
+    python_version = "3.11",
+    is_default = True,
+)
+
+python.toolchain(
+    python_version = "3.12",
+)
+
+# BUILD.bazel
+load("@rules_python//python:py_binary.bzl", "py_binary")
+load("@rules_python//python:py_test.bzl", "py_test")
+
+# Defaults to 3.11
+py_binary(...)
+py_test(...)
+
+# Explicitly use Python 3.11
+py_binary(..., python_version = "3.11")
+py_test(..., python_version = "3.11")
+
+# Explicitly use Python 3.12
+py_binary(..., python_version = "3.12")
+py_test(..., python_version = "3.12")
+```
 
 For more documentation, see the bzlmod examples under the {gh-path}`examples`
 folder.  Look for the examples that contain a `MODULE.bazel` file.
@@ -159,6 +197,16 @@ The `python.toolchain()` call makes its contents available under a repo named
 Remember to call `use_repo()` to make repos visible to your module:
 `use_repo(python, "python_3_11")`
 
+
+:::{deprecated} 1.1.0
+The toolchain specific `py_binary` and `py_test` symbols are aliases to the regular rules. 
+i.e. Deprecated `load("@python_versions//3.11:defs.bzl", "py_binary")` & `load("@python_versions//3.11:defs.bzl", "py_test")`
+
+Usages of them should be changed to load the regular rules directly; 
+i.e.  Use `load("@rules_python//python:py_binary.bzl", "py_binary")` & `load("@rules_python//python:py_test.bzl", "py_test")` and then specify the `python_version` when using the rules corresponding to the python version you defined in your toolchain. {ref}`Library modules with version constraints`
+:::
+
+
 #### Toolchain usage in other rules
 
 Python toolchains can be utilized in other bazel rules, such as `genrule()`, by
@@ -508,4 +556,4 @@ of available toolchains.
 Currently the following flags are used to influence toolchain selection:
 * {obj}`--@rules_python//python/config_settings:py_linux_libc` for selecting the Linux libc variant.
 * {obj}`--@rules_python//python/config_settings:py_freethreaded` for selecting
-  the freethreaded experimental Python builds available from `3.13.0` onwards.
+  the freethreaded experimental Python builds available from `3.13.0` onwards.

python/private/pypi/dependency_resolver/dependency_resolver.py

@@ -16,7 +16,6 @@
 
 import atexit
 import os
-import re
 import shutil
 import sys
 from pathlib import Path
@@ -118,6 +117,7 @@ def main(
     absolute_path_prefix = resolved_requirements_file[
         : -(len(requirements_file) - len(repository_prefix))
     ]
+
     # As srcs might contain references to generated files we want to
     # use the runfiles file first. Thus, we need to compute the relative path
     # from the execution root.
@@ -162,19 +162,12 @@ def main(
     argv.append(
         f"--output-file={requirements_file_relative if UPDATE else requirements_out}"
     )
-    src_files = [
+    argv.extend(
         (src_relative if Path(src_relative).exists() else resolved_src)
         for src_relative, resolved_src in zip(srcs_relative, resolved_srcs)
-    ]
-    argv.extend(src_files)
+    )
     argv.extend(extra_args)
 
-    # Replace in the output lock file
-    # the lines like: # via -r /absolute/path/to/<requirements_file>
-    # with: # via -r <requirements_file>
-    # For Windows, we should explicitly call .as_posix() to convert \\ -> /
-    absolute_src_prefixes = [Path(src).absolute().parent.as_posix() + "/" for src in src_files]
-
     if UPDATE:
         print("Updating " + requirements_file_relative)
 
@@ -192,14 +185,14 @@ def main(
             # and we should copy the updated requirements back to the source tree.
             if not absolute_output_file.samefile(requirements_file_tree):
                 atexit.register(
-                    lambda: shutil.copy(absolute_output_file, requirements_file_tree)
+                    lambda: shutil.copy(
+                        absolute_output_file, requirements_file_tree
+                    )
                 )
-        cli(argv, standalone_mode=False)
+        cli(argv, standalone_mode = False)
         requirements_file_relative_path = Path(requirements_file_relative)
         content = requirements_file_relative_path.read_text()
         content = content.replace(absolute_path_prefix, "")
-        for absolute_src_prefix in absolute_src_prefixes:
-            content = content.replace(absolute_src_prefix, "")
         requirements_file_relative_path.write_text(content)
     else:
         # cli will exit(0) on success
@@ -221,15 +214,6 @@ def main(
                 golden = open(_locate(bazel_runfiles, requirements_file)).readlines()
                 out = open(requirements_out).readlines()
                 out = [line.replace(absolute_path_prefix, "") for line in out]
-
-                def replace_via_minus_r(line):
-                    if "# via -r " in line:
-                        for absolute_src_prefix in absolute_src_prefixes:
-                            line = line.replace(absolute_src_prefix, "")
-                        return line
-                    return line
-
-                out = [replace_via_minus_r(line) for line in out]
                 if golden != out:
                     import difflib
 

python/private/pypi/deps.bzl

@@ -51,8 +51,8 @@ _RULE_DEPS = [
     ),
     (
         "pypi__packaging",
-        "https://files.pythonhosted.org/packages/88/ef/eb23f262cca3c0c4eb7ab1933c3b1f03d021f2c48f54763065b6f0e321be/packaging-24.2-py3-none-any.whl",
-        "09abb1bccd265c01f4a3aa3f7a7db064b36514d2cba19a2f694fe6150451a759",
+        "https://files.pythonhosted.org/packages/49/df/1fceb2f8900f8639e278b056416d49134fb8d84c5942ffaa01ad34782422/packaging-24.0-py3-none-any.whl",
+        "2ddfb553fdf02fb784c234c7ba6ccc288296ceabec964ad2eae3777778130bc5",
     ),
     (
         "pypi__pep517",
@@ -61,8 +61,8 @@ _RULE_DEPS = [
     ),
     (
         "pypi__pip",
-        "https://files.pythonhosted.org/packages/ef/7d/500c9ad20238fcfcb4cb9243eede163594d7020ce87bd9610c9e02771876/pip-24.3.1-py3-none-any.whl",
-        "3790624780082365f47549d032f3770eeb2b1e8bd1f7b2e02dace1afa361b4ed",
+        "https://files.pythonhosted.org/packages/8a/6a/19e9fe04fca059ccf770861c7d5721ab4c2aebc539889e97c7977528a53b/pip-24.0-py3-none-any.whl",
+        "ba0d021a166865d2265246961bec0152ff124de910c5cc39f1156ce3fa7c69dc",
     ),
     (
         "pypi__pip_tools",

python/private/zip_main_template.py

@@ -286,10 +286,10 @@ def main():
     # The bin/ directory may not exist if it is empty.
     os.makedirs(os.path.dirname(python_program), exist_ok=True)
     try:
-        os.symlink(_PYTHON_BINARY_ACTUAL, python_program)
+        os.symlink(symlink_to, python_program)
     except OSError as e:
         raise Exception(
-            f"Unable to create venv python interpreter symlink: {python_program} -> {PYTHON_BINARY_ACTUAL}"
+            f"Unable to create venv python interpreter symlink: {python_program} -> {symlink_to}"
         ) from e
 
     # Some older Python versions on macOS (namely Python 3.7) may unintentionally

tests/bootstrap_impls/BUILD.bazel

@@ -1,3 +1,5 @@
+load("@rules_shell//shell:sh_test.bzl", "sh_test")
+
 # Copyright 2023 The Bazel Authors. All rights reserved.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
@@ -11,10 +13,40 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-load("//tests/support:sh_py_run_test.bzl", "py_reconfig_test", "sh_py_run_test")
+load("//tests/support:sh_py_run_test.bzl", "py_reconfig_binary", "py_reconfig_test", "sh_py_run_test")
 load("//tests/support:support.bzl", "SUPPORTS_BOOTSTRAP_SCRIPT")
 load(":venv_relative_path_tests.bzl", "relative_path_test_suite")
 
+py_reconfig_binary(
+    name = "bootstrap_script_zipapp_bin",
+    srcs = ["bin.py"],
+    bootstrap_impl = "script",
+    # Force it to not be self-executable
+    build_python_zip = "no",
+    main = "bin.py",
+    target_compatible_with = SUPPORTS_BOOTSTRAP_SCRIPT,
+)
+
+filegroup(
+    name = "bootstrap_script_zipapp_zip",
+    testonly = 1,
+    srcs = [":bootstrap_script_zipapp_bin"],
+    output_group = "python_zip_file",
+)
+
+sh_test(
+    name = "bootstrap_script_zipapp_test",
+    srcs = ["bootstrap_script_zipapp_test.sh"],
+    data = [":bootstrap_script_zipapp_zip"],
+    env = {
+        "ZIP_RLOCATION": "$(rlocationpaths :bootstrap_script_zipapp_zip)".format(),
+    },
+    target_compatible_with = SUPPORTS_BOOTSTRAP_SCRIPT,
+    deps = [
+        "@bazel_tools//tools/bash/runfiles",
+    ],
+)
+
 sh_py_run_test(
     name = "run_binary_zip_no_test",
     build_python_zip = "no",

tests/bootstrap_impls/bootstrap_script_zipapp_test.sh

@@ -0,0 +1,47 @@
+# Copyright 2024 The Bazel Authors. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# --- begin runfiles.bash initialization v3 ---
+# Copy-pasted from the Bazel Bash runfiles library v3.
+set -uo pipefail; set +e; f=bazel_tools/tools/bash/runfiles/runfiles.bash
+source "${RUNFILES_DIR:-/dev/null}/$f" 2>/dev/null || \
+  source "$(grep -sm1 "^$f " "${RUNFILES_MANIFEST_FILE:-/dev/null}" | cut -f2- -d' ')" 2>/dev/null || \
+  source "$0.runfiles/$f" 2>/dev/null || \
+  source "$(grep -sm1 "^$f " "$0.runfiles_manifest" | cut -f2- -d' ')" 2>/dev/null || \
+  source "$(grep -sm1 "^$f " "$0.exe.runfiles_manifest" | cut -f2- -d' ')" 2>/dev/null || \
+  { echo>&2 "ERROR: cannot find $f"; exit 1; }; f=; set -e
+# --- end runfiles.bash initialization v3 ---
+set +e
+
+bin=$(rlocation $ZIP_RLOCATION)
+if [[ -z "$bin" ]]; then
+  echo "Unable to locate test binary: $ZIP_RLOCATION"
+  exit 1
+fi
+set -x
+actual=$(python3 $bin)
+
+# How we detect if a zip file was executed from depends on which bootstrap
+# is used.
+# bootstrap_impl=script outputs RULES_PYTHON_ZIP_DIR=<somepath>
+# bootstrap_impl=system_python outputs file:.*Bazel.runfiles
+expected_pattern="Hello"
+if ! (echo "$actual" | grep "$expected_pattern" ) >/dev/null; then
+  echo "Test case failed: $1"
+  echo "expected output to match: $expected_pattern"
+  echo "but got:\n$actual"
+  exit 1
+fi
+
+exit 0