Merge remote-tracking branch 'upstream/main' into HEAD

Author: philsc

.github/workflows/mypy.yaml

@@ -15,7 +15,7 @@ defaults:
 
 jobs:
   ci:
-    runs-on: ubuntu-20.04
+    runs-on: ubuntu-latest
     steps:
       # Checkout the code
       - uses: actions/checkout@v4

CHANGELOG.md

@@ -21,7 +21,7 @@ A brief description of the categories of changes:
   `(docs)`.
 
 <!--
-Unreleased changes template.
+BEGIN_UNRELEASED_TEMPLATE
 
 {#v0-0-0}
 ## Unreleased
@@ -43,8 +43,9 @@ Unreleased changes template.
 {#v0-0-0-removed}
 ### Removed
 * Nothing removed.
--->
 
+END_UNRELEASED_TEMPLATE
+-->
 
 {#v0-0-0}
 ## Unreleased
@@ -53,6 +54,29 @@ Unreleased changes template.
 
 {#v0-0-0-changed}
 ### Changed
+* Nothing changed.
+
+{#v0-0-0-fixed}
+### Fixed
+* (rules) PyInfo provider is now advertised by py_test, py_binary, and py_library;
+  this allows aspects using required_providers to function correctly.
+  ([#2506](https://github.com/bazel-contrib/rules_python/issues/2506)).
+
+{#v0-0-0-added}
+### Added
+* Nothing added.
+
+{#v0-0-0-removed}
+### Removed
+* Nothing removed.
+
+{#1-4-0}
+## [1.4.0] - 2025-04-19
+
+[1.4.0]: https://github.com/bazel-contrib/rules_python/releases/tag/1.4.0
+
+{#1-4-0-changed}
+### Changed
 * (toolchain) The `exec` configuration toolchain now has the forwarded
   `exec_interpreter` now also forwards the `ToolchainInfo` provider. This is
   for increased compatibility with the `RBE` setups where access to the `exec`
@@ -72,15 +96,29 @@ Unreleased changes template.
 * (toolchains) Previously [#2636](https://github.com/bazel-contrib/rules_python/pull/2636)
   changed the semantics of `ignore_root_user_error` from "ignore" to "warning". This is now
   flipped back to ignoring the issue, and will only emit a warning when the attribute is set
-  `False`.  
+  `False`.
 * (pypi) The PyPI extension will no longer write the lock file entries as the
   extension has been marked reproducible.
   Fixes [#2434](https://github.com/bazel-contrib/rules_python/issues/2434).
+* (gazelle) Lazily load and parse manifest files when running Gazelle. This ensures no
+  manifest files are loaded when Gazelle is run over a set of non-python directories
+  [PR #2746](https://github.com/bazel-contrib/rules_python/pull/2746).
+* (rules) {attr}`py_binary.srcs` and {attr}`py_test.srcs` is no longer mandatory when
+  `main_module` is specified (for `--bootstrap_impl=script`)
+* (pypi) From now on the `Requires-Dist` from the wheel metadata is analysed in
+  the loading phase instead of repository rule phase giving better caching
+  performance when the target platforms are changed (e.g. target python
+  versions). This is preparatory work for stabilizing the cross-platform wheel
+  support. From now on the usage of `experimental_target_platforms` should be
+  avoided and the `requirements_by_platform` values should be instead used to
+  specify the target platforms for the given dependencies.
 
 [20250317]: https://github.com/astral-sh/python-build-standalone/releases/tag/20250317
 
-{#v0-0-0-fixed}
+{#1-4-0-fixed}
 ### Fixed
+* (pypi) Platform specific extras are now correctly handled when using
+  universal lock files with environment markers. Fixes [#2690](https://github.com/bazel-contrib/rules_python/pull/2690).
 * (runfiles) ({obj}`--bootstrap_impl=script`) Follow symlinks when searching for runfiles.
 * (toolchains) Do not try to run `chmod` when downloading non-windows hermetic toolchain
   repositories on Windows. Fixes
@@ -91,8 +129,12 @@ Unreleased changes template.
   Fixes [#2685](https://github.com/bazel-contrib/rules_python/issues/2685).
 * (toolchains) Run the check on the Python interpreter in isolated mode, to ensure it's not affected by userland environment variables, such as `PYTHONPATH`.
 * (toolchains) Ensure temporary `.pyc` and `.pyo` files are also excluded from the interpreters repository files.
+* (pypi) Run interpreter version call in isolated mode, to ensure it's not affected by userland environment variables, such as `PYTHONPATH`.
+* (packaging) An empty `requires_file` is treated as if it were omitted, resulting in a valid `METADATA` file.
+* (rules) py_wheel and sphinxdocs rules now propagate `target_compatible_with` to all targets they create.
+  [PR #2788](https://github.com/bazel-contrib/rules_python/pull/2788).
 
-{#v0-0-0-added}
+{#1-4-0-added}
 ### Added
 * (pypi) From now on `sha256` values in the `requirements.txt` is no longer
   mandatory when enabling {attr}`pip.parse.experimental_index_url` feature.
@@ -116,14 +158,20 @@ Unreleased changes template.
   allow specifying links to create within the venv site packages (only
   applicable with {obj}`--bootstrap_impl=script`)
   ([#2156](https://github.com/bazelbuild/rules_python/issues/2156)).
+* (toolchains) Local Python installs can be used to create a toolchain
+  equivalent to the standard toolchains. See [Local toolchains] docs for how to
+  configure them.
+* (toolchains) Expose `$(PYTHON2_ROOTPATH)` and `$(PYTHON3_ROOTPATH)` which are runfiles
+  locations equivalents of `$(PYTHON2)` and `$(PYTHON3) respectively.
 
-{#v0-0-0-removed}
+
+{#1-4-0-removed}
 ### Removed
 * Nothing removed.
 
 
 {#v1-3-0}
-## Unreleased
+## [1.3.0] - 2025-03-27
 
 [1.3.0]: https://github.com/bazel-contrib/rules_python/releases/tag/1.3.0
 

RELEASING.md

@@ -14,7 +14,14 @@ These are the steps for a regularly scheduled release from HEAD.
 
 1. [Determine the next semantic version number](#determining-semantic-version).
 1. Update CHANGELOG.md: replace the `v0-0-0` and `0.0.0` with `X.Y.0`.
+   ```
+   awk -v version=X.Y.0 'BEGIN { hv=version; gsub(/\./, "-", hv) } /END_UNRELEASED_TEMPLATE/ { found_marker = 1 } found_marker { gsub(/v0-0-0/, hv, $0); gsub(/Unreleased/, "[" version "] - " strftime("%Y-%m-%d"), $0); gsub(/0.0.0/, version, $0); } { print } ' CHANGELOG.md > /tmp/changelog && cp /tmp/changelog CHANGELOG.md
+   ```
 1. Replace `VERSION_NEXT_*` strings with `X.Y.0`.
+   ```
+   grep -l --exclude=CONTRIBUTING.md --exclude=RELEASING.md --exclude-dir=.* VERSION_NEXT_ -r \
+     | xargs sed -i -e 's/VERSION_NEXT_FEATURE/X.Y.0/' -e 's/VERSION_NEXT_PATCH/X.Y.0/'
+   ```
 1. Send these changes for review and get them merged.
 1. Create a branch for the new release, named `release/X.Y`
    ```
@@ -90,6 +97,20 @@ It will be promoted to stable next week, pending feedback.
 It's traditional to include notable changes from the changelog, but not
 required.
 
+### Re-releasing a version
+
+Re-releasing a version (i.e. changing the commit a tag points to)  is
+*sometimes* possible, but it depends on how far into the release process it got.
+
+The two points of no return are:
+ * If the PyPI package has been published: PyPI disallows using the same
+   filename/version twice. Once published, it cannot be replaced.
+ * If the BCR package has been published: Once it's been committed to the BCR
+   registry, it cannot be replaced.
+
+If release steps fail _prior_ to those steps, then its OK to change the tag. You
+may need to manually delete the GitHub release.
+
 ## Secrets
 
 ### PyPI user rules-python

config.bzl.tmpl.bzlmod

@@ -108,6 +108,7 @@ sphinx_stardocs(
         "//python/cc:py_cc_toolchain_bzl",
         "//python/cc:py_cc_toolchain_info_bzl",
         "//python/entry_points:py_console_script_binary_bzl",
+        "//python/local_toolchains:repos_bzl",
         "//python/private:attr_builders_bzl",
         "//python/private:builders_util_bzl",
         "//python/private:py_binary_rule_bzl",

docs/BUILD.bazel

@@ -351,13 +351,13 @@ sphinxcontrib-serializinghtml==2.0.0 \
     --hash=sha256:6e2cb0eef194e10c27ec0023bfeb25badbbb5868244cf5bc5bdc04e4464bf331 \
     --hash=sha256:e9d912827f872c029017a53f0ef2180b327c3f7fd23c87229f7a8e8b70031d4d
     # via sphinx
-typing-extensions==4.12.2 \
-    --hash=sha256:04e5ca0351e0f3f85c6853954072df659d0d13fac324d0072316b67d7794700d \
-    --hash=sha256:1a7ead55c7e559dd4dee8856e3a88b41225abfe1ce8df57b7c13915fe121ffb8
+typing-extensions==4.13.2 \
+    --hash=sha256:a439e7c04b49fec3e5d3e2beaa21755cadbbdc391694e28ccdd36ca4a1408f8c \
+    --hash=sha256:e6c81219bd689f51865d9e372991c540bda33a0379d5573cddb9a3a23f7caaef
     # via
     #   rules-python-docs (docs/pyproject.toml)
     #   sphinx-autodoc2
-urllib3==2.3.0 \
-    --hash=sha256:1cee9ad369867bfdbbb48b7dd50374c0967a0bb7710050facf0dd6911440e3df \
-    --hash=sha256:f8c5449b3cf0861679ce7e0503c7b44b5ec981bec0d1d3795a07f1ba96f0204d
+urllib3==2.4.0 \
+    --hash=sha256:414bc6535b787febd7567804cc015fee39daab8ad86268f1310a9250697de466 \
+    --hash=sha256:4e16665048960a0900c702d4a66415956a584919c03361cac9f1df5c5dd7e813
     # via requests

docs/requirements.txt

@@ -31,11 +31,35 @@ minor/patch versions.
 See [Bazel's release support matrix](https://bazel.build/release#support-matrix)
 for what versions are the rolling, active, and prior releases.
 
+## Supported Python versions
+
+As a general rule we test all released non-EOL Python versions. Different
+interpreter versions may work but are not guaranteed. We are interested in
+staying compatible with upcoming unreleased versions, so if you see that things
+stop working, please create tickets or, more preferably, pull requests.
+
 ## Supported Platforms
 
 We only support the platforms that our continuous integration jobs run, which
-is Linux, Mac, and Windows. Code to support other platforms is allowed, but
-can only be on a best-effort basis.
+is Linux, Mac, and Windows.
+
+In order to better describe different support levels, the below acts as a rough
+guideline for different platform tiers:
+* Tier 0 - The platforms that our CI runs: `linux_x86_64`, `osx_x86_64`, `RBE linux_x86_64`.
+* Tier 1 - The platforms that are similar enough to what the CI runs: `linux_aarch64`, `osx_arm64`.
+  What is more, `windows_x86_64` is in this list as we run tests in CI but
+  developing for Windows is more challenging and features may come later to
+  this platform.
+* Tier 2 - The rest of the platforms that may have varying level of support, e.g.
+  `linux_s390x`, `linux_ppc64le`, `windows_arm64`.
+
+:::{note}
+Code to support Tier 2 platforms is allowed, but regressions will be fixed on a
+best-effort basis, so feel free to contribute by creating PRs.
+
+If you would like to provide/sponsor CI setup for a platform that is not Tier 0,
+please create a ticket or contact the maintainers on Slack.
+:::
 
 ## Compatibility Policy
 

docs/support.md

@@ -199,10 +199,10 @@ Remember to call `use_repo()` to make repos visible to your module:
 
 
 :::{deprecated} 1.1.0
-The toolchain specific `py_binary` and `py_test` symbols are aliases to the regular rules. 
+The toolchain specific `py_binary` and `py_test` symbols are aliases to the regular rules.
 i.e. Deprecated `load("@python_versions//3.11:defs.bzl", "py_binary")` & `load("@python_versions//3.11:defs.bzl", "py_test")`
 
-Usages of them should be changed to load the regular rules directly; 
+Usages of them should be changed to load the regular rules directly;
 i.e.  Use `load("@rules_python//python:py_binary.bzl", "py_binary")` & `load("@rules_python//python:py_test.bzl", "py_test")` and then specify the `python_version` when using the rules corresponding to the python version you defined in your toolchain. {ref}`Library modules with version constraints`
 :::
 
@@ -215,7 +215,11 @@ attribute. You can obtain the path to the Python interpreter using the
 `$(PYTHON2)` and `$(PYTHON3)` ["Make"
 Variables](https://bazel.build/reference/be/make-variables). See the
 {gh-path}`test_current_py_toolchain <tests/load_from_macro/BUILD.bazel>` target
-for an example.
+for an example. We also make available `$(PYTHON2_ROOTPATH)` and `$(PYTHON3_ROOTPATH)`
+which are Make Variable equivalents of `$(PYTHON2)` and `$(PYTHON3)` but for runfiles
+locations. These will be helpful if you need to set env vars of binary/test rules
+while using [`--nolegacy_external_runfiles`](https://bazel.build/reference/command-line-reference#flag--legacy_external_runfiles).
+The original make variables still work in exec contexts such as genrules.
 
 ### Overriding toolchain defaults and adding more versions
 
@@ -327,7 +331,97 @@ After registration, your Python targets will use the toolchain's interpreter dur
 is still used to 'bootstrap' Python targets (see https://github.com/bazel-contrib/rules_python/issues/691).
 You may also find some quirks while using this toolchain. Please refer to [python-build-standalone documentation's _Quirks_ section](https://gregoryszorc.com/docs/python-build-standalone/main/quirks.html).
 
-## Autodetecting toolchain
+## Local toolchain
+
+It's possible to use a locally installed Python runtime instead of the regular
+prebuilt, remotely downloaded ones. A local toolchain contains the Python
+runtime metadata (Python version, headers, ABI flags, etc) that the regular
+remotely downloaded runtimes contain, which makes it possible to build e.g. C
+extensions (unlike the autodetecting and runtime environment toolchains).
+
+For simple cases, some rules are provided that will introspect
+a Python installation and create an appropriate Bazel definition from
+it. To do this, three pieces need to be wired together:
+
+1. Specify a path or command to a Python interpreter (multiple can be defined).
+2. Create toolchains for the runtimes in (1)
+3. Register the toolchains created by (2)
+
+The below is an example that will use `python3` from PATH to find the
+interpreter, then introspect its installation to generate a full toolchain.
+
+```starlark
+# File: MODULE.bazel
+
+local_runtime_repo = use_repo_rule(
+    "@rules_python//python/local_toolchains:repos.bzl",
+    "local_runtime_repo",
+    dev_dependency = True,
+)
+
+local_runtime_toolchains_repo = use_repo_rule(
+    "@rules_python//python/local_toolchains:repos.bzl"
+    "local_runtime_toolchains_repo"
+    dev_dependency = True,
+)
+
+# Step 1: Define the Python runtime
+local_runtime_repo(
+    name = "local_python3",
+    interpreter_path = "python3",
+    on_failure = "fail",
+)
+
+# Step 2: Create toolchains for the runtimes
+local_runtime_toolchains_repo(
+    name = "local_toolchains",
+    runtimes = ["local_python3"],
+)
+
+# Step 3: Register the toolchains
+register_toolchains("@local_toolchains//:all", dev_dependency = True)
+```
+
+Note that `register_toolchains` will insert the local toolchain earlier in the
+toolchain ordering, so it will take precedence over other registered toolchains.
+
+:::{important}
+Be sure to set `dev_dependency = True`. Using a local toolchain only makes sense
+for the root module.
+
+If an intermediate module does it, then the `register_toolchains()` call will
+take precedence over the default rules_python toolchains and cause problems for
+downstream modules.
+:::
+
+Multiple runtimes and/or toolchains can be defined, which allows for multiple
+Python versions and/or platforms to be configured in a single `MODULE.bazel`.
+
+## Runtime environment toolchain
+
+The runtime environment toolchain is a minimal toolchain that doesn't provide
+information about Python at build time. In particular, this means it is not able
+to build C extensions -- doing so requires knowing, at build time, what Python
+headers to use.
+
+In effect, all it does is generate a small wrapper script that simply calls e.g.
+`/usr/bin/env python3` to run a program. This makes it easy to change what
+Python is used to run a program, but also makes it easy to use a Python version
+that isn't compatible with build-time assumptions.
+
+```
+register_toolchains("@rules_python//python/runtime_env_toolchains:all")
+```
+
+Note that this toolchain has no constraints, i.e. it will match any platform,
+Python version, etc.
+
+:::{seealso}
+[Local toolchain], which creates a more full featured toolchain from a
+locally installed Python.
+:::
+
+### Autodetecting toolchain
 
 The autodetecting toolchain is a deprecated toolchain that is built into Bazel.
 It's name is a bit misleading: it doesn't autodetect anything. All it does is
@@ -345,7 +439,6 @@ To aid migration off the Bazel-builtin toolchain, rules_python provides
 {bzl:obj}`@rules_python//python/runtime_env_toolchains:all`. This is an equivalent
 toolchain, but is implemented using rules_python's objects.
 
-
 ## Custom toolchains
 
 While rules_python provides toolchains by default, it is not required to use

docs/toolchains.md

@@ -294,6 +294,12 @@ starlark  # Example comment
 """.splitlines(),
 )
 
+write_file(
+    name = "empty_requires_file",
+    out = "empty_requires.txt",
+    content = [""],
+)
+
 write_file(
     name = "extra_requires_file",
     out = "extra_requires.txt",
@@ -324,6 +330,15 @@ py_wheel(
     deps = [":example_pkg"],
 )
 
+py_wheel(
+    name = "empty_requires_files",
+    distribution = "empty_requires_files",
+    python_tag = "py3",
+    requires_file = ":empty_requires.txt",
+    version = "0.0.1",
+    deps = [":example_pkg"],
+)
+
 # Package just a specific py_libraries, without their dependencies
 py_wheel(
     name = "minimal_data_files",
@@ -367,6 +382,7 @@ py_test(
         ":custom_package_root_multi_prefix",
         ":custom_package_root_multi_prefix_reverse_order",
         ":customized",
+        ":empty_requires_files",
         ":extra_requires",
         ":filename_escaping",
         ":minimal_data_files",