refactor!(toolchain): remove uname dep in the repository_rule stage (#2406)

Before this PR we would shell out to `uname` on UNIX systems to get the
`arch`
of the toolchain - on Windows we would not need to do it because there
used to
be only a single Windows platform. With this change we can correctly
support
the resolution of the python interpreter on various platforms and I have
also
added an env variable to customize the selection, so that users can use
`musl`
or a `freethreaded` interpreter if they wish.

As part of this change, I have restricted visibility of the config
settings
used in the toolchain alias repo so that we are creating fewer targets.
This is
a very good time to do this before `1.0.0`.

Fixes #2145
Work towards #2276
Work towards #2386
Work towards #1211 to unblock #2402
Work towards #1361

---------

Co-authored-by: Richard Levasseur <[email protected]>

Author: aignas

CHANGELOG.md

@@ -52,16 +52,39 @@ Unreleased changes template.
 
 {#v0-0-0-changed}
 ### Changed
+
+**Breaking**:
+* (toolchains) stop exposing config settings in python toolchain alias repos.
+  Please consider depending on the flags defined in
+  `//python/config_setting/...` and the `@platforms` package instead.
+* (toolchains) consumers who were depending on the `MACOS_NAME` and the `arch`
+  attribute in the `PLATFORMS` list, please update your code to respect the new
+  values. The values now correspond to the values available in the
+  `@platforms//` package constraint values.
+* (toolchains) `host_platform` and `interpreter` constants are no longer created
+  in the `toolchain` generated alias `.bzl` files. If you need to access the
+  host interpreter during the `repository_rule` evaluation, please use the
+  `@python_{version}_host//:python` targets created by
+  {bzl:obj}`python_register_toolchains` and
+  {bzl:obj}`python_register_multi_toolchains` macros or the {bzl:obj}`python`
+  bzlmod extension.
+
+Other changes:
 * (python_repository) Start honoring the `strip_prefix` field for `zstd` archives.
 
 {#v0-0-0-fixed}
 ### Fixed
-* Nothing fixed.
+* (toolchains) stop depending on `uname` to get the value of the host platform.
 
 {#v0-0-0-added}
 ### Added
 * (gazelle): Parser failures will now be logged to the terminal. Additional
   details can be logged by setting `GAZELLE_VERBOSE=1`.
+* (toolchains) allow users to select which variant of the support host toolchain
+  they would like to use through
+  `RULES_PYTHON_REPO_TOOLCHAIN_{VERSION}_{OS}_{ARCH}` env variable setting. For
+  example, this allows one to use `freethreaded` python interpreter in the
+  `repository_rule` to build a wheel from `sdist`.
 
 {#v0-0-0-removed}
 ### Removed

WORKSPACE

@@ -86,7 +86,7 @@ load("@rules_python_gazelle_plugin//:deps.bzl", _py_gazelle_deps = "gazelle_deps
 _py_gazelle_deps()
 
 # This interpreter is used for various rules_python dev-time tools
-load("@python//3.11.9:defs.bzl", "interpreter")
+interpreter = "@python_3_11_9_host//:python"
 
 #####################
 # Install twine for our own runfiles wheel publishing.

docs/environment-variables.md

@@ -15,6 +15,16 @@ Determines the verbosity of logging output for repo rules. Valid values:
 * `TRACE`
 :::
 
+:::{envvar} RULES_PYTHON_REPO_TOOLCHAIN_VERSION_OS_ARCH
+
+Determines the python interpreter platform to be used for a particular
+interpreter `(version, os, arch)` triple to be used in repository rules.
+Replace the `VERSION_OS_ARCH` part with actual values when using, e.g.
+`3_13_0_linux_x86_64`. The version values must have `_` instead of `.` and the
+os, arch values are the same as the ones mentioned in the
+`//python:versions.bzl` file.
+:::
+
 :::{envvar} RULES_PYTHON_PIP_ISOLATED
 
 Determines if `--isolated` is used with pip.

examples/bzlmod/MODULE.bazel.lock

@@ -160,7 +160,11 @@ def python_register_toolchains(
                 platform = platform,
             ))
 
-    host_toolchain(name = name + "_host")
+    host_toolchain(
+        name = name + "_host",
+        platforms = loaded_platforms,
+        python_version = python_version,
+    )
 
     toolchain_aliases(
         name = name,

python/private/python_register_toolchains.bzl

@@ -41,6 +41,10 @@ def _logger(mrctx, name = None):
 
     Returns:
         A struct with attributes logging: trace, debug, info, warn, fail.
+        Please use `return logger.fail` when using the `fail` method, because
+        it makes `buildifier` happy and ensures that other implementation of
+        the logger injected into the function work as expected by terminating
+        on the given line.
     """
     if _is_repo_debug_enabled(mrctx):
         verbosity_level = "DEBUG"
@@ -140,7 +144,7 @@ def _execute_internal(
     result = mrctx.execute(arguments, environment = environment, **kwargs)
 
     if fail_on_error and result.return_code != 0:
-        logger.fail((
+        return logger.fail((
             "repo.execute: {op}: end: failure:\n" +
             "  command: {cmd}\n" +
             "  return code: {return_code}\n" +

python/private/repo_utils.bzl

@@ -0,0 +1,74 @@
+# Copyright 2024 The Bazel Authors. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Create toolchain alias targets."""
+
+load("@rules_python//python:versions.bzl", "PLATFORMS")
+
+def toolchain_aliases(*, name, platforms, visibility = None, native = native):
+    """Create toolchain aliases for the python toolchains.
+
+    Args:
+        name: {type}`str` The name of the current repository.
+        platforms: {type}`platforms` The list of platforms that are supported
+            for the current toolchain repository.
+        visibility: {type}`list[Target] | None` The visibility of the aliases.
+        native: The native struct used in the macro, useful for testing.
+    """
+    for platform in PLATFORMS.keys():
+        if platform not in platforms:
+            continue
+
+        native.config_setting(
+            name = platform,
+            flag_values = PLATFORMS[platform].flag_values,
+            constraint_values = PLATFORMS[platform].compatible_with,
+            visibility = ["//visibility:private"],
+        )
+
+    prefix = name
+    for name in [
+        "files",
+        "includes",
+        "libpython",
+        "py3_runtime",
+        "python_headers",
+        "python_runtimes",
+    ]:
+        native.alias(
+            name = name,
+            actual = select({
+                ":" + platform: "@{}_{}//:{}".format(prefix, platform, name)
+                for platform in platforms
+            }),
+            visibility = visibility,
+        )
+
+    native.alias(
+        name = "python3",
+        actual = select({
+            ":" + platform: "@{}_{}//:{}".format(prefix, platform, "python.exe" if "windows" in platform else "bin/python3")
+            for platform in platforms
+        }),
+        visibility = visibility,
+    )
+    native.alias(
+        name = "pip",
+        actual = select({
+            ":" + platform: "@{}_{}//:python_runtimes".format(prefix, platform)
+            for platform in platforms
+            if "windows" not in platform
+        }),
+        visibility = visibility,
+    )