feat(toolchains): ABI3 Python headers target (#3274)

Until now, we silently link extensions with both stable and unstable ABI
libs,
with the latter taking precedence in symbol resolution, because it
appears first
in the linker command AND, crucially, contains all CPython symbols
present in
the stable ABI library, thus overriding them. This has the effect that
stable
ABI extensions on Windows are usable only with the Python distribution
that they
were built on.

To fix, a separate ABI3 header target is introduced, and should be used
for C++
extensions on Windows if stable ABI builds are requested.

Idea as formulated by `@dgrunwald-qt` in
nicholasjng/nanobind-bazel#72 (comment).

This is motivated by
nicholasjng/nanobind-bazel#72.

This change shifts stable ABI selection on Windows to the extension
developer, where
it has arguably always been (they had to set the `Py_LIMITED_API`
macro).

An upside of this approach is that with a separate target, the question
"stable ABI
or not" can be decided on an extension-by-extension basis, giving
maximum flexibility
to developers. This should not influence the wheel platform target,
because a wheel
is marked ABI3 if and only if all of its extensions are marked as ABI3.

---------

Co-authored-by: Richard Levasseur <[email protected]>
Co-authored-by: Richard Levasseur <[email protected]>

Author: nicholasjng

AGENTS.md

@@ -19,6 +19,14 @@ using a grandoise title.
 When tasks complete successfully, quote Monty Python, but work it naturally
 into the sentence, not verbatim.
 
+When adding `{versionadded}` or `{versionchanged}` sections, add them add the
+end of the documentation text.
+
+### Starlark style
+
+For doc strings, using triple quoted strings when the doc string is more than
+three lines. Do not use a trailing backslack (`\`) for the opening triple-quote.
+
 ### bzl_library targets for bzl source files
 
 * A `bzl_library` target should be defined for every `.bzl` file outside
@@ -78,7 +86,17 @@ When modifying documentation
  * Act as an expert in tech writing, Sphinx, MyST, and markdown.
  * Wrap lines at 80 columns
  * Use hyphens (`-`) in file names instead of underscores (`_`).
-
+ * In Sphinx MyST markup, outer directives must have more colons than inner
+   directives. For example:
+   ```
+   ::::{outerdirective}
+   outer text
+
+   :::{innertdirective}
+   inner text
+   :::
+   ::::
+   ```
 
 Generated API references can be found by:
 * Running `bazel build //docs:docs` and inspecting the generated files

CHANGELOG.md

@@ -96,6 +96,15 @@ END_UNRELEASED_TEMPLATE
   `WORKSPACE` files. See the
   {ref}`common-deps-with-multiple-pypi-versions` guide on using common
   dependencies with multiple PyPI versions` for an example.
+* (toolchains) Stable ABI headers support added. To use, depend on
+  {obj}`//python/cc:current_py_cc_headers_abi3`. This allows Windows builds
+  a way to depend on headers without the potentially Python unstable ABI
+  objects from the regular {obj}`//python/cc:current_py_cc_headers` target
+  being included.
+  * Adds {obj}`//python/cc:current_py_cc_headers_abi3`,
+    {obj}`py_cc_toolchain.headers_abi3`, and {obj}`PyCcToolchainInfo.headers_abi3`.
+  * {obj}`//python:features.bzl%features.headers_abi3` can be used to
+    feature-detect the presense of the above.
 
 {#v1-6-2}
 ## [1.6.2] - 2025-09-21

docs/api/rules_python/python/cc/index.md

@@ -4,7 +4,7 @@
 :::
 # //python/cc
 
-:::{bzl:target} current_py_cc_headers
+::::{bzl:target} current_py_cc_headers
 
 A convenience target that provides the Python headers. It uses toolchain
 resolution to find the headers for the Python runtime matching the interpreter
@@ -14,7 +14,32 @@ that will be used. This basically forwards the underlying
 This target provides:
 
 * `CcInfo`: The C++ information about the Python headers.
+
+:::{seealso}
+
+The {obj}`:current_py_cc_headers_abi3` target for explicitly using the
+stable ABI.
+:::
+
+::::
+
+::::{bzl:target} current_py_cc_headers_abi3
+
+A convenience target that provides the Python ABI3 headers (stable ABI headers).
+It uses toolchain resolution to find the headers for the Python runtime matching
+the interpreter that will be used. This basically forwards the underlying
+`cc_library(name="python_headers_abi3")` target defined in the `@python_X_Y`
+repo.
+
+This target provides:
+
+* `CcInfo`: The C++ information about the Python ABI3 headers.
+
+:::{versionadded} VERSION_NEXT_FEATURE
+The {obj}`features.headers_abi3` attribute can be used to detect if this target
+is available or not.
 :::
+::::
 
 :::{bzl:target} current_py_cc_libs
 

docs/howto/python-headers.md

@@ -8,9 +8,10 @@ files. This guide shows how to get the necessary include paths from the Python
 [toolchain](toolchains).
 
 The recommended way to get the headers is to depend on the
-`@rules_python//python/cc:current_py_cc_headers` target. This is a helper
-target that uses toolchain resolution to find the correct headers for the
-target platform.
+{obj}`@rules_python//python/cc:current_py_cc_headers` or
+{obj}`@rules_python//python/cc:current_py_cc_headers_abi3`
+targets. These are convenience targets that use toolchain resolution to find
+the correct headers for the target platform.
 
 ## Using the headers
 
@@ -27,4 +28,27 @@ cc_library(
 ```
 
 This setup ensures that your C extension code can find and use the Python
-headers during compilation.
+headers during compilation.
+
+:::{note}
+The `:current_py_cc_headers` target provides all the Python headers. This _may_
+include ABI-specific information.
+:::
+
+## Using the stable ABI headers
+
+If you're building for the [Python stable ABI](https://docs.python.org/3/c-api/stable.html),
+then depend on {obj}`@rules_python//python/cc:current_py_cc_headers_abi3`. This
+target contains only objects relevant to the Python stable ABI. Remember to
+define
+[`Py_LIMITED_API`](https://docs.python.org/3/c-api/stable.html#c.Py_LIMITED_API)
+when building such extensions.
+
+```bazel
+# BUILD.bazel
+cc_library(
+    name = "my_stable_abi_extension",
+    srcs = ["my_stable_abi_extension.c"],
+    deps = ["@rules_python//python/cc:current_py_cc_headers_abi3"],
+)
+```

docs/pyproject.toml

@@ -12,5 +12,6 @@ dependencies = [
     "readthedocs-sphinx-ext",
     "absl-py",
     "typing-extensions",
-    "sphinx-reredirects"
+    "sphinx-reredirects",
+    "pefile"
 ]

docs/requirements.txt

@@ -111,9 +111,9 @@ colorama==0.4.6 ; sys_platform == 'win32' \
     --hash=sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44 \
     --hash=sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6
     # via sphinx
-docutils==0.22 \
-    --hash=sha256:4ed966a0e96a0477d852f7af31bdcb3adc049fbb35ccba358c2ea8a03287615e \
-    --hash=sha256:ba9d57750e92331ebe7c08a1bbf7a7f8143b86c476acd51528b042216a6aad0f
+docutils==0.21.2 \
+    --hash=sha256:3a6b18732edf182daa3cd12775bbb338cf5691468f91eeeb109deff6ebfa986f \
+    --hash=sha256:dafca5b9e384f0e419294eb4d2ff9fa826435bf15f15b7bd45723e8ad76811b2
     # via
     #   myst-parser
     #   sphinx
@@ -232,6 +232,10 @@ packaging==25.0 \
     # via
     #   readthedocs-sphinx-ext
     #   sphinx
+pefile==2024.8.26 \
+    --hash=sha256:3ff6c5d8b43e8c37bb6e6dd5085658d658a7a0bdcd20b6a07b1fcfc1c4e9d632 \
+    --hash=sha256:76f8b485dcd3b1bb8166f1128d395fa3d87af26360c2358fb75b80019b957c6f
+    # via rules-python-docs (docs/pyproject.toml)
 pygments==2.19.2 \
     --hash=sha256:636cb2477cec7f8952536970bc533bc43743542f70392ae026374600add5b887 \
     --hash=sha256:86540386c03d588bb81d44bc3928634ff26449851e99741617ecb9037ee5ec0b

python/cc/BUILD.bazel

@@ -2,7 +2,7 @@
 
 load("@bazel_skylib//:bzl_library.bzl", "bzl_library")
 load("//python/private:bzlmod_enabled.bzl", "BZLMOD_ENABLED")
-load("//python/private:current_py_cc_headers.bzl", "current_py_cc_headers")
+load("//python/private:current_py_cc_headers.bzl", "current_py_cc_headers", "current_py_cc_headers_abi3")
 load("//python/private:current_py_cc_libs.bzl", "current_py_cc_libs")
 
 package(
@@ -20,6 +20,17 @@ current_py_cc_headers(
     visibility = ["//visibility:public"],
 )
 
+# This target provides the C ABI3 headers for whatever the current toolchain is
+# for the consuming rule. It basically acts like a cc_library by forwarding
+# on the providers for the underlying cc_library that the toolchain is using.
+current_py_cc_headers_abi3(
+    name = "current_py_cc_headers_abi3",
+    # Building this directly will fail unless a py cc toolchain is registered,
+    # and it's only under bzlmod that one is registered by default.
+    tags = [] if BZLMOD_ENABLED else ["manual"],
+    visibility = ["//visibility:public"],
+)
+
 # This target provides the C libraries for whatever the current toolchain is for
 # the consuming rule. It basically acts like a cc_library by forwarding on the
 # providers for the underlying cc_library that the toolchain is using.

python/features.bzl

@@ -22,6 +22,16 @@ _VERSION_PRIVATE = "$Format:%(describe:tags=true)$"
 def _features_typedef():
     """Information about features rules_python has implemented.
 
+    ::::{field} headers_abi3
+    :type: bool
+
+    True if the {obj}`@rules_python//python/cc:current_py_cc_headers_abi3`
+    target is available.
+
+    :::{versionadded} VERSION_NEXT_FEATURE
+    :::
+    ::::
+
     ::::{field} precompile
     :type: bool
 
@@ -60,6 +70,7 @@ def _features_typedef():
 features = struct(
     TYPEDEF = _features_typedef,
     # keep sorted
+    headers_abi3 = True,
     precompile = True,
     py_info_venv_symlinks = True,
     uses_builtin_rules = not config.enable_pystar,

python/private/BUILD.bazel

@@ -31,6 +31,7 @@ filegroup(
     name = "distribution",
     srcs = glob(["**"]) + [
         "//python/private/api:distribution",
+        "//python/private/cc:distribution",
         "//python/private/pypi:distribution",
         "//python/private/whl_filegroup:distribution",
         "//tools/build_defs/python/private:distribution",
@@ -360,6 +361,7 @@ bzl_library(
         ":common_labels.bzl",
         ":py_cc_toolchain_info_bzl",
         ":rules_cc_srcs_bzl",
+        ":sentinel_bzl",
         ":util_bzl",
         "@bazel_skylib//rules:common_settings",
     ],

python/private/cc/BUILD.bazel

@@ -0,0 +1,20 @@
+load("@rules_cc//cc:cc_library.bzl", "cc_library")
+load("//python/private:visibility.bzl", "NOT_ACTUALLY_PUBLIC")
+
+package(
+    default_visibility = ["//:__subpackages__"],
+)
+
+licenses(["notice"])
+
+filegroup(
+    name = "distribution",
+    srcs = glob(["**"]),
+)
+
+# An empty cc target for use when a cc target is needed to satisfy
+# Bazel, but its contents don't matter.
+cc_library(
+    name = "empty",
+    visibility = NOT_ACTUALLY_PUBLIC,
+)