feat(rules): allow deriving custom rules from core rules

Author: rickeylev

CHANGELOG.md

@@ -91,6 +91,9 @@ Unreleased changes template.
 * (pypi) Direct HTTP urls for wheels and sdists are now supported when using
   {obj}`experimental_index_url` (bazel downloader).
   Partially fixes [#2363](https://github.com/bazelbuild/rules_python/issues/2363).
+* (rules) APIs for creating custom rules based on the core py_binary, py_test,
+  and py_library rules
+  ([#1647](https://github.com/bazelbuild/rules_python/issues/1647))
 
 {#v0-0-0-removed}
 ### Removed

docs/_includes/volatile_api.md

@@ -0,0 +1,5 @@
+:::{important}
+
+**Public, but volatile, API.** Some parts are stable, while others are
+implementation details and may change more frequently.
+:::

docs/extending.md

@@ -0,0 +1,129 @@
+# Extending the rules
+
+:::{important}
+**This is public, but volatile, functionality.**
+
+Extending and customizing the rules is supported functionality, but with weaker
+backwards compatibility guarantees, and is not fully subject to the normal
+backwards compatibility procedures and policies. It's simply not feasible to
+support every possible customization with strong backwards compatibility
+guarantees.
+:::
+
+Because of the rich ecosystem of tools and variety of use cases, APIs are
+provided to make it easy to create custom rules using the existing rules as a
+basis. This allows implementing behaviors that aren't possible using
+wrapper macros around the core rules, and can make certain types of changes
+much easier and transparent to implement.
+
+:::{note}
+It is not required to extend a core rule. The minimum requirement for a custom
+rule is to return the appropriate provider (e.g. {bzl:obj}`PyInfo` etc).
+Extending the core rules is most useful when you want all or most of the
+behavior of a core rule.
+:::
+
+Follow or comment on https://github.com/bazelbuild/rules_python/issues/1647
+for the development of APIs to support custom derived rules.
+
+## Creating custom rules
+
+Custom rules can be created using the core rules as a basis by using their rule
+builder APIs.
+
+* {bzl:obj}`//python/apis:executables.bzl%executables` for builders for executables
+* {bzl:obj}`//python/apis:libraries.bzl%libraries` for builders for libraries
+
+These builders create {bzl:obj}`ruleb.Rule` objects, which are thin
+wrappers around the keyword arguments eventually passed to the `rule()`
+function. These builder APIs give access to the _entire_ rule definition and
+allow arbitrary modifications.
+
+### Example: validating a source file
+
+In this example, we derive from `py_library` a custom rule that verifies source
+code contains the word "snakes". It does this by:
+
+* Adding an implicit dependency on a checker program
+* Calling the base implementation function
+* Running the checker on the srcs files
+* Adding the result to the `_validation` output group (a special output
+  group for validation behaviors).
+
+To users, they can use `has_snakes_library` the same as `py_library`. The same
+is true for other targets that might consume the rule.
+
+```
+load("@rules_python//python/api:libraries.bzl", "libraries")
+load("@rules_python//python/api:attr_builders.bzl", "attrb")
+
+def _has_snakes_impl(ctx, base):
+    providers = base(ctx)
+
+    out = ctx.actions.declare_file(ctx.label.name + "_snakes.check")
+    ctx.actions.run(
+        inputs = ctx.files.srcs,
+        outputs = [out],
+        executable = ctx.attr._checker[DefaultInfo].files_to_run,
+        args = [out.path] + [f.path for f in ctx.files.srcs],
+    )
+    prior_ogi = None
+    for i, p in enumerate(providers):
+        if type(p) == "OutputGroupInfo":
+            prior_ogi = (i, p)
+            break
+    if prior_ogi:
+        groups = {k: getattr(prior_ogi[1], k) for k in dir(prior_ogi)}
+        if "_validation" in groups:
+            groups["_validation"] = depset([out], transitive=groups["_validation"])
+        else:
+            groups["_validation"] = depset([out])
+        providers[prior_ogi[0]] = OutputGroupInfo(**groups)
+    else:
+        providers.append(OutputGroupInfo(_validation=depset([out])))
+    return providers
+
+def create_has_snakes_rule():
+    r = libraries.py_library_builder()
+    base_impl = r.implementation()
+    r.set_implementation(lambda ctx: _has_snakes_impl(ctx, base_impl))
+    r.attrs["_checker"] = attrb.Label(
+        default="//:checker",
+        executable = True,
+    )
+    return r.build()
+has_snakes_library = create_has_snakes_rule()
+```
+
+### Example: adding transitions
+
+In this example, we derive from `py_binary` to force building for a particular
+platform. We do this by:
+
+* Adding an additional output to the rule's cfg
+* Calling the base transition function
+* Returning the new transition outputs
+
+```starlark
+
+load("@rules_python//python/api:executables.bzl", "executables")
+
+def _force_linux_impl(settings, attr, base_impl):
+    settings = base_impl(settings, attr)
+    settings["//command_line_option:platforms"] = ["//my/platforms:linux"]
+    return settings
+
+def create_rule():
+    r = executables.py_binary_rule_builder()
+    base_impl = r.cfg.implementation()
+    r.cfg.set_implementation(
+        lambda settings, attr: _force_linux_impl(settings, attr, base_impl)
+    )
+    r.cfg.add_output("//command_line_option:platforms")
+    return r.build()
+
+py_linux_binary = create_linux_binary_rule()
+```
+
+Users can then use `py_linux_binary` the same as a regular py_binary. It will
+act as if `--platforms=//my/platforms:linux` was specified when building it.

docs/index.md

@@ -101,6 +101,7 @@ pip
 coverage
 precompiling
 gazelle
+Extending <extending>
 Contributing <contributing>
 support
 Changelog <changelog>

python/api/executables.bzl

@@ -0,0 +1,29 @@
+# Copyright 2025 The Bazel Authors. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Loading-phase APIs specific to executables (binaries/tests).
+
+:::{versionadded} VERSION_NEXT_FEATURE
+:::
+"""
+
+load("//python/private:py_binary_rule.bzl", "create_py_binary_rule_builder")
+load("//python/private:py_executable.bzl", "create_executable_rule_builder")
+load("//python/private:py_test_rule.bzl", "create_test_rule_builder")
+
+executables = struct(
+    py_binary_rule_builder = create_py_binary_rule_builder,
+    py_test_rule_builder = create_test_rule_builder,
+    executable_rule_builder = create_executable_rule_builder,
+)

python/api/libraries.bzl

@@ -0,0 +1,25 @@
+# Copyright 2025 The Bazel Authors. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Loading-phase APIs specific to executables (binaries/tests).
+
+:::{versionadded} VERSION_NEXT_FEATURE
+:::
+"""
+
+load("//python/private:py_library.bzl", "create_py_library_rule_builder")
+
+libraries = struct(
+    py_library_rule_builder = create_py_library_rule_builder,
+)

python/private/attr_builders.bzl

@@ -12,7 +12,11 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-"""Builders for creating attributes et al."""
+"""Builders for creating attributes et al.
+
+:::{versionadded} VERSION_NEXT_FEATURE
+:::
+"""
 
 load("@bazel_skylib//lib:types.bzl", "types")
 load(

python/private/py_binary_rule.bzl

@@ -27,12 +27,25 @@ def _py_binary_impl(ctx):
         inherited_environment = [],
     )
 
-def create_binary_rule_builder():
+# NOTE: Exported publicly
+def create_py_binary_rule_builder():
+    """Create a rule builder for a py_binary.
+
+    :::{include} /_includes/volatile_api.md
+    :::
+
+    :::{versionadded} VERSION_NEXT_FEATURE
+    :::
+
+    Returns:
+        {type}`ruleb.Rule` with the necessary settings
+        for creating a `py_binary` rule.
+    """
     builder = create_executable_rule_builder(
         implementation = _py_binary_impl,
         executable = True,
     )
     builder.attrs.update(AGNOSTIC_BINARY_ATTRS)
     return builder
 
-py_binary = create_binary_rule_builder().build()
+py_binary = create_py_binary_rule_builder().build()

python/private/py_executable.bzl

@@ -1737,7 +1737,24 @@ def create_base_executable_rule():
     """
     return create_executable_rule_builder().build()
 
+# NOTE: Exported publicly
 def create_executable_rule_builder(implementation, **kwargs):
+    """Create a rule builder for an executable Python program.
+
+    :::{include} /_includes/volatile_api.md
+    :::
+
+    An executable rule is one that sets either `executable=True` or `test=True`,
+    and the output is something that can be run directly (e.g. `bazel run`,
+    `exec(...)` etc)
+
+    :::{versionadded} VERSION_NEXT_FEATURE
+    :::
+
+    Returns:
+        {type}`ruleb.Rule` with the necessary settings
+        for creating an executable Python rule.
+    """
     builder = ruleb.Rule(
         implementation = implementation,
         attrs = EXECUTABLE_ATTRS,

python/private/py_library.bzl

@@ -141,32 +141,28 @@ Source files are no longer added to the runfiles directly.
 :::
 """
 
-def create_py_library_rule_builder(*, attrs = {}, **kwargs):
-    """Creates a py_library rule.
+# NOTE: Exported publicaly
+def create_py_library_rule_builder():
+    """Create a rule builder for a py_library.
 
-    Args:
-        attrs: dict of rule attributes.
-        **kwargs: Additional kwargs to pass onto {obj}`ruleb.Rule()`.
+    :::{include} /_includes/volatile_api.md
+    :::
+
+    :::{versionadded} VERSION_NEXT_FEATURE
+    :::
 
     Returns:
-        {type}`ruleb.Rule` builder object.
+        {type}`ruleb.Rule` with the necessary settings
+        for creating a `py_library` rule.
     """
-
-    # Within Google, the doc attribute is overridden
-    kwargs.setdefault("doc", _DEFAULT_PY_LIBRARY_DOC)
-
-    # TODO: b/253818097 - fragments=py is only necessary so that
-    # RequiredConfigFragmentsTest passes
-    fragments = kwargs.pop("fragments", None) or []
-    kwargs["exec_groups"] = REQUIRED_EXEC_GROUP_BUILDERS | (kwargs.get("exec_groups") or {})
-
     builder = ruleb.Rule(
-        attrs = dicts.add(LIBRARY_ATTRS, attrs),
-        fragments = fragments + ["py"],
+        doc = _DEFAULT_PY_LIBRARY_DOC,
+        exec_groups = REQUIRED_EXEC_GROUP_BUILDERS,
+        attrs = LIBRARY_ATTRS,
+        fragments = ["py"],
         toolchains = [
             ruleb.ToolchainType(TOOLCHAIN_TYPE, mandatory = False),
             ruleb.ToolchainType(EXEC_TOOLS_TOOLCHAIN_TYPE, mandatory = False),
         ],
-        **kwargs
     )
     return builder