feat: add the ability to declare safe tools in a cross-build environment. (#2317)

* Add the ability to declare safe tools in a cross-build environment.

* Add an xfail if cmake isn't available on the test machine.

* Placate linter regarding positional args.

* Rework test to provide more robust confirmation of safe tools.

* Remove a test skip condition that is no longer needed.

Co-authored-by: Joe Rickerby <[email protected]>

* Rename the setting to xbuild-tools.

* Add docs to clarify that xbuild-tools is transitive.

* Raise a warning if xbuild-tools isn't defined.

* Correct a bad copy-paste in the schema generator.

* .. and now fix the indentation.

* Move sentinel handling earlier into the parsing process.

* Remove serialization from tests that won't start a test suite.

---------

Co-authored-by: Joe Rickerby <[email protected]>

Author: freakboy3742

README.md

@@ -139,6 +139,7 @@ Options
 |   | [`CIBW_ENVIRONMENT_PASS_LINUX`](https://cibuildwheel.pypa.io/en/stable/options/#environment-pass)  | Set environment variables on the host to pass-through to the container during the build. |
 |   | [`CIBW_BEFORE_ALL`](https://cibuildwheel.pypa.io/en/stable/options/#before-all)  | Execute a shell command on the build system before any wheels are built. |
 |   | [`CIBW_BEFORE_BUILD`](https://cibuildwheel.pypa.io/en/stable/options/#before-build)  | Execute a shell command preparing each wheel's build |
+|   | [`CIBW_XBUILD_TOOLS`](https://cibuildwheel.pypa.io/en/stable/options/#xbuild-tools)  | Binaries on the path that should be included in an isolated cross-build environment. |
 |   | [`CIBW_REPAIR_WHEEL_COMMAND`](https://cibuildwheel.pypa.io/en/stable/options/#repair-wheel-command)  | Execute a shell command to repair each built wheel |
 |   | [`CIBW_MANYLINUX_*_IMAGE`<br/>`CIBW_MUSLLINUX_*_IMAGE`](https://cibuildwheel.pypa.io/en/stable/options/#linux-image)  | Specify alternative manylinux / musllinux Docker images |
 |   | [`CIBW_CONTAINER_ENGINE`](https://cibuildwheel.pypa.io/en/stable/options/#container-engine)  | Specify which container engine to use when building Linux wheels |

bin/generate_schema.py

@@ -181,9 +181,12 @@
   musllinux-x86_64-image:
     type: string
     description: Specify alternative manylinux / musllinux container images
-  repair-wheel-command:
+  xbuild-tools:
+    description: Binaries on the path that should be included in an isolated cross-build environment
     type: string_array
+  repair-wheel-command:
     description: Execute a shell command to repair each built wheel.
+    type: string_array
   skip:
     description: Choose the Python versions to skip.
     type: string_array
@@ -273,6 +276,7 @@
       properties:
         before-all: {"$ref": "#/$defs/inherit"}
         before-build: {"$ref": "#/$defs/inherit"}
+        xbuild-tools: {"$ref": "#/$defs/inherit"}
         before-test: {"$ref": "#/$defs/inherit"}
         config-settings: {"$ref": "#/$defs/inherit"}
         container-engine: {"$ref": "#/$defs/inherit"}

cibuildwheel/options.py

@@ -94,6 +94,7 @@ class BuildOptions:
     environment: ParsedEnvironment
     before_all: str
     before_build: str | None
+    xbuild_tools: list[str] | None
     repair_command: str
     manylinux_images: dict[str, str] | None
     musllinux_images: dict[str, str] | None
@@ -718,6 +719,18 @@ def _compute_build_options(self, identifier: str | None) -> BuildOptions:
 
             test_command = self.reader.get("test-command", option_format=ListFormat(sep=" && "))
             before_test = self.reader.get("before-test", option_format=ListFormat(sep=" && "))
+            xbuild_tools: list[str] | None = shlex.split(
+                self.reader.get(
+                    "xbuild-tools", option_format=ListFormat(sep=" ", quote=shlex.quote)
+                )
+            )
+            # ["\u0000"] is a sentinel value used as a default, because TOML
+            # doesn't have an explicit NULL value. If xbuild-tools is set to the
+            # sentinel, it indicates that the user hasn't defined xbuild-tools
+            # *at all* (not even an `xbuild-tools = []` definition).
+            if xbuild_tools == ["\u0000"]:
+                xbuild_tools = None
+
             test_sources = shlex.split(
                 self.reader.get(
                     "test-sources", option_format=ListFormat(sep=" ", quote=shlex.quote)
@@ -835,6 +848,7 @@ def _compute_build_options(self, identifier: str | None) -> BuildOptions:
                 before_build=before_build,
                 before_all=before_all,
                 build_verbosity=build_verbosity,
+                xbuild_tools=xbuild_tools,
                 repair_command=repair_command,
                 environment=environment,
                 dependency_constraints=dependency_constraints,

cibuildwheel/platforms/ios.py

@@ -5,6 +5,7 @@
 import shutil
 import subprocess
 import sys
+import textwrap
 from collections.abc import Sequence, Set
 from dataclasses import dataclass
 from pathlib import Path
@@ -151,6 +152,7 @@ def cross_virtualenv(
     build_python: Path,
     venv_path: Path,
     dependency_constraint_flags: Sequence[PathOrStr],
+    xbuild_tools: Sequence[str] | None,
 ) -> dict[str, str]:
     """Create a cross-compilation virtual environment.
 
@@ -178,6 +180,8 @@ def cross_virtualenv(
         created.
     :param dependency_constraint_flags: Any flags that should be used when
         constraining dependencies in the environment.
+    :param xbuild_tools: A list of executable names (without paths) that are
+        on the path, but must be preserved in the cross environment.
     """
     # Create an initial macOS virtual environment
     env = virtualenv(
@@ -210,14 +214,52 @@ def cross_virtualenv(
     #
     # To prevent problems, set the PATH to isolate the build environment from
     # sources that could introduce incompatible binaries.
+    #
+    # However, there may be some tools on the path that are needed for the
+    # build. Find their location on the path, and link the underlying binaries
+    # (fully resolving symlinks) to a "safe" location that will *only* contain
+    # those tools. This avoids needing to add *all* of Homebrew to the path just
+    # to get access to (for example) cmake for build purposes. A value of None
+    # means the user hasn't provided a list of xbuild tools.
+    xbuild_tools_path = venv_path / "cibw_xbuild_tools"
+    xbuild_tools_path.mkdir()
+    if xbuild_tools is None:
+        log.warning(
+            textwrap.dedent(
+                """
+                Your project configuration does not define any cross-build tools.
+
+                iOS builds use an isolated build environment; if your build process requires any
+                third-party tools (such as cmake, ninja, or rustc), you must explicitly declare
+                that those tools are required using xbuild-tools/CIBW_XBUILD_TOOLS. This will
+                likely manifest as a "somebuildtool: command not found" error.
+
+                If the build succeeds, you can silence this warning by setting adding
+                `xbuild-tools = []` to your pyproject.toml configuration, or exporting
+                CIBW_XBUILD_TOOLS as an empty string into your environment.
+                """
+            )
+        )
+    else:
+        for tool in xbuild_tools:
+            tool_path = shutil.which(tool)
+            if tool_path is None:
+                msg = f"Could not find a {tool!r} executable on the path."
+                raise errors.FatalError(msg)
+
+            # Link the binary into the safe tools directory
+            original = Path(tool_path).resolve()
+            print(f"{tool!r} will be included in the cross-build environment (using {original})")
+            (xbuild_tools_path / tool).symlink_to(original)
+
     env["PATH"] = os.pathsep.join(
         [
             # The target python's binary directory
             str(target_python.parent),
-            # The cross-platform environments binary directory
+            # The cross-platform environment's binary directory
             str(venv_path / "bin"),
-            # Cargo's binary directory (to allow for Rust compilation)
-            str(Path.home() / ".cargo" / "bin"),
+            # The directory of cross-build tools
+            str(xbuild_tools_path),
             # The bare minimum Apple system paths.
             "/usr/bin",
             "/bin",
@@ -235,10 +277,12 @@ def cross_virtualenv(
 
 def setup_python(
     tmp: Path,
+    *,
     python_configuration: PythonConfiguration,
     dependency_constraint_flags: Sequence[PathOrStr],
     environment: ParsedEnvironment,
     build_frontend: BuildFrontendName,
+    xbuild_tools: Sequence[str] | None,
 ) -> tuple[Path, dict[str, str]]:
     if build_frontend == "build[uv]":
         msg = "uv doesn't support iOS"
@@ -291,6 +335,7 @@ def setup_python(
         build_python=build_python,
         venv_path=venv_path,
         dependency_constraint_flags=dependency_constraint_flags,
+        xbuild_tools=xbuild_tools,
     )
     venv_bin_path = venv_path / "bin"
     assert venv_bin_path.exists()
@@ -414,10 +459,11 @@ def build(options: Options, tmp_path: Path) -> None:
 
             target_install_path, env = setup_python(
                 identifier_tmp_dir / "build",
-                config,
-                dependency_constraint_flags,
-                build_options.environment,
-                build_frontend.name,
+                python_configuration=config,
+                dependency_constraint_flags=dependency_constraint_flags,
+                environment=build_options.environment,
+                build_frontend=build_frontend.name,
+                xbuild_tools=build_options.xbuild_tools,
             )
             pip_version = get_pip_version(env)
 

cibuildwheel/resources/cibuildwheel.schema.json

@@ -397,6 +397,21 @@
       "description": "Specify alternative manylinux / musllinux container images",
       "title": "CIBW_MUSLLINUX_X86_64_IMAGE"
     },
+    "xbuild-tools": {
+      "description": "Binaries on the path that should be included in an isolated cross-build environment",
+      "oneOf": [
+        {
+          "type": "string"
+        },
+        {
+          "type": "array",
+          "items": {
+            "type": "string"
+          }
+        }
+      ],
+      "title": "CIBW_XBUILD_TOOLS"
+    },
     "repair-wheel-command": {
       "description": "Execute a shell command to repair each built wheel.",
       "oneOf": [
@@ -566,6 +581,9 @@
               "environment-pass": {
                 "$ref": "#/$defs/inherit"
               },
+              "xbuild-tools": {
+                "$ref": "#/$defs/inherit"
+              },
               "repair-wheel-command": {
                 "$ref": "#/$defs/inherit"
               },
@@ -991,6 +1009,9 @@
         "repair-wheel-command": {
           "$ref": "#/properties/repair-wheel-command"
         },
+        "xbuild-tools": {
+          "$ref": "#/properties/xbuild-tools"
+        },
         "test-command": {
           "$ref": "#/properties/test-command"
         },

cibuildwheel/resources/defaults.toml

@@ -14,6 +14,8 @@ build-verbosity = 0
 
 before-all = ""
 before-build = ""
+# TOML doesn't support explicit NULLs; use ["\u0000"] as a sentinel value.
+xbuild-tools = ["\u0000"]
 repair-wheel-command = ""
 
 test-command = ""

docs/options.md

@@ -1043,6 +1043,49 @@ Platform-specific environment variables are also available:<br/>
     [PEP 517]: https://www.python.org/dev/peps/pep-0517/
     [PEP 518]: https://www.python.org/dev/peps/pep-0517/
 
+### `CIBW_XBUILD_TOOLS` {: #xbuild-tools}
+> Binaries on the path that should be included in an isolated cross-build environment.
+
+When building in a cross-platform environment, it is sometimes necessary to isolate the ``PATH`` so that binaries from the build machine don't accidentally get linked into the cross-platform binary. However, this isolation process will also hide tools that might be required to build your wheel.
+
+If there are binaries present on the `PATH` when you invoke cibuildwheel, and those binaries are required to build your wheels, those binaries can be explicitly included in the isolated cross-build environment using `CIBW_XBUILD_TOOLS`. The binaries listed in this setting will be linked into an isolated location, and that isolated location will be put on the `PATH` of the isolated environment. You do not need to provide the full path to the binary - only the executable name that would be found by the shell.
+
+If you declare a tool as a cross-build tool, and that tool cannot be found in the runtime environment, an error will be raised.
+
+If you do not define `CIBW_XBUILD_TOOLS`, and you build for a platform that uses a cross-platform environment, a warning will be raised. If your project does not require any cross-build tools, you can set `CIBW_XBUILD_TOOLS` to an empty list to silence this warning.
+
+*Any* tool used by the build process must be included in the `CIBW_XBUILD_TOOLS` list, not just tools that cibuildwheel will invoke directly. For example, if your build invokes `cmake`, and the `cmake` script invokes `magick` to perform some image transformations, both `cmake` and `magick` must be included in your safe tools list.
+
+Platform-specific environment variables are also available on platforms that use cross-platform environment isolation:<br/>
+ `CIBW_XBUILD_TOOLS_IOS`
+
+#### Examples
+
+!!! tab examples "Environment variables"
+
+    ```yaml
+    # Allow access to the cmake and rustc binaries in the isolated cross-build environment.
+    CIBW_XBUILD_TOOLS: cmake rustc
+    ```
+
+    ```yaml
+    # No cross-build tools are required
+    CIBW_XBUILD_TOOLS:
+    ```
+
+!!! tab examples "pyproject.toml"
+
+    ```toml
+    [tool.cibuildwheel]
+    # Allow access to the cmake and rustc binaries in the isolated cross-build environment.
+    xbuild-tools = ["cmake", "rustc"]
+    ```
+
+    ```toml
+    [tool.cibuildwheel]
+    # No cross-build tools are required
+    xbuild-tools = []
+    ```
 
 ### `CIBW_REPAIR_WHEEL_COMMAND` {: #repair-wheel-command}
 > Execute a shell command to repair each built wheel

docs/platforms/ios.md

@@ -57,7 +57,9 @@ iOS builds support both the `pip` and `build` build frontends. In principle, sup
 
 ## Build environment
 
-The environment used to run builds does not inherit the full user environment - in particular, `PATH` is deliberately re-written. This is because UNIX C tooling doesn't do a great job differentiating between "macOS ARM64" and "iOS ARM64" binaries. If (for example) Homebrew is on the path when compilation commands are invoked, it's easy for a macOS version of a library to be linked into the iOS binary, rendering it unusable on iOS. To prevent this, iOS builds always force `PATH` to a "known minimal" path, that includes only the bare system utilities, plus the current user's cargo folder (to facilitate Rust builds).
+The environment used to run builds does not inherit the full user environment - in particular, `PATH` is deliberately re-written. This is because UNIX C tooling doesn't do a great job differentiating between "macOS ARM64" and "iOS ARM64" binaries. If (for example) Homebrew is on the path when compilation commands are invoked, it's easy for a macOS version of a library to be linked into the iOS binary, rendering it unusable on iOS. To prevent this, iOS builds always force `PATH` to a "known minimal" path, that includes only the bare system utilities, and the iOS compiler toolchain.
+
+If your project requires additional tools to build (such as `cmake`, `ninja`, or `rustc`), those tools must be explicitly declared as cross-build tools using [`CIBW_XBUILD_TOOLS`](../../options#xbuild-tools). *Any* tool used by the build process must be included in the `CIBW_XBUILD_TOOLS` list, not just tools that cibuildwheel will invoke directly. For example, if your build script invokes `cmake`, and the `cmake` script invokes `magick` to perform some image transformations, both `cmake` and `magick` must be included in your cross-build tools list.
 
 ## Tests