pyo3: `libpython.so` missing in the runfiles output from `py_pyo3_library`

[jherland]

This hit us after a recent rules_python upgrade from 1.4.1 to 1.6.0, but I believe that only exposes an underlying issue that have been present for a longer time.

The rules_python upgrade included this change, which bumps the default Python 3.11 version provosioned by rules_python from 3.11.11 to 3.11.13. That bump itself is not problematic, but it does include some other changes in how that Python distribution is built by the upstream astral-sh/python-build-standalone project, specifically this change.

So the Python distribution provided by rules_python has changed from a version where the python executable loads libpython.so at runtime (the readelf -d output includes this line: 0x0000000000000001 (NEEDED) Shared library: [$ORIGIN/../lib/libpython3.11.so.1.0]) to a version where libpython has ben statically linked into the the python executable (the readelf -d output no longer lists libpython at all). What does this have to do with rules_pyo3?

We are building a py_library that imports a pyo3_extension at runtime, the relevant BUILD file looks something like this:

rust_library(
    name = "foo",
    srcs = glob(["src/*.rs"]),
    deps = [
        "@crates//:bar",
        "@crates//:baz",
    ],
)

pyo3_extension(
    name = "foo_pyo3",
    srcs = ["src/python.rs"],
    deps = [
        ":foo",
        "@crates//:anyhow",
        "@crates//:async-broadcast",
        "@crates//:bytes",
        "@crates//:smol",
    ],
)

py_library(
    name = "foo_py",
    srcs = ["foo_py.py"],
    visibility = ["//visibility:public"],
    deps = [":foo_pyo3"],
)

The resulting foo_pyo3.so library has this dynamic section:

Dynamic section at offset 0x31a9d8 contains 31 entries:
  Tag        Type                         Name/Value
 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN/../../../_solib_k8/_U_A_Arules_Upython++python+python_U3_U11_Ux86_U64-unknown-linux-gnu_S_S_Clibpython___Ulib]
 0x0000000000000001 (NEEDED)             Shared library: [libpython3.11.so.1.0]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libm.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-x86-64.so.2]

which I interpret as "This library depends on libpython.so, which - at runtime - will be found at _solib_k8/_U_A_Arules_Upython++python+python_U3_U11_Ux86_U64-unknown-linux-gnu_S_S_Clibpython___Ulib under the Bazel runfiles directory". So far, so good.

However, at runtime we get failures like this:

Traceback:
[...]
path/to/foo/foo_py.py:123: in <module>
    import path.to.foo.foo_pyo3 as foo_pyo3
E   ImportError: libpython3.11.so.1.0: cannot open shared object file: No such file or directory

When I look inside the runfiles directory created by Bazel, the above _solib_k8/_U_A_A... path is nowhere to be found.

Adding debug prints across rules_rust/rules_pyo3, I can see that the rustc_compile_action for the foo_pyo3_shared target is returning a DefaultInfo whose .runfiles do include the libpython.so library, but this library cannot be found in the corresponding foo_pyo3 py_pyo3_library target, and thus these files are missing from the runfiles closure provided to the rules_python part of the build graph.

This is supported by querying these targets explicitly:

$ bazel cquery "//path/to/foo:foo_pyo3_shared" --output=starlark --starlark:expr="providers(target)['DefaultInfo'].default_runfiles.files"
depset([
  <generated file _solib_k8/_U_A_Arules_Upython++python+python_U3_U11_Ux86_U64-unknown-linux-gnu_S_S_Clibpython___Ulib/libpython3.11.so>,
  <generated file _solib_k8/_U_A_Arules_Upython++python+python_U3_U11_Ux86_U64-unknown-linux-gnu_S_S_Clibpython___Ulib/libpython3.11.so.1.0>
], order = "postorder")

$ bazel cquery "//path/to/foo:foo_pyo3" --output=starlark --starlark:expr="providers(target)['DefaultInfo'].default_runfiles.files"
depset([
  <generated file path/to/foo/foo_pyo3.so>
], order = "postorder")

The following patch on rules_rust/rules_pyo3 appears to fix the problem, but I'm not sure it is an appropriate fix in all circumstances:

diff --git a/extensions/pyo3/private/pyo3.bzl b/extensions/pyo3/private/pyo3.bzl
index 0e087b55c..3f0054307 100644
--- a/extensions/pyo3/private/pyo3.bzl
+++ b/extensions/pyo3/private/pyo3.bzl
@@ -114,10 +114,12 @@ def _py_pyo3_library_impl(ctx):
         )
         files.append(stub)
 
+    runfiles = ctx.runfiles(transitive_files = depset(files, transitive = [crate_info.data]))
+    runfiles = runfiles.merge(ctx.attr.extension[DefaultInfo].default_runfiles)
     providers = [
         DefaultInfo(
             files = depset([ext]),
-            runfiles = ctx.runfiles(transitive_files = depset(files, transitive = [crate_info.data])),
+            runfiles = runfiles,
         ),
         PyInfo(
             imports = _get_imports(ctx, ctx.attr.imports),

I believe the reason this issue hasn't shown up until now is that previous distributions of Python - by virtue of explicitly loading libpython.so into the process when the python interpreter is started - has enabled an optimization in the loader while loading foo_pyo3.so: its libpython.so dependency is already satisfied, and so the loader never has to follow the RUNPATH in foo_pyo3.so in order to actually find libpython.so.

Now that libpython is statically compiled into the interpreter, the optimization no longer triggers, and the missing RUNPATH directory actually causes the library load to fail.

[UebelAndre]

I'd be happy to review a PR for the diff. I don't think there'd be a big issue in bringing runfiles from the rust_shared_library target.

[jherland]

Thanks. I started a related thread on the Bazel Slack (https://bazelbuild.slack.com/archives/CA306CEV6/p1761573682612859) where a comment has been raised as to whether the pyo3 shared library should link with libpython.so at all.

While researching that, I'm finding that later versions of rules_pyo3 (just prior to it being merged into rules_rust) is changing if/how this linking is performed.

I'll prepare the PR if I can convince myself that this issue remains valid, even after upgrading to the latest versions of rules_rust + rules_rust_pyo3.

[jherland]

Closing this issue as the underlying cause was that pyo3 extension modules should not link to libpython.so at all. This issue/diff merely fixes a symptom, but does not address the root cause. For future reference, this is the commit un rules_pyo3 that resolves the issue: abrisco/rules_pyo3@f1df71a (any version of rules_rust_pyo3 already includes this fix)