pipstar regression: Could not rename ... : already exists

[thirtyseven]

🐞 bug report

Affected Rule

The issue is caused by the rule: pip.parse

Is this a regression?

Yes, the previous version in which this bug was not present was: 1.7.0

Description

Some pip packages fail to build in 1.8.0-rc1 with errors like:

ERROR: no such package '@@rules_python++pip+pypi_310_libclang//': java.io.IOException: Could not rename /private/var/tmp/_bazel_tkaplan/4c0c47e0a98a460290f6962870658619/external/rules_python++pip+pypi_310_libclang/site-packages/libclang-18.1.1.data/platlib to /private/var/tmp/_bazel_tkaplan/4c0c47e0a98a460290f6962870658619/external/rules_python++pip+pypi_310_libclang/site-packages: already exists

🔬 Minimal Reproduction

MODULE.bazel:

bazel_dep(name = "rules_python", version = "1.8.0-rc1")

python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(
    python_version = "3.10",
)

pip = use_extension("@rules_python//python/extensions:pip.bzl", "pip")
pip.parse(
    hub_name = "pypi",
    python_version = "3.10",
    requirements_by_platform = {
        "//:requirements_linux.txt": "linux_*",
        "//:requirements_mac.txt": "osx_*",
    },
)
use_repo(pip, "pypi")

uv = use_extension(
    "@rules_python//python/uv:uv.bzl",
    "uv",
    dev_dependency = True,
)

BUILD.bazel:

load("@rules_python//python/uv:lock.bzl", "lock")

DEFAULT_UV_ARGS = [
    "--index-strategy=unsafe-best-match",
]

# Generate lock files for default Python version
lock(
    name = "requirements_mac",
    srcs = [":requirements.in"],
    out = "requirements_mac.txt",
    args = DEFAULT_UV_ARGS + ["--python-platform=macos"],
)

lock(
    name = "requirements_linux",
    srcs = [":requirements.in"],
    out = "requirements_linux.txt",
    args = DEFAULT_UV_ARGS + ["--python-platform=linux"],
)

requirements.in:

tensorflow==2.18.0

Run:

bazel run :requirements_mac.update
bazel run :requirements_linux.update
bazel build @pypi//...

🔥 Exception or Error

bazel build @pypi//...     
INFO: Repository rules_python++pip+pypi_310_libclang instantiated at:
  : in 
Repository rule whl_library defined at:
  /private/var/tmp/_bazel_tkaplan/4c0c47e0a98a460290f6962870658619/external/rules_python+/python/private/pypi/whl_library.bzl:730:30: in 
ERROR: /private/var/tmp/_bazel_tkaplan/4c0c47e0a98a460290f6962870658619/external/rules_python+/python/private/pypi/whl_library.bzl:304:28: An error occurred during the fetch of repository 'rules_python++pip+pypi_310_libclang':
   Traceback (most recent call last):
	File "/private/var/tmp/_bazel_tkaplan/4c0c47e0a98a460290f6962870658619/external/rules_python+/python/private/pypi/whl_library.bzl", line 451, column 26, in _whl_library_impl
		_extract_whl_star(rctx, whl_path = whl_path, logger = logger)
	File "/private/var/tmp/_bazel_tkaplan/4c0c47e0a98a460290f6962870658619/external/rules_python+/python/private/pypi/whl_library.bzl", line 304, column 28, in _extract_whl_star
		rctx.rename(src, dest)
Error in rename: java.io.IOException: Could not rename /private/var/tmp/_bazel_tkaplan/4c0c47e0a98a460290f6962870658619/external/rules_python++pip+pypi_310_libclang/site-packages/libclang-18.1.1.data/platlib to /private/var/tmp/_bazel_tkaplan/4c0c47e0a98a460290f6962870658619/external/rules_python++pip+pypi_310_libclang/site-packages: already exists
ERROR: no such package '@@rules_python++pip+pypi_310_libclang//': java.io.IOException: Could not rename /private/var/tmp/_bazel_tkaplan/4c0c47e0a98a460290f6962870658619/external/rules_python++pip+pypi_310_libclang/site-packages/libclang-18.1.1.data/platlib to /private/var/tmp/_bazel_tkaplan/4c0c47e0a98a460290f6962870658619/external/rules_python++pip+pypi_310_libclang/site-packages: already exists
ERROR: /private/var/tmp/_bazel_tkaplan/4c0c47e0a98a460290f6962870658619/external/rules_python++pip+pypi/libclang/BUILD.bazel:5:12: @@rules_python++pip+pypi//libclang:dist_info depends on @@rules_python++pip+pypi_310_libclang//:dist_info in repository @@rules_python++pip+pypi_310_libclang which failed to fetch. no such package '@@rules_python++pip+pypi_310_libclang//': java.io.IOException: Could not rename /private/var/tmp/_bazel_tkaplan/4c0c47e0a98a460290f6962870658619/external/rules_python++pip+pypi_310_libclang/site-packages/libclang-18.1.1.data/platlib to /private/var/tmp/_bazel_tkaplan/4c0c47e0a98a460290f6962870658619/external/rules_python++pip+pypi_310_libclang/site-packages: already exists
Use --verbose_failures to see the command lines of failed build steps.
ERROR: Analysis of target '@@rules_python++pip+pypi//libclang:dist_info' failed; build aborted: Analysis failed

🌍 Your Environment

Operating System:

  
Mac OS 15.7.3
  

Output of bazel version:

  
8.5.0
  

Rules_python version:

  
1.8.0-rc1
  

Anything else relevant?

This issue doesn't occur when using --repo_env=RULES_PYTHON_ENABLE_PIPSTAR=0

[FrankPortman]

Just out of curiosity, what happens when you don't do requirements by platform like you did, and instead pass in a --universal flag to uv, and supply a single lockfile to pip.parse? As far as I can tell, that is the only large difference between our two setups, and 1.8.0-rc1 works for me. We are on tensorflow==2.19.0 but I assume (perhaps very wrongfully) that that isn't the difference.

Other differences from my setup, Python 3.12 and newer Mac OSX. So perhaps those could be at play as well.

[aignas]

Sorry for that, this was easy to fix, please check the attached PR.

The reason is that the libclang wheel stores everything in .data/platlib folder and the site-packages to which we would be moving things already existed. Doing a one level merge would have been sufficient but I decided to be extra careful and handle cases where wheels have platlib and purelib folders in .data and they need to be deep-merged.

For reference, the libclang wheel contents can be seen: https://pypi-browser.org/package/libclang/libclang-18.1.1-1-py2.py3-none-macosx_11_0_arm64.whl