Unable to use whl dependency which is compiled from source due (arch not provided in pypi)

[shkarupa-alex]

🐞 bug report

Affected Rule

he issue is caused by the rule: pip_parse

Is this a regression?

No

Description

I'm developing a tensorflow addon with custom ops on MacBook with Intel CPU. All tensorflow versions below 2.17 had prebuilt macosx__x86_64.whl packages. But since v2.17 there are only arm64 versions. The only way i can use fresh tensorflow is to built it itself. As result i have a custom tensorflow whl for my arch.

But this custom-arch self-compiled whl does not exists in PyPi, pip_parse can't find it. And build fails...

Is there any workaround to use locally-built whl?

🔬 Minimal Reproduction

Exception is related to PyPi package dependency (tensorflow in my case) with specific version and arch (MacOS x86_64 in my case).

🔥 Exception or Error

===== stderr start =====
ERROR: Could not find a version that satisfies the requirement tensorflow==2.17.0 (from versions: 2.12.0rc0, 2.12.0rc1, 2.12.0, 2.12.1, 2.13.0rc0, 2.13.0rc1, 2.13.0rc2, 2.13.0, 2.13.1, 2.14.0rc0, 2.14.0rc1, 2.14.0, 2.14.1, 2.15.0rc0, 2.15.0rc1, 2.15.0, 2.15.1, 2.16.0rc0, 2.16.1, 2.16.2)
ERROR: No matching distribution found for tensorflow==2.17.0
Traceback (most recent call last):
  File "", line 198, in _run_module_as_main
  File "", line 88, in _run_code
  File "/private/var/tmp/_bazel_alex/9f232a1fa475e5223cc8bd12f98cbb55/external/rules_python~0.34.0/python/private/pypi/whl_installer/wheel_installer.py", line 205, in 
    main()
  File "/private/var/tmp/_bazel_alex/9f232a1fa475e5223cc8bd12f98cbb55/external/rules_python~0.34.0/python/private/pypi/whl_installer/wheel_installer.py", line 190, in main
    subprocess.run(pip_args, check=True, env=env)
  File "/private/var/tmp/_bazel_alex/9f232a1fa475e5223cc8bd12f98cbb55/external/rules_python~0.34.0~python~python_3_11_x86_64-apple-darwin/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/private/var/tmp/_bazel_alex/9f232a1fa475e5223cc8bd12f98cbb55/external/rules_python~0.34.0~python~python_3_11_host/python', '-m', 'pip', '--isolated', 'wheel', '--no-deps', '-r', '/var/folders/46/2h_pg_7s1r70lcfx37hb67nw0000gn/T/tmpoiw9bif1']' returned non-zero exit status 1.
===== stderr end =====

🌍 Your Environment

Operating System:

  
MacOS X 14.5 (23F79)
  

Output of bazel version:

  
Bazelisk version: 1.20.0
Build label: 6.5.0
Build target: bazel-out/darwin-opt/bin/src/main/java/com/google/devtools/build/lib/bazel/BazelServer_deploy.jar
Build time: Tue Jan 23 16:54:12 2024 (1706028852)
Build timestamp: 1706028852
Build timestamp as int: 1706028852
  

Rules_python version:

  
0.34.0
  

[TimotheusBachinger]

I am confronted with (I guess) the same issue. however, I am already on bzlmod (so pip.parse).
We're building our own wheel via https://github.com/Checkmk/checkmk/blob/379798b102e60f213b1866fa318f36a7a50ec85a/omd/packages/rrdtool/BUILD.rrdtool-native.bazel#L221
The wheel is available in bazel (under bazel-bin/external/rrdtool_native/rrdtool-1-py3-none-any.whl).

How can I teach pip.parse to use that wheel instead of the one which is (in my case) provided on pypi?

[shkarupa-alex]

I am confronted with (I guess) the same issue. however, I am already on bzlmod (so pip.parse). We're building our own wheel via https://github.com/Checkmk/checkmk/blob/379798b102e60f213b1866fa318f36a7a50ec85a/omd/packages/rrdtool/BUILD.rrdtool-native.bazel#L221 The wheel is available in bazel (under bazel-bin/external/rrdtool_native/rrdtool-1-py3-none-any.whl).

How can I teach pip.parse to use that wheel instead of the one which is (in my case) provided on pypi?

I found this workaround tensorflow/text#1319 (comment)

[TimotheusBachinger]

hey @shkarupa-alex unfortunately this would not work at all for me the absolute path to the wheel is only known to bazel and would change from machine to machine... there must be a proper bazel way

[aignas]

FYI, there is an example on how to include pytorch in #2449.

As the ticket right now is, it is unclear exactly what the repo structure exactly is. Could you please provide a minimum reproducible example?.

[TimotheusBachinger]

in the meantime, my colleague found a way to implement this - however as we're using rules_uv under the hood, it would not directly apply for rules_python.
to get an idea what we want, here is the commit: Checkmk/checkmk@68d1290