pip.parse parsing all requirements files fails on platform specific wheel downloads

[keith]

🐞 bug report

Affected Rule

On this branch: #2449 which sets up pypi for 3 different platforms with different requirements files. Running bazel query 'deps(...)' results in a wheel download failure if you're not on x86_64 linux, since the deps for that platform are not available on other platforms:

===== stderr end ===== and referenced by '@@rules_python~~pip~pip//torch:pkg'
ERROR: Evaluation of query "deps(...)" failed: preloading transitive closure failed: no such package '@@rules_python~~pip~pip_311_torch_linux_x86_64//':
rules_python:whl_library(@@rules_python~~pip~pip_311_torch_linux_x86_64) FAIL: repo.execute: whl_library.ResolveRequirement(rules_python~~pip~pip_311_torch_linux_x86_64, torch==2.4.1+cpu): end: failure:
  command: /home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~python~python_3_11_host/python -m python.private.pypi.whl_installer.wheel_installer --requirement torch==2.4.1+cpu --isolated --extra_pip_args "{\"arg\":[\"--index-url\",\"https://pypi.org/simple\",\"--extra-index-url\",\"https://download.pytorch.org/whl/cpu\"]}" --pip_data_exclude "{\"arg\":[]}" --environment "{\"arg\":{}}"
  return code: 1
  working dir: <default: /home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~pip~pip_311_torch_linux_x86_64>
  timeout: 600
  environment:
PYTHONPATH="/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__build:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__click:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__colorama:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__importlib_metadata:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__installer:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__more_itertools:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__packaging:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__pep517:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__pip:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__pip_tools:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__pyproject_hooks:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__setuptools:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__tomli:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__wheel:/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~internal_deps~pypi__zipp"
CPPFLAGS="-isystem /home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~python~python_3_11_host/include/python3.11"
===== stdout start =====
Looking in indexes: https://pypi.org/simple, https://download.pytorch.org/whl/cpu
===== stdout end =====
===== stderr start =====
ERROR: Could not find a version that satisfies the requirement torch==2.4.1+cpu (from versions: 2.0.0, 2.0.1, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.3.1, 2.4.0, 2.4.1, 2.5.0, 2.5.1)
ERROR: No matching distribution found for torch==2.4.1+cpu
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~/python/private/pypi/whl_installer/wheel_installer.py", line 205, in <module>
    main()
  File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~/python/private/pypi/whl_installer/wheel_installer.py", line 190, in main
    subprocess.run(pip_args, check=True, env=env)
  File "/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~python~python_3_11_aarch64-unknown-linux-gnu/lib/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python~~python~python_3_11_host/python', '-m', 'pip', '--isolated', 'wheel', '--no-deps', '--index-url', 'https://pypi.org/simple', '--extra-index-url', 'https://download.pytorch.org/whl/cpu', '-r', '/tmp/tmprruvzh7j']' returned non-zero exit status 1.
===== stderr end =====

Is this a regression?

This was caused by this default flip: daed352

Related to https://github.com/bazelbuild/rules_python/issues/keith

Currently @ HEAD you have to revert the root cause of #2446 to repro this issue

[aignas]

This is somewhat of a known thing - bazel query forces all of the repos to be downloaded, not only the ones that are needed for a particular platform. There are multiple ways this can be solved:

• Users switch to bazel cquery
• Users start using download_only = True attribute.
• Users use download_only = True with experimental_index_url.
• rules_python strips the env marker, in this case platform_machine == foo from the requirement_line that is passed to whl_library. That will make the fetching not fail, but the result that is materialized within the repo rule would be different depending on the platform.

I personally think we as maintainers can only document what users can do and strip the markers from the whl_library because they don't add any value when fetching the wheels.

Would that be acceptable?

[keith]

I don't know the full implications of suggestion 3 there, but trying that it doesn't seem to work. I still see failures across platforms for the inability to download the platform specific wheels. I repro'd on this branch #2449 by adding:

    download_only = True,
    experimental_index_url = "https://pypi.org/simple",  # use Bazel downloader

to the MODULE.bazel and running bazel query 'deps(...)'.

[aignas]

For this to work you would need to have lock files with hashes --generate-hashes needs to be passed to uv. #2023 is tracking to support this without the hashes.

[keith]

nice looks like that does fix that example, although it does produce a whole bunch of these warnings:

DEBUG: /home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python+/python/private/repo_utils.bzl:79:16:
rules_python:pypi:create_whl_repos WARNING: Could not find a whl or an sdist with sha256=8800deef0026011d502c0c256cc4b67d002347f63c3a38cd8e45f1f445c61364
DEBUG: /home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python+/python/private/repo_utils.bzl:79:16:
rules_python:pypi:create_whl_repos WARNING: Could not find a whl or an sdist with sha256=c4f2c3c026e876d4dad7629170ec14fff48c076d6c2ae0e354ab3fdc09024f00

not sure if that's on rules_python or upstream tho

[keith]

and then doing bazel test ... on that branch results in:

ERROR: /home/ubuntu/.cache/bazel/_bazel_ubuntu/46f0c9813505c30d178a2aff399354ea/external/rules_python++pip+pip/torch/BUILD.bazel:6:12: configurable attribute "actual" in @@rules_python++pip+pip//torch:pkg doesn't match this configuration: No matching wheel for current configuration's Python version.

The current build configuration's Python version doesn't match any of the Python
wheels available for this distribution. This distribution supports the following Python
configuration settings:
    //_config:is_cp3.11_cp3x_none_osx_11_0_aarch64
    //_config:is_cp3.11_cp3x_none_osx_aarch64

not sure if that's related

[keith]

@aignas can we reopen, seems like @ HEAD this is still an issue, pushed the repro case to #2449 with bazel query 'deps(...)' on a non-linux x86_64 machine

[keith]

reopened as #2622