Add fallback platforms for stager#38688
Conversation
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request improves the reliability of the Apache Beam Stager by implementing a fallback mechanism for manylinux platform tags. By attempting downloads with progressively more compatible platform tags when the preferred tag fails, the system avoids unnecessary errors during the staging of dependencies that may not have published modern wheels. Highlights
New Features🧠 You can now enable Memory (public preview) to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a fallback mechanism for downloading Python dependencies using pip, iterating through a list of older manylinux platform tags if the preferred modern tag fails. The review feedback highlights that the hardcoded _MANYLINUX_PLATFORMS list only supports _x86_64 architectures, which bypasses the fallback mechanism on ARM64 (aarch64) environments. To resolve this, it is recommended to define base tags and dynamically append the architecture suffix, while filtering out unsupported combinations like manylinux2010 on ARM64.
| _MANYLINUX_PLATFORMS = [ | ||
| 'manylinux_2_28_x86_64', | ||
| 'manylinux2014_x86_64', # equivalent to manylinux_2_17 | ||
| 'manylinux2010_x86_64', # equivalent to manylinux_2_12 | ||
| ] |
There was a problem hiding this comment.
The hardcoded _MANYLINUX_PLATFORMS list only contains _x86_64 platform tags. This means that on ARM64 (aarch64) environments (such as Apple Silicon or AWS Graviton), the fallback mechanism will be completely bypassed because the preferred platform (e.g., manylinux_2_28_aarch64) won't be found in this list.
To support both x86_64 and aarch64 architectures, we should define a list of base manylinux tags and dynamically append the appropriate architecture suffix.
| _MANYLINUX_PLATFORMS = [ | |
| 'manylinux_2_28_x86_64', | |
| 'manylinux2014_x86_64', # equivalent to manylinux_2_17 | |
| 'manylinux2010_x86_64', # equivalent to manylinux_2_12 | |
| ] | |
| _MANYLINUX_BASE_TAGS = [ | |
| 'manylinux_2_28', | |
| 'manylinux2014', # equivalent to manylinux_2_17 | |
| 'manylinux2010', # equivalent to manylinux_2_12 | |
| ] |
| try: | ||
| start_idx = _MANYLINUX_PLATFORMS.index(preferred_platform) | ||
| platforms = _MANYLINUX_PLATFORMS[start_idx:] | ||
| except ValueError: | ||
| platforms = [preferred_platform] |
There was a problem hiding this comment.
Since we've refactored _MANYLINUX_PLATFORMS to _MANYLINUX_BASE_TAGS to support multiple architectures, we should dynamically detect the architecture suffix (e.g., _x86_64 or _aarch64) from the preferred_platform and reconstruct the platform list. Note that manylinux2010 is not supported on aarch64, so we should exclude it for ARM64.
| try: | |
| start_idx = _MANYLINUX_PLATFORMS.index(preferred_platform) | |
| platforms = _MANYLINUX_PLATFORMS[start_idx:] | |
| except ValueError: | |
| platforms = [preferred_platform] | |
| arch = None | |
| for suffix in ['_x86_64', '_aarch64']: | |
| if preferred_platform.endswith(suffix): | |
| arch = suffix | |
| break | |
| if arch: | |
| platforms_with_arch = [ | |
| f'{base}{arch}' | |
| for base in _MANYLINUX_BASE_TAGS | |
| if not (base == 'manylinux2010' and arch == '_aarch64') | |
| ] | |
| try: | |
| start_idx = platforms_with_arch.index(preferred_platform) | |
| platforms = platforms_with_arch[start_idx:] | |
| except ValueError: | |
| platforms = [preferred_platform] | |
| else: | |
| platforms = [preferred_platform] |
|
Assigning reviewers: R: @tvalentyn for label python. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
1 participant
Add fallback manylinux platform tags for pip download in Stager. (follow-up of #38611)
Ensures that cross-platform binary downloads for environment staging do not fail if PyPI dependencies (e.g. TensorFlow, Numpy, etc) do not publish wheels matching the strictest modern manylinux tag. Falls back dynamically to older, compatible manylinux standards (manylinux2014/manylinux2010).
This fixed PreCommit Python 3.10 failure on
MLTest.test_ml_preprocessing_yaml(https://github.com/apache/beam/actions/runs/26420471407/job/77774009271), and PostCommit Python 3.10 failure onVertexAIImageEmbeddingsTest.test_image_embedding_pipeline_from_path(https://github.com/apache/beam/actions/runs/26424456835/job/77785233507).