Regenerate beam container image dependencies to include newer version of grpc. by tvalentyn · Pull Request #37822 · apache/beam

tvalentyn · 2026-03-11T01:24:16Z

This is a followup for: #37817

Notable updates:

grpcio: 1.65.0 -> 1.78.0
protobuf: 5.x -> 6.x
envoy-data-plane -- updates to various newer versions, depending on python version.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

gemini-code-assist · 2026-03-11T01:24:33Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request aims to streamline and simplify the dependency management for gRPC-related packages within the Python SDK. It unifies grpcio-tools and envoy-data-plane to single version specifications and refines the grpcio dependency bounds, reducing complexity and potential version conflicts across different Python environments.

Highlights

grpcio-tools Dependency Unification: The grpcio-tools dependency in pyproject.toml has been unified to version 1.71.0, removing the conditional pinning based on Python versions (<= 3.12 and >= 3.13).
envoy-data-plane Dependency Simplification: The envoy-data-plane dependency in setup.py has been simplified, removing the conditional pinning for Python versions < 3.13 and >= 3.13, and now uses a single range envoy-data-plane>=1.0.3,<2.
grpcio Dependency Refinement: The grpcio dependency in setup.py has been updated, removing the Python version-specific bounds and consolidating the exclusion list. The TODO comment regarding consolidation has also been removed.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

sdks/python/pyproject.toml
- Unified grpcio-tools dependency to version 1.71.0 for all Python versions.
sdks/python/setup.py
- Simplified envoy-data-plane dependency to a single version range.
- Refined grpcio dependency bounds, removing Python version-specific conditions and consolidating exclusions.

Ignored Files

Ignored by pattern: .github/workflows/** (1)
- .github/workflows/beam_PreCommit_GoPortable.yml

Activity

No specific human activity (comments, reviews) was recorded for this pull request based on the provided context. The pull request description contains a standard contribution checklist and CI status links.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

tvalentyn · 2026-03-13T13:34:10Z

sdks/python/setup.py

              # hard requirement in setup.py.
              'dill',
              'tensorflow-transform',
+              'protobuf<4; python_version<"3.11"',


Note: without this change,

beam_PreCommit_Yaml_Xlang_Direct suite failed to resolve dependencies. Looking at successful logs (https://github.com/apache/beam/actions/runs/23049196161/job/66945823677) the suite already installed protobuf 3:

2026-03-13T11:58:14.2393149Z Successfully installed absl-py-1.4.0 annotated-doc-0.0.4 apache-beam-2.73.0.dev0 astunparse-1.6.3 click-8.3.1 coloredlogs-15.0.1 contourpy-1.3.2 cuda-bindings-12.9.4 cuda-pathfinder-1.4.2 cycler-0.12.1 datatable-1.1.0 dill-0.4.1 embeddings-0.0.8 flatbuffers-25.12.19 fonttools-4.62.0 fsspec-2026.2.0 gast-0.7.0 google-api-core-2.29.0 google-api-python-client-1.12.11 google_pasta-0.2.0 grpcio-status-1.49.0rc1 h5py-3.16.0 hf-xet-1.4.2 huggingface-hub-1.7.1 humanfriendly-10.0 js2py-0.74 jsonpatch-1.33 jsonpointer-3.0.0 jsonschema-4.26.0 jsonschema-specifications-2025.9.1 keras-3.12.1 kiwisolver-1.5.0 langchain-1.2.12 langchain-core-1.2.18 langgraph-1.1.2 langgraph-checkpoint-4.0.1 langgraph-prebuilt-1.0.8 langgraph-sdk-0.3.11 langsmith-0.7.17 libclang-18.1.1 llvmlite-0.46.0 markdown-3.10.2 markdown-it-py-4.0.0 matplotlib-3.10.8 mdurl-0.1.2 ml-dtypes-0.3.2 mpmath-1.3.0 namex-0.1.0 networkx-3.4.2 numba-0.64.0 numpy-1.26.4 nvidia-cublas-cu12-12.8.4.1 nvidia-cuda-cupti-cu12-12.8.90 nvidia-cuda-nvrtc-cu12-12.8.93 nvidia-cuda-runtime-cu12-12.8.90 nvidia-cudnn-cu12-9.10.2.21 nvidia-cufft-cu12-11.3.3.83 nvidia-cufile-cu12-1.13.1.3 nvidia-curand-cu12-10.3.9.90 nvidia-cusolver-cu12-11.7.3.90 nvidia-cusparse-cu12-12.5.8.93 nvidia-cusparselt-cu12-0.7.1 nvidia-nccl-cu12-2.27.5 nvidia-nvjitlink-cu12-12.8.93 nvidia-nvshmem-cu12-3.4.5 nvidia-nvtx-cu12-12.8.90 onnx-1.17.0 onnxruntime-1.23.2 opt_einsum-3.4.0 optree-0.19.0 ormsgpack-1.12.2 pandas-1.5.3 protobuf-3.20.3 pyarrow-10.0.1 pydot-1.4.2 pyjsparser-2.7.1 pyod-2.0.7 referencing-0.37.0 requests-toolbelt-1.0.0 rich-14.3.3 rpds-py-0.30.0 safetensors-0.7.0 sentence-transformers-5.3.0 shellingham-1.5.4 skl2onnx-1.20.0 sympy-1.14.0 tensorboard-2.16.2 tensorboard-data-server-0.7.2 tensorflow-2.16.2 tensorflow-hub-0.16.1 tensorflow-io-gcs-filesystem-0.37.1 tensorflow-metadata-1.16.1 tensorflow-serving-api-2.16.1 tensorflow-transform-1.16.0 termcolor-3.3.0 tf-keras-2.16.0 tf2onnx-1.17.0 tfx-bsl-1.16.1 tokenizers-0.22.2 torch-2.10.0 tqdm-4.67.3 transformers-5.3.0 triton-3.6.0 typer-0.24.1 tzlocal-5.3.1 uritemplate-3.0.1 uuid-utils-0.14.1 werkzeug-3.1.6 wheel-0.46.3 xxhash-3.6.0 2026-03-13T11:58:22.8382660Z 2026-03-13T11:58:22.8383507Z > Task :sdks:python:yamlIntegrationTests

Yaml suite now fails with a different error. It might be that stubs built with protobuf 6 are not compatible with protobuf 3.
We might have to figure out what in tftransform & friends requires downgrading protobuf...

TFT still doesn't support protobuf 6: https://github.com/tensorflow/transform/blob/339b0cba4ab6e511c47ba20d576d6f2f08d603ba/setup.py#L51

tvalentyn · 2026-03-13T13:39:34Z

sdks/python/setup.py

          'tensorflow-hub': ['tensorflow-hub>=0.14.0,<0.16.0'],
          'milvus': milvus_dependency,
-          'vllm': ['openai==1.107.1', 'vllm==0.14', 'triton==3.3.1']
+          'vllm': ['openai==1.107.1', 'vllm==0.10.1.1', 'triton==3.3.1']


Running gradlew :sdks:python:container:generatePythonRequirementsAll failed to resolve some dependencies without this revert.

If tests pass on this PR, I'd suggest to make another change to upgrade vllm and regenerate the dependencies. I suspect generating requirements on master is currently broken but haven't checked.

cc: @claudevdm

tvalentyn · 2026-03-13T16:54:54Z

Wordcount IT failed with a strange error, likely a flake at submission:

scheduling tests via LoadScheduling

malloc(): unsorted double linked list corrupted
Fatal Python error: Aborted

Thread 0x00007824cc57e700 (most recent call first):
  File "/opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/ssl.py", line 1163 in read
  File "/opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/ssl.py", line 1307 in recv_into
  File "/opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/socket.py", line 717 in readinto
  File "/opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/http/client.py", line 279 in _read_status
  File "/opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/http/client.py", line 318 in begin
  File "/opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/http/client.py", line 1375 in getresponse
  File "/runner/_work/beam/beam/build/gradleenv/2050596098/lib/python3.10/site-packages/httplib2/__init__.py", line 1399 in _conn_request
  File "/runner/_work/beam/beam/build/gradleenv/2050596098/lib/python3.10/site-packages/httplib2/__init__.py", line 1447 in _request
  File "/runner/_work/beam/beam/build/gradleenv/2050596098/lib/python3.10/site-packages/httplib2/__init__.py", line 1727 in request
  File "/runner/_work/beam/beam/build/gradleenv/2050596098/lib/python3.10/site-packages/google_auth_httplib2.py", line 216 in request
  File "/runner/_work/beam/beam/build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/http_wrapper.py", line 397 in _MakeRequestNoRetry
  File "/runner/_work/beam/beam/build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/http_wrapper.py", line 348 in MakeRequest
  File "/runner/_work/beam/beam/build/gradleenv/2050596098/lib/python3.10/site-packages/apitools/base/py/base_api.py", line 728 in _RunMethod
  File "/runner/_work/beam/beam/sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py", line 554 in List
  File "/runner/_work/beam/beam/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py", line 1050 in list_messages
  File "/runner/_work/beam/beam/sdks/python/apache_beam/utils/retry.py", line 300 in wrapper
  File "/runner/_work/beam/beam/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py", line 185 in poll_for_job_completion
  File "/opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/threading.py", line 953 in run
  File "/opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/threading.py", line 1016 in _bootstrap_inner
  File "/opt/hostedtoolcache/Python/3.10.17/x64/lib/python3.10/threading.py", line 973 in _bootstrap

let's see if this happens again.

tvalentyn · 2026-03-13T18:05:17Z

sdks/python/pyproject.toml

 requires = [
    "setuptools",
    "wheel>=0.36.0",
-    "grpcio-tools==1.62.1; python_version <= '3.12'",


grpcio-tools==1.62.1 pulls in protobuf 4.
grpcio-tools==1.71.0 pulls in protobuf 5.

Stubs generated with protobuf 5 won't work with protobuf 4:

Successfully installed protobuf-4.25.8 (py310b) 10:58:26 ::python$ python Python 3.10.19 (main, Jan 5 2026, 12:29:08) [GCC 14.2.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> from google.protobuf import runtime_version Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: cannot import name 'runtime_version' from 'google.protobuf' (/home/valentyn/.pyenv/versions/py310b/lib/python3.10/site-packages/google/protobuf/__init__.py)

so if any dependency causes installation of protobuf 4, for example, pip install apache-beam[tft], beam stops working.

codecov · 2026-03-13T22:52:29Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 56.87%. Comparing base (8e0736a) to head (08ef814).
⚠️ Report is 19 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #37822      +/-   ##
============================================
+ Coverage     56.86%   56.87%   +0.01%     
  Complexity     3414     3414              
============================================
  Files          1178     1178              
  Lines        187498   187540      +42     
  Branches       3590     3580      -10     
============================================
+ Hits         106623   106670      +47     
- Misses        77480    77485       +5     
+ Partials       3395     3385      -10

Flag	Coverage Δ
python	`80.09% <ø> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions bot added python build labels Mar 11, 2026

github-actions bot added the docker label Mar 13, 2026

tvalentyn changed the title ~~Grpc followup~~ Regenerate beam container image dependencies to include newer version of grpc. Mar 13, 2026

tvalentyn force-pushed the grpc_followup branch from 2679417 to 57f1532 Compare March 13, 2026 01:05

github-actions bot removed the build label Mar 13, 2026

tvalentyn force-pushed the grpc_followup branch 2 times, most recently from 78bdb41 to 2ac1a2f Compare March 13, 2026 13:26

github-actions bot removed the docker label Mar 13, 2026

tvalentyn commented Mar 13, 2026

View reviewed changes

github-actions bot added the docker label Mar 13, 2026

tvalentyn commented Mar 13, 2026

View reviewed changes

tvalentyn force-pushed the grpc_followup branch from b129909 to 0c428da Compare March 13, 2026 20:53

github-actions bot added the build label Mar 13, 2026

tvalentyn added 2 commits March 13, 2026 15:57

Remove branching by Python version.

db0d4ce

Adjust bounds for envoy-data-plane

c06b1e5

tvalentyn force-pushed the grpc_followup branch from 08ef814 to f1d9039 Compare March 13, 2026 22:58

github-actions bot removed the build label Mar 13, 2026

Adjust dependencies.

e2faa0f

tvalentyn force-pushed the grpc_followup branch from f1d9039 to f4dfaef Compare March 13, 2026 23:30

github-actions bot removed the docker label Mar 13, 2026

tvalentyn force-pushed the grpc_followup branch from f4dfaef to 1162875 Compare March 14, 2026 00:39

Tune TFX deps more.

bc3c190

tvalentyn force-pushed the grpc_followup branch from 1162875 to bc3c190 Compare March 14, 2026 00:56

Regenerate dependencies.

95e0011

github-actions bot added the docker label Mar 14, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Regenerate beam container image dependencies to include newer version of grpc. #37822

Regenerate beam container image dependencies to include newer version of grpc. #37822
tvalentyn wants to merge 5 commits intoapache:masterfrom
tvalentyn:grpc_followup

tvalentyn commented Mar 11, 2026 •

edited

Loading

Uh oh!

gemini-code-assist bot commented Mar 11, 2026

Uh oh!

tvalentyn Mar 13, 2026 •

edited

Loading

Uh oh!

tvalentyn Mar 13, 2026

Uh oh!

tvalentyn Mar 13, 2026

Uh oh!

tvalentyn Mar 13, 2026

Uh oh!

tvalentyn commented Mar 13, 2026

Uh oh!

tvalentyn Mar 13, 2026

Uh oh!

codecov bot commented Mar 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

tvalentyn commented Mar 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

GitHub Actions Tests Status (on master branch)

Uh oh!

gemini-code-assist bot commented Mar 11, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

tvalentyn Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tvalentyn Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

tvalentyn Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

tvalentyn Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

tvalentyn commented Mar 13, 2026

Uh oh!

tvalentyn Mar 13, 2026

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Mar 13, 2026

Codecov Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

tvalentyn commented Mar 11, 2026 •

edited

Loading

tvalentyn Mar 13, 2026 •

edited

Loading