Fix flink runner test by liferoad · Pull Request #34913 · apache/beam

liferoad · 2025-05-12T01:36:03Z

Tests:

./gradlew :sdks:python:test-suites:portable:py39:flinkCompatibilityMatrixDOCKER
./gradlew :sdks:python:test-suites:portable:py39:flinkCompatibilityMatrixLOOPBACK
./gradlew :sdks:python:test-suites:portable:py39:flinkCompatibilityMatrixPROCESS

DOCKER and LOOPBACK are fine but PROCESS has this issue:
java.lang.IllegalStateException: Process died with exit code 2: This is the most common error, affecting a majority of the failed tests across FlinkRunnerTest, FlinkRunnerTestOptimized, and FlinkRunnerTestStreaming. It indicates that the Flink process executing the pipeline is terminating unexpectedly. Examples:

test_assert_that
test_batch_pardo
test_create
test_group_by_key
test_pardo
test_sdf
Many tests involving state, timers, and windowing.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

github-actions · 2025-07-11T12:46:44Z

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.

github-actions · 2025-07-18T12:47:10Z

This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

liferoad · 2025-07-19T20:56:32Z

./gradlew :sdks:python:test-suites:portable:py39:flinkCompatibilityMatrixDOCKER

=========================== short test summary info ============================
FAILED apache_beam/runners/portability/flink_runner_test.py::FlinkRunnerTestStreaming::test_batch_pardo_fusion_break
FAILED apache_beam/runners/portability/flink_runner_test.py::FlinkRunnerTestStreaming::test_batch_to_element_pardo
FAILED apache_beam/runners/portability/flink_runner_test.py::FlinkRunnerTestStreaming::test_expand_kafka_read
FAILED apache_beam/runners/portability/flink_runner_test.py::FlinkRunnerTestStreaming::test_flattened_side_input
FAILED apache_beam/runners/portability/flink_runner_test.py::FlinkRunnerTestStreaming::test_gbk_side_input
FAILED apache_beam/runners/portability/flink_runner_test.py::FlinkRunnerTestStreaming::test_multimap_multiside_input
FAILED apache_beam/runners/portability/flink_runner_test.py::FlinkRunnerTestStreaming::test_multimap_side_input
FAILED apache_beam/runners/portability/flink_runner_test.py::FlinkRunnerTestStreaming::test_multimap_side_input_type_coercion
FAILED apache_beam/runners/portability/flink_runner_test.py::FlinkRunnerTestStreaming::test_pack_combiners
FAILED apache_beam/runners/portability/flink_runner_test.py::FlinkRunnerTestStreaming::test_pardo_side_input_dependencies
FAILED apache_beam/runners/portability/flink_runner_test.py::FlinkRunnerTestStreaming::test_pardo_side_inputs
FAILED apache_beam/runners/portability/flink_runner_test.py::FlinkRunnerTestStreaming::test_pardo_unfusable_side_inputs
FAILED apache_beam/runners/portability/flink_runner_test.py::FlinkRunnerTestStreaming::test_pardo_unfusable_side_inputs_with_separation
FAILED apache_beam/runners/portability/flink_runner_test.py::FlinkRunnerTestStreaming::test_pardo_windowed_side_inputs
FAILED apache_beam/runners/portability/flink_runner_test.py::FlinkRunnerTestStreaming::test_sdf_synthetic_source
=========== 15 failed, 164 passed, 55 skipped in 2829.80s (0:47:09) ============

liferoad · 2025-07-20T16:14:52Z

./gradlew :sdks:python:test-suites:portable:py39:flinkCompatibilityMatrixLOOPBACK: BUILD SUCCESSFUL in 18m 19s

liferoad · 2025-07-20T17:14:14Z

./gradlew :sdks:python:test-suites:portable:py39:flinkCompatibilityMatrixDOCKER: ================= 179 passed, 55 skipped in 2699.45s (0:44:59) =================

…VM options chore(common.gradle): improve sdk worker script for better execution

The import was not used in the test file and was removed to clean up the code

Add new GitHub workflow to run postcommit tests for Python Portable Flink. Includes trigger file and workflow configuration with scheduled runs, pull request triggers, and artifact uploads for test results.

github-actions · 2025-07-21T03:37:44Z

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

…atest published version

codecov · 2025-07-21T20:20:40Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 56.54%. Comparing base (8eba198) to head (29c0dd7).
Report is 12 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##             master   #34913    +/-   ##
==========================================
  Coverage     56.54%   56.54%            
  Complexity     3378     3378            
==========================================
  Files          1205     1205            
  Lines        183660   183844   +184     
  Branches       3475     3475            
==========================================
+ Hits         103847   103952   +105     
- Misses        76495    76574    +79     
  Partials       3318     3318

Flag	Coverage Δ
python	`80.66% <ø> (-0.06%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2025-07-21T22:08:04Z

Assigning reviewers:

R: @jrmccluskey for label python.
R: @damccorm for label build.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

damccorm

Thanks, had a couple of minor comments, but feel free to go ahead and merge when addressed.

.github/workflows/beam_PostCommit_Python_Portable_Flink.yml

Co-authored-by: Danny McCormick <dannymccormick@google.com>

liferoad · 2025-07-24T23:12:34Z

Thanks, had a couple of minor comments, but feel free to go ahead and merge when addressed.

Got it. FYI. The workflow probably won't succeed with flinkCompatibilityMatrixPROCESS. I plan to merge it first and then debug since I have some troubles to reliably run flinkCompatibilityMatrixPROCESS on my machine.

gemini-code-assist · 2025-07-25T15:45:20Z

Important

Installation incomplete: to start using Gemini Code Assist, please ask the organization owner(s) to visit the Gemini Code Assist Admin Console and sign the Terms of Services.

gemini-code-assist · 2025-07-25T17:19:22Z

Important

Installation incomplete: to start using Gemini Code Assist, please ask the organization owner(s) to visit the Gemini Code Assist Admin Console and sign the Terms of Services.

liferoad added 3 commits May 11, 2025 21:36

added the jvm container to support run in docker

a35c435

skip test_batch_rebatch_pardos with docker

0040455

skip some tests

ea5229f

github-actions bot added python runners labels May 12, 2025

github-actions bot added the stale label Jul 11, 2025

github-actions bot closed this Jul 18, 2025

liferoad reopened this Jul 18, 2025

github-actions bot removed the stale label Jul 19, 2025

Merge branch 'master' into fix-flink-runner-test

ef40da2

liferoad and others added 3 commits July 20, 2025 19:46

fix(flink_runner_test): update Flink job server jar version and add J…

c3a5a64

…VM options chore(common.gradle): improve sdk worker script for better execution

refactor: remove unused FlinkRunnerOptions import

769795c

The import was not used in the test file and was removed to clean up the code

ci(workflows): add postcommit workflow for python portable flink tests

ed52a2e

Add new GitHub workflow to run postcommit tests for Python Portable Flink. Includes trigger file and workflow configuration with scheduled runs, pull request triggers, and artifact uploads for test results.

liferoad marked this pull request as ready for review July 21, 2025 01:27

github-actions bot added the build label Jul 21, 2025

liferoad added 2 commits July 21, 2025 18:49

fix(flink_runner_test): update Flink job server jar path to use the l…

b497f34

…atest published version

fix format

29c0dd7

github-actions bot added the Next Action: Reviewers label Jul 21, 2025

damccorm approved these changes Jul 22, 2025

View reviewed changes

.github/workflows/beam_PostCommit_Python_Portable_Flink.yml Show resolved Hide resolved

.github/workflows/beam_PostCommit_Python_Portable_Flink.yml Outdated Show resolved Hide resolved

Update .github/workflows/beam_PostCommit_Python_Portable_Flink.yml

1bfe746

Co-authored-by: Danny McCormick <dannymccormick@google.com>

liferoad merged commit 80f1052 into apache:master Jul 25, 2025
96 of 97 checks passed

liferoad mentioned this pull request Jul 26, 2025

[Failing Test]: :sdks:python:test-suites:portable:py39:flinkCompatibilityMatrixPROCESS falis #35702

Closed

17 tasks

Conversation

liferoad commented May 12, 2025

GitHub Actions Tests Status (on master branch)

Uh oh!

github-actions bot commented Jul 11, 2025

Uh oh!

github-actions bot commented Jul 18, 2025

Uh oh!

liferoad commented Jul 19, 2025

Uh oh!

liferoad commented Jul 20, 2025

Uh oh!

liferoad commented Jul 20, 2025

Uh oh!

github-actions bot commented Jul 21, 2025

Uh oh!

codecov bot commented Jul 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Jul 21, 2025

Uh oh!

damccorm left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

liferoad commented Jul 24, 2025

Uh oh!

gemini-code-assist bot commented Jul 25, 2025

Uh oh!

Uh oh!

gemini-code-assist bot commented Jul 25, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

codecov bot commented Jul 21, 2025 •

edited

Loading