Skip to content

Refactor: separate SplitAndSizeRestriction from FnApiDoFnRunner#34919

Merged
kennknowles merged 2 commits intoapache:masterfrom
kennknowles:FnApiDoFnRunner-SplitAndSizeRestrictions
May 21, 2025
Merged

Refactor: separate SplitAndSizeRestriction from FnApiDoFnRunner#34919
kennknowles merged 2 commits intoapache:masterfrom
kennknowles:FnApiDoFnRunner-SplitAndSizeRestrictions

Conversation

@kennknowles
Copy link
Member

This builds on #34883 and #34907 which are steps 1 and 2 of untangling FnApiDoFnRunner. This splits off the still-very-simple transform that calls SplitRestriction on each restriction and SizeRestriction on each of the sub-parts that is emitted.


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

Build python source distribution and wheels
Python tests
Java tests
Go tests

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

@github-actions github-actions bot added the java label May 12, 2025
@kennknowles kennknowles force-pushed the FnApiDoFnRunner-SplitAndSizeRestrictions branch 4 times, most recently from 96111bc to 544e5c0 Compare May 14, 2025 17:53
@kennknowles kennknowles marked this pull request as ready for review May 14, 2025 17:57
@kennknowles kennknowles requested a review from scwhittle May 14, 2025 17:57
@github-actions
Copy link
Contributor

Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment assign set of reviewers

@kennknowles kennknowles force-pushed the FnApiDoFnRunner-SplitAndSizeRestrictions branch 2 times, most recently from 939637a to f6d9878 Compare May 15, 2025 17:50
@kennknowles kennknowles force-pushed the FnApiDoFnRunner-SplitAndSizeRestrictions branch from f6d9878 to 712d29d Compare May 15, 2025 18:54
@kennknowles kennknowles force-pushed the FnApiDoFnRunner-SplitAndSizeRestrictions branch 3 times, most recently from 138b7b2 to 82555e7 Compare May 16, 2025 15:15
@github-actions
Copy link
Contributor

Assigning reviewers:

R: @m-trieu for label java.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

Copy link
Contributor

@scwhittle scwhittle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, with some small cleanups

PaneInfo.NO_FIRING);
}

@Test
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: move down so grouped with other tests?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

@kennknowles kennknowles force-pushed the FnApiDoFnRunner-SplitAndSizeRestrictions branch from 82555e7 to 9765f7c Compare May 19, 2025 16:23
@kennknowles
Copy link
Member Author

Thanks for bearing with me and paying close attention to this! I know the diffs are not particularly helpful in seeing what the real change is. I do think the positive delta is almost all boilerplate and things are shrinking in spirit and cyclomatic complexity is reducing. Just a couple more and it'll be more like straight line code based on the URN, which should lend itself to (1) pulling out more shared and meaningful abstractions and (2) optimizing and further simplifying specific paths by finding more stuff that was in there just because it was hard to tell which operation we were performing.

This is a further step in untangling FnApiDoFnRunner which
implements a variety of loosely-related transforms, which
mostly have in common that they call a user's DoFn but otherwise
are disjoint.
@kennknowles kennknowles force-pushed the FnApiDoFnRunner-SplitAndSizeRestrictions branch from 9765f7c to 3b09453 Compare May 19, 2025 18:32
@scwhittle
Copy link
Contributor

Thanks for bearing with me and paying close attention to this!

No problem, thanks for making this more maintainable. I think that the fact that I added the currentTrackerClaimed creation to this method when it was part of the huge file shows how difficult it was to see what exactly was going on there :)

@kennknowles kennknowles merged commit 060cc1a into apache:master May 21, 2025
27 checks passed
@kennknowles kennknowles deleted the FnApiDoFnRunner-SplitAndSizeRestrictions branch May 21, 2025 14:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants