Add an ExtractWindowingInfo transform.#34051
Conversation
|
Assigning reviewers. If you would like to opt out of this review, comment R: @damccorm for label python. Available commands:
The PR bot will only process comments in the main thread (not review comments). |
damccorm
left a comment
There was a problem hiding this comment.
Thanks! Feedback is minor (string fixes), otherwise LGTM.
Please fix linting as well before merging
|
|
||
|
|
||
| _WINDOWING_INFO_TYPES = { | ||
| 'timestamp': Timestamp, |
There was a problem hiding this comment.
(optional, not sure if I like this more) - thoughts on event_timestamp? It might be a little more explicit.
We could even allow a user to specify processing_timestamp (resolves to time.time()) as a convenience.
There was a problem hiding this comment.
Interesting idea.
I'm going to leave it as timestamp. Event timestamps are actually attached to the element, and windows are always in terms of this time. It's also a bit deceptive because if I try to inspect/reify processing time, that's not what will be used when anything else tries to act on it. (In addition, one needs triggers/timers to get into processing time and that's a whole can of worms we don't expose to YAML.)
Co-authored-by: Danny McCormick <dannymccormick@google.com>
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #34051 +/- ##
============================================
+ Coverage 57.63% 57.69% +0.06%
Complexity 1474 1474
============================================
Files 989 993 +4
Lines 157126 157354 +228
Branches 1076 1076
============================================
+ Hits 90564 90790 +226
- Misses 64320 64322 +2
Partials 2242 2242
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
YAML users don't have access to DoFn params or annotations, so we provide this as an explicit transform.
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.