Skip to content
Merged
Show file tree
Hide file tree
Changes from 24 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
f1d9e93
Log Warning if process function return None
Sep 24, 2024
3cdb629
Fix get function without inner
Sep 25, 2024
9170808
check the first def_line also
Sep 25, 2024
9370ed2
rename variable
Sep 25, 2024
6219664
add strip function
Sep 25, 2024
5a3c1a6
reformat function
Sep 25, 2024
3bfc43d
refactor code
Sep 25, 2024
278d186
fix bug in get function body
Sep 29, 2024
f07d2cd
Merge branch 'master' into master
DKER2 Sep 29, 2024
e819366
retrigger test
Sep 29, 2024
25d5431
retrigger test
Sep 29, 2024
52f59ff
fix: unexpected error when transform two pcoll
DKER2 Aug 17, 2025
69ed085
revert redundant
DKER2 Aug 17, 2025
e6b636c
Merge branch 'master' into fx-30445
DKER2 Aug 17, 2025
d194838
fix test
DKER2 Aug 20, 2025
4acae69
reformat file
DKER2 Aug 20, 2025
12c4973
simply change test case
DKER2 Aug 20, 2025
d5f052c
change test case
DKER2 Aug 20, 2025
9193339
change test case
DKER2 Aug 20, 2025
f1e5fd7
retrigger test
DKER2 Aug 23, 2025
faa4fd0
Merge branch 'master' into fx-30445
DKER2 Aug 23, 2025
f04d6c7
Merge branch 'master' into fx-30445
DKER2 Sep 4, 2025
d3ceae7
update change.md
DKER2 Sep 4, 2025
a966505
update change.md
DKER2 Sep 4, 2025
4338599
Merge branch 'master' into fx-30445
DKER2 Sep 21, 2025
06d0bda
update change.md
DKER2 Sep 21, 2025
5364c52
update change.md
DKER2 Sep 21, 2025
4539d04
update format of change.md
DKER2 Sep 21, 2025
422b3f7
update format of change.md
DKER2 Sep 21, 2025
e18b6be
update format of change.md
DKER2 Sep 21, 2025
c36572e
update format of change.md
DKER2 Sep 21, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGES.md
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,7 @@
## Bugfixes

* (Python) Fixed Java YAML provider fails on Windows ([#35617](https://github.com/apache/beam/issues/35617)).
* (Python) Fixed transform naming conflict when executing DataTransform on a dictionary of PColls ([#30445][https://github.com/apache/beam/issues/30445])
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, missed this PR - would you mind specifically calling out that this could break update compatibility

Suggested change
* (Python) Fixed transform naming conflict when executing DataTransform on a dictionary of PColls ([#30445][https://github.com/apache/beam/issues/30445])
* (Python) Fixed transform naming conflict when executing DataTransform on a dictionary of PColls ([#30445][https://github.com/apache/beam/issues/30445]). This may break update compatibility if you don't provide a `--transform_name_mapping`.

Also, could you move it to 2.69.0's breaking changes section?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @damccorm , done. I dont think build error is related to this PR

* Fixed BigQueryIO creating temporary datasets in wrong project when temp_dataset is specified with a different project than the pipeline project. For some jobs, temporary datasets will now be created in the correct project (Python) ([#35813](https://github.com/apache/beam/issues/35813)).
* (Go) Fix duplicates due to reads after blind writes to Bag State ([#35869](https://github.com/apache/beam/issues/35869)).
* Earlier Go SDK versions can avoid the issue by not reading in the same call after a blind write.
Expand Down
2 changes: 1 addition & 1 deletion sdks/python/apache_beam/dataframe/transforms.py
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ def expand(self, input_pcolls):
for tag in input_dict
}
input_frames: dict[Any, frame_base.DeferredFrame] = {
k: convert.to_dataframe(pc, proxies[k])
k: convert.to_dataframe(pc, proxies[k], str(k))
for k, pc in input_dict.items()
} # noqa: F821

Expand Down
20 changes: 20 additions & 0 deletions sdks/python/apache_beam/dataframe/transforms_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -317,6 +317,26 @@ def check(actual):
lambda x: {'res': 3 * x}, proxy, yield_elements='pandas')
assert_that(res['res'], equal_to_series(three_series), 'CheckDictOut')

def test_multiple_dataframes_transforms(self):
expected_output = ["Bryan", "DKER2"]

def transform_func(a, b):
b["name"] = "DKER2"
return a, b

with beam.Pipeline() as p:
pcol1 = p | "Create1" >> beam.Create([beam.Row(name="Bryan")])
pcol2 = p | "Create2" >> beam.Create([beam.Row(name="common")])

result = ({
"a": pcol1, "b": pcol2
}
|
"TransformDF" >> transforms.DataframeTransform(transform_func)
| "Flatten" >> beam.Flatten()
| transforms.DataframeTransform(lambda df: df.name))
assert_that(result, equal_to(expected_output))

def test_cat(self):
# verify that cat works with a List[Series] since this is
# missing from doctests
Expand Down
Loading