Fix the error for directory separator #34390

liferoad · 2025-03-22T22:14:08Z

#34364 reverted the PR.

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

github-actions · 2025-03-22T23:34:54Z

Assigning reviewers. If you would like to opt out of this review, comment assign to next reviewer:

R: @shunping for label python.
R: @chamikaramj for label io.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

codecov · 2025-03-23T01:09:11Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 56.37%. Comparing base (3595a33) to head (ddbb675).

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #34390      +/-   ##
============================================
- Coverage     56.37%   56.37%   -0.01%     
  Complexity     3285     3285              
============================================
  Files          1174     1174              
  Lines        179353   179362       +9     
  Branches       3398     3398              
============================================
+ Hits         101119   101122       +3     
- Misses        74974    74980       +6     
  Partials       3260     3260

Flag	Coverage Δ
python	`81.30% <100.00%> (-0.01%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

kennknowles · 2025-03-26T17:56:46Z

sdks/python/apache_beam/io/textio_test.py

+        "when calling ReadFromText")
+
+    with unittest.mock.patch('os.path.dirname') as mock_dirname:
+      mock_dirname.side_effect = TypeError(expected_error_message)


Is it possible to actually cause this error, instead? Mocking makes the test possibly masking things if the presumed behavior of os.path.dirname is not quite right, or if it changes.

The code changes are quite clear in sdks/python/apache_beam/io/textio.py. We mock TypeError and then this causes the OS error since the test file does not exist. This mainly tests TypeError is captured and passes.

kennknowles · 2025-04-04T16:37:35Z

sdks/python/apache_beam/io/textio.py

+      try:
+        if not os.path.dirname(file_pattern):
+          file_pattern = os.path.join('.', file_pattern)
+      except TypeError:


Seems like a very broad exception catch. What is the failure mode that you are handling?

file_pattern coud be passed in as other types, for example, ValueProvider.

added virtualenv-clone

* Create a varint32 coder and used it for RowCoder * Fix checks * Handle a race when multiple process downloading jar at the same time

This will allow other users to replace the default sql provider used by 'Join' and 'Combine-sql'.

…t stream to prevent precondition exception on reuse (apache#34418)

…#34416) Bumps [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) from 1.17.62 to 1.17.67. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json) - [Commits](aws/aws-sdk-go-v2@credentials/v1.17.62...feature/s3/manager/v1.17.67) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

…ache#34413)

…ormance metrics (apache#34361)

Bumps [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2) from 1.17.63 to 1.17.64. - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/changelog-template.json) - [Commits](aws/aws-sdk-go-v2@credentials/v1.17.63...credentials/v1.17.64) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2/credentials dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

Co-authored-by: Naireen <[email protected]>

See: apache#21298

liferoad added 4 commits March 16, 2025 21:12

Support directory separator for Python local filesystem

52b380f

fixed the tests

d2eac7a

Merge branch 'master' into directory-separator

f671d1b

ignore TypeError

54b60de

github-actions bot added python io labels Mar 22, 2025

github-actions bot added the Next Action: Reviewers label Mar 22, 2025

fix lint

ddbb675

kennknowles reviewed Mar 26, 2025

View reviewed changes

minor fix

6599655

kennknowles reviewed Apr 4, 2025

View reviewed changes

damccorm and others added 17 commits April 4, 2025 19:41

Enforce ordering to avoid svn conflicts (apache#34405)

3b14a15

Update build.gradle (apache#34401)

d2e06a1

added virtualenv-clone

Create a varint32 coder and used it for RowCoder (apache#34354)

65807ca

* Create a varint32 coder and used it for RowCoder * Fix checks * Handle a race when multiple process downloading jar at the same time

Factor out a get_default_sql_provider() function.

2a58f05

This will allow other users to replace the default sql provider used by 'Join' and 'Combine-sql'.

Add warning comment so that folks don't change this

3005046

Add to CHANGES.md follow up negative int32 fix (apache#34408)

10a7b58

[Dataflow Streaming] Improve exception handling in PubsubSink to rese…

1310864

…t stream to prevent precondition exception on reuse (apache#34418)

Add docs for yaml dependencies. (apache#34345)

aec38c8

track GetWork from WindmillWorker to UserWorker in proxyless path (ap…

b622914

…ache#34413)

fix: python ref in guide for combining pcol single value (apache#34282)

13421c6

Update dataflow_cost_benchmark. Update website: add ML pipelines perf…

5e7af20

…ormance metrics (apache#34361)

SnowflakeIO: be consistent with backslash escape char

dc20ad4

tolerate integer overflow when encoding varint32 (apache#34439)

87cdb31

remove dependency of core-runners in kafka:io (apache#34442)

4213a02

Co-authored-by: Naireen <[email protected]>

Use list instead of set to make type hint serialization deterministic

6afe60e

See: apache#21298

github-actions bot added go build website infra docker extensions runners dataflow gcp jdbc kafka core spanner yaml labels Apr 4, 2025

liferoad closed this Apr 4, 2025

github-actions bot removed java go build website infra docker extensions runners dataflow gcp jdbc kafka core spanner yaml labels Apr 4, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fix the error for directory separator #34390

Fix the error for directory separator #34390

Uh oh!

liferoad commented Mar 22, 2025

Uh oh!

github-actions bot commented Mar 22, 2025

Uh oh!

codecov bot commented Mar 23, 2025

Uh oh!

kennknowles Mar 26, 2025 •

edited

Loading

Uh oh!

liferoad Mar 29, 2025

Uh oh!

kennknowles Apr 4, 2025

Uh oh!

liferoad Apr 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

Fix the error for directory separator #34390

Fix the error for directory separator #34390

Uh oh!

Conversation

liferoad commented Mar 22, 2025

GitHub Actions Tests Status (on master branch)

Uh oh!

github-actions bot commented Mar 22, 2025

Uh oh!

codecov bot commented Mar 23, 2025

Codecov Report

Uh oh!

kennknowles Mar 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liferoad Mar 29, 2025

Choose a reason for hiding this comment

Uh oh!

kennknowles Apr 4, 2025

Choose a reason for hiding this comment

Uh oh!

liferoad Apr 4, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

kennknowles Mar 26, 2025 •

edited

Loading