BEAM-9626: Move out optional packages from core dependencies - 20259

[rakeshcusat]

More information could be found here:

BEAM-9626
#20259

Once this change is merged, the Pymongo could be installed with this command:

pip install apache_beam[mongodb,hdfs,redis]

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[rakeshcusat]

PTAL @damccorm

I also feel that we should update the CHANGES.md file to mention this change.

[github-actions]

Assigning reviewers:

R: @liferoad for label python.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

• stop reviewer notifications - opt out of the automated review tooling
• remind me after tests pass - tag the comment author after tests pass
• waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

[damccorm]

Could we mention that the others have been moved to separate (existing) extras as well?

[rakeshcusat]

Good point.

[damccorm]

Looks like some of these are causing failures because the appropriate extras aren't installed when running the test - could you take a look? (or feel free to revert/simplify to a smaller set of dependencies)

[damccorm]

I see you started on this, I think its probably just adding interactive/yaml to a few tox suites

[rakeshcusat]

yes, I am not done with this change yet. I will update the tox suites soon.

[rakeshcusat]

@damccorm what is the easiest way to find the mapping of test environment and the tox.ini section? This could help me to easily identify the broken environment and fix it. Otherwise, it will take longer time.

Also, I realized that jsonschema is part of core and can't be moved to extra section of yaml.

[rakeshcusat]

would this line (Task :sdks:python:test-suites:tox:py312:testPython312 ) in the log good enough to identify the environment? or would you choose a different strategy to identify the environment?

[damccorm]

would this line (Task :sdks:python:test-suites:tox:py312:testPython312 ) in the log good enough to identify the environment? or would you choose a different strategy to identify the environment?

Yeah, that's what I usually use, and then you can look at the gradle task. An alternative at the failed test log, you can see something like:

________________________ YamlMappingTest.test_validate _________________________
[gw1] linux -- Python 3.9.22 /runner/_work/beam/beam/sdks/python/test-suites/tox/py39/build/srcs/sdks/python/target/.tox-py39-cloud/py39-cloud/bin/python
...

Tox will create a folder with the task name (in this case py39-cloud) as part of the path name. So in this case, the task is py39-cloud -

beam/sdks/python/tox.ini

Line 99 in 9d75411

[testenv:py{39,310,311,312,313}-cloud]

You'll also see this task show up elsewhere in the logs (e.g. tox-py39-cloud shows up a bunch of places). An llm probably could figure it out too, but I haven't tried it yet :)

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 54.51%. Comparing base (cecfa61) to head (ccc51b0).
⚠️ Report is 468 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##             master   #35297    +/-   ##
==========================================
  Coverage     54.50%   54.51%            
  Complexity     1559     1559            
==========================================
  Files          1035     1036     +1     
  Lines        161595   161757   +162     
  Branches       1139     1139            
==========================================
+ Hits          88084    88175    +91     
- Misses        71380    71451    +71     
  Partials       2131     2131            

Flag	Coverage Δ
python	80.83% <100.00%> (-0.06%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

Reminder, please take a look at this pr: @liferoad

[damccorm]

waiting on author

[github-actions]

This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

[github-actions]

This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.