[YAML] - Jinja % include example by derrickaw · Pull Request #35914 · apache/beam

derrickaw · 2025-08-20T00:48:27Z

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
Update CHANGES.md with noteworthy changes.
If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

derrickaw · 2025-08-20T13:44:32Z

Run Python PreCommit 3.10

derrickaw · 2025-08-20T13:44:36Z

Run Python PreCommit 3.11

derrickaw · 2025-08-20T15:17:20Z

Run Python PreCommit 3.11

derrickaw · 2025-08-20T16:36:08Z

assign set of reviewers

github-actions · 2025-08-20T16:37:16Z

Assigning reviewers:

R: @liferoad for label python.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

liferoad · 2025-08-20T19:00:04Z

Love the jinia idea for include. :)
We can add more examples more explicitly here later if you want: https://beam.apache.org/documentation/sdks/yaml/#jinja-templatization

derrickaw · 2025-08-21T18:20:13Z

Run Python_Transforms PreCommit 3.10

codecov · 2025-08-21T18:39:21Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 56.67%. Comparing base (f59a45e) to head (ffcda88).
⚠️ Report is 14 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff            @@
##             master   #35914   +/-   ##
=========================================
  Coverage     56.67%   56.67%           
  Complexity     3380     3380           
=========================================
  Files          1219     1219           
  Lines        184561   184565    +4     
  Branches       3507     3507           
=========================================
+ Hits         104595   104605   +10     
+ Misses        76639    76633    -6     
  Partials       3327     3327

Flag	Coverage Δ
python	`80.82% <100.00%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

derrickaw · 2025-08-21T19:52:34Z

Run Python PreCommit 3.12

damccorm

Thanks for putting this together!

damccorm · 2025-08-21T20:09:46Z

sdks/python/apache_beam/yaml/examples/README.md

+- [wordCount.yaml](#TODO: pending)
+
+% include:
+- [wordCount.yaml](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/yaml/examples/transforms/jinja/include/wordCount.yaml)


Should this be wordCountInclude.yaml?

It would also be good to give an example of invoking this (and how you would pass in appropriate parameters)

I had it here before, but I moved it to the other Readme. Will update name.

Similar pattern that Charles used on some of his ML work with Readme usage more in the applicable folder and not in the main one since multiple folders will be under Jinja.

Similar pattern that Charles used on some of his ML work with Readme usage more in the applicable folder and not in the main one since multiple folders will be under Jinja.

I missed that Readme entirely when I did my review. SGTM

Maybe we could just link to that folder or the Readme here then? I worry that it will be easy to miss if you just follow the link (basically how I reviewed this)

Will do. Thanks

damccorm · 2025-08-21T20:10:50Z

sdks/python/apache_beam/yaml/examples/README.md

+inputs from the user through `% include`, `% import`, and inheritance
+directives.
+
+% import:


Should these be code blocks? Also looks like some pieces are still missing, we should probably not include them for now

It was more of a heading and not usable code. Will try to rethink this.

Updated, thanks

damccorm · 2025-08-21T20:13:28Z

sdks/python/apache_beam/yaml/examples/transforms/jinja/include/wordCountInclude.yaml

+    - name: Read from GCS
+      type: ReadFromText
+      config:
+{% include 'apache_beam/yaml/examples/transforms/jinja/include/submodules/readFromText.yaml' %}


Rather than just including the config, could we maybe map in whole transforms for simple ones like this?

Since this is an example, we should also explain what we're doing in comments

Practically, I don't think it would be common to include just the config like this, especially if it is very simple (and also parameterized)

I'd also recommend using more descriptive file names. So for examples:

readFromTextTransform.yaml (assuming we put the whole transform there)

or

mapToFieldsCountConfig.yaml (assuming we keep that one as config since it is more complicated)

We generally don't want to put the whole transform here because it limits us from changing names or inputs. Wanted to give an example of how to use. How about a mix? Some transforms, some configs only?

Ok on better names.

We generally don't want to put the whole transform here because it limits us from changing names or inputs.

I don't follow - could you give an example of someething we'd want to change but wouldn't be able to?

How about a mix? Some transforms, some configs only?

This SGTM - I think the more complicated configs make sense, the one liners seem less useful

I don't follow - could you give an example of someething we'd want to change but wouldn't be able to?

Maybe a mute point now with both options, but if we have the whole transform in the include yaml. We can't change the name or input paths etc unless we pass even more stuff through the jinjia variables. So the option I went with was to give an example where a user could control or orient the pipeline graph from the main yaml file and then include what configs they needed from a library. When we push everything to a library, we lose that control, but maybe that is fine.

Updated, thanks.

liferoad · 2025-08-21T21:47:37Z

/gemini review

liferoad · 2025-08-21T21:48:27Z

I am trying to test how good Gemini knows our YAML. The review comments could be skipped. :)

gemini-code-assist

Code Review

This pull request introduces an example for using Jinja's % include directive in Beam YAML pipelines. The changes include new example files, corresponding tests, and documentation updates. While the core functionality is a valuable addition, there are several issues that need to be addressed. These include broken commands and incorrect links in the documentation, fragile test logic, and potential blockers in the test data, such as a reference to a missing file. Addressing these points will improve the quality and usability of the new example.

sdks/python/apache_beam/yaml/examples/testing/input_data.py

sdks/python/apache_beam/yaml/examples/testing/examples_test.py

sdks/python/apache_beam/yaml/examples/transforms/jinja/include/README.md

.../python/apache_beam/yaml/examples/transforms/jinja/include/submodules/mapToFields_split.yaml

sdks/python/apache_beam/yaml/examples/README.md

sdks/python/apache_beam/yaml/examples/testing/examples_test.py

sdks/python/apache_beam/yaml/examples/testing/input_data.py

sdks/python/apache_beam/yaml/examples/transforms/jinja/include/README.md

derrickaw · 2025-08-22T03:03:21Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces a new example demonstrating the use of Jinja's % include directive in Beam YAML pipelines. The changes include the new example files, corresponding tests, and updates to documentation. My review focuses on improving code clarity, fixing a bug in the documentation, and enhancing the maintainability of the new test code and YAML examples. Overall, this is a great addition that showcases a powerful feature of Beam YAML.

sdks/python/apache_beam/yaml/examples/transforms/jinja/include/README.md

sdks/python/apache_beam/yaml/examples/testing/examples_test.py

sdks/python/apache_beam/yaml/examples/testing/input_data.py

sdks/python/apache_beam/yaml/examples/transforms/jinja/include/submodules/combineTransform.yaml

sdks/python/apache_beam/yaml/examples/transforms/jinja/include/wordCountInclude.yaml

derrickaw · 2025-08-22T13:36:13Z

Run Prism_Python PreCommit 3.12

derrickaw · 2025-08-22T13:38:49Z

Run Python_Integration PreCommit 3.9

derrickaw · 2025-08-22T13:42:28Z

Run Python_Transforms PreCommit 3.10

derrickaw · 2025-08-22T13:43:18Z

Run Python_ML PreCommit 3.10

damccorm

This LGTM. There are failing precommits which look to me like flakes, I'm going to try to run them again and see if they pass before merging

derrickaw added 4 commits August 20, 2025 00:43

update readme for jinja

531590e

fix conflicts

24e4b72

add jinja data

04007e8

add jinja submodules

83d8ab9

github-actions bot added python yaml labels Aug 20, 2025

derrickaw added 2 commits August 20, 2025 01:57

fix lint, whitespace, and save file issues

8019df2

fix rebase conflict and exception etc

c289db9

derrickaw marked this pull request as ready for review August 20, 2025 16:18

github-actions bot added the Next Action: Reviewers label Aug 20, 2025

liferoad requested review from chamikaramj and damccorm August 20, 2025 18:58

derrickaw changed the title ~~[YAML] - Jinja example~~ [YAML] - Jinja % include example Aug 21, 2025

derrickaw added 6 commits August 21, 2025 15:30

update readme

e366785

move around files

6b46e95

add new readme

8a7994c

move more files around

f966c0e

fix whitespace

bbdbec3

fix additional errors after shifting folders

d741ff8

liferoad approved these changes Aug 21, 2025

View reviewed changes

damccorm reviewed Aug 21, 2025

View reviewed changes

gemini-code-assist bot reviewed Aug 21, 2025

View reviewed changes

derrickaw added 5 commits August 21, 2025 23:37

address comments

819b9dd

fix jinja processor

e3a4960

fix readme per new names for submodules

53e567f

add todo ids

cd18b24

add comments

2978634

gemini-code-assist bot reviewed Aug 22, 2025

View reviewed changes

minor gemini fixes

ffcda88

derrickaw requested review from damccorm and liferoad August 22, 2025 13:45

damccorm approved these changes Aug 22, 2025

View reviewed changes

damccorm merged commit 90f8c50 into apache:master Aug 22, 2025
130 of 142 checks passed

derrickaw deleted the jinja_example branch August 22, 2025 15:35

Conversation

derrickaw commented Aug 20, 2025

GitHub Actions Tests Status (on master branch)

Uh oh!

derrickaw commented Aug 20, 2025

Uh oh!

derrickaw commented Aug 20, 2025

Uh oh!

derrickaw commented Aug 20, 2025

Uh oh!

derrickaw commented Aug 20, 2025

Uh oh!

github-actions bot commented Aug 20, 2025

Uh oh!

liferoad commented Aug 20, 2025

Uh oh!

derrickaw commented Aug 21, 2025

Uh oh!

codecov bot commented Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

derrickaw commented Aug 21, 2025

Uh oh!

damccorm left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

derrickaw Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

derrickaw Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

liferoad commented Aug 21, 2025

Uh oh!

liferoad commented Aug 21, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

codecov bot commented Aug 21, 2025 •

edited

Loading

derrickaw Aug 21, 2025 •

edited

Loading

derrickaw Aug 21, 2025 •

edited

Loading