Add support for UDF function transformations in PubSubToRedis template

[wladabutenko]

Adding optional UDF support to the PubSubToRedis pipeline, enabling user-defined transformations before writing to Redis

[gemini-code-assist]

Summary of Changes

Hello @wladabutenko, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the PubSubToRedis Dataflow template by integrating optional User-Defined Function (UDF) support. This allows users to apply custom JavaScript transformations to their Pub/Sub messages before they are persisted to Redis, providing greater flexibility in data processing. The changes also introduce robust error handling and dead-lettering capabilities for both UDF execution and Redis data transformations, ensuring data integrity and easier debugging for failed messages. The template's configuration and documentation have been updated to reflect these new features, alongside the addition of thorough testing.

Highlights

• UDF Support Added: The PubSubToRedis template now supports optional User-Defined Functions (UDFs) written in JavaScript, allowing for custom transformations of messages before they are written to Redis.
• New Configuration Parameters: New template parameters have been introduced to configure UDFs, including javascriptTextTransformGcsPath (GCS path to the UDF file), javascriptTextTransformFunctionName (the function to execute), javascriptTextTransformReloadIntervalMinutes (for dynamic UDF updates), and deadletterTopic (for failed UDF transformations).
• Robust Error Handling: The pipeline now includes enhanced error handling for both UDF transformations and Redis sink operations. Messages that fail UDF processing or Redis transformation can be routed to a specified dead-letter Pub/Sub topic or logged.
• Updated Redis Sink Logic: The internal logic for transforming Pub/Sub messages into Redis String, Hash, Logs, and Streams formats has been refactored into 'safe' transforms, each incorporating its own error handling and dead-lettering mechanism.
• Integration and Unit Tests: Comprehensive integration tests (PubSubToRedisIT.java) and unit tests (PubSubToRedisTest.java) have been added to validate the new UDF functionality and the various Redis sink types, including scenarios with successful, failed, and absent UDFs.
• Metadata and Documentation Updates: The MetadataUtils.java file was updated to correctly handle negative integers and longs in template parameters. The README_Cloud_PubSub_to_Redis.md documentation has been expanded to detail the new UDF features and parameters, along with updated example commands.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[codecov]

Codecov Report

❌ Patch coverage is 15.83012% with 218 lines in your changes missing coverage. Please review.
✅ Project coverage is 50.90%. Comparing base (cb18ecc) to head (97df455).
⚠️ Report is 15 commits behind head on main.

Files with missing lines	Patch %	Lines
...gle/cloud/teleport/v2/templates/PubSubToRedis.java	15.95%	216 Missing ⚠️
...le/cloud/teleport/metadata/util/MetadataUtils.java	0.00%	2 Missing ⚠️

❌ Your patch check has failed because the patch coverage (15.83%) is below the target coverage (80.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #3262      +/-   ##
============================================
- Coverage     50.93%   50.90%   -0.03%     
- Complexity     5160     5169       +9     
============================================
  Files           977      978       +1     
  Lines         60408    60689     +281     
  Branches       6612     6638      +26     
============================================
+ Hits          30766    30896     +130     
- Misses        27484    27624     +140     
- Partials       2158     2169      +11     

Components	Coverage Δ
spanner-templates	70.83% <ø> (-0.17%)	⬇️
spanner-import-export	68.89% <ø> (-0.17%)	⬇️
spanner-live-forward-migration	80.04% <ø> (-0.08%)	⬇️
spanner-live-reverse-replication	77.58% <ø> (ø)
spanner-bulk-migration	88.08% <ø> (-0.38%)	⬇️

Files with missing lines	Coverage Δ
...le/cloud/teleport/metadata/util/MetadataUtils.java	50.00% <0.00%> (ø)
...gle/cloud/teleport/v2/templates/PubSubToRedis.java	14.78% <15.95%> (+14.78%)	⬆️

... and 17 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[tarun-google]

LGTM, Thanks for adding this missing feature!