Skip to content

Refactor template to avoid using session file for type info#2529

Merged
Deep1998 merged 4 commits intoGoogleCloudPlatform:mainfrom
Deep1998:rev-cp-2
Jul 21, 2025
Merged

Refactor template to avoid using session file for type info#2529
Deep1998 merged 4 commits intoGoogleCloudPlatform:mainfrom
Deep1998:rev-cp-2

Conversation

@Deep1998
Copy link
Copy Markdown
Contributor

@Deep1998 Deep1998 commented Jul 9, 2025

This PR refactors the reverse template to avoid using session file for anything other than table/column name-mappings. It instead relies of the Ddl and SourceDdl object for schema specific info. This refactoring is necessary to introduce the schemamapper and henceforth, overrides.

@Deep1998 Deep1998 changed the title Rev cp 2 Refactor template to avoid using session file for type info Jul 9, 2025
@Deep1998 Deep1998 force-pushed the rev-cp-2 branch 2 times, most recently from 49e10db to 78e6291 Compare July 9, 2025 11:30
@codecov
Copy link
Copy Markdown

codecov bot commented Jul 9, 2025

Codecov Report

Attention: Patch coverage is 59.14634% with 67 lines in your changes missing coverage. Please review.

Project coverage is 49.71%. Comparing base (c0fb979) to head (cb13298).
Report is 11 commits behind head on main.

Files with missing lines Patch % Lines
...cloud/teleport/v2/templates/SpannerToSourceDb.java 0.00% 35 Missing ⚠️
...rt/v2/templates/dbutils/dml/MySQLDMLGenerator.java 72.13% 9 Missing and 8 partials ⚠️
...2/templates/dbutils/dml/CassandraDMLGenerator.java 80.00% 4 Missing and 5 partials ⚠️
...eport/v2/templates/transforms/AssignShardIdFn.java 66.66% 3 Missing and 1 partial ⚠️
...v2/templates/dbutils/dml/CassandraTypeHandler.java 85.71% 0 Missing and 1 partial ⚠️
...v2/templates/transforms/SourceWriterTransform.java 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #2529      +/-   ##
============================================
- Coverage     49.73%   49.71%   -0.03%     
- Complexity     4822     5179     +357     
============================================
  Files           948      948              
  Lines         57938    58038     +100     
  Branches       6259     6275      +16     
============================================
+ Hits          28818    28853      +35     
- Misses        27070    27123      +53     
- Partials       2050     2062      +12     
Components Coverage Δ
spanner-templates 69.94% <59.14%> (-0.15%) ⬇️
spanner-import-export 68.61% <ø> (+0.13%) ⬆️
spanner-live-forward-migration 79.33% <ø> (+0.10%) ⬆️
spanner-live-reverse-replication 77.02% <59.14%> (-0.78%) ⬇️
spanner-bulk-migration 88.05% <ø> (+0.06%) ⬆️
Files with missing lines Coverage Δ
...plates/dbutils/processor/InputRecordProcessor.java 86.44% <100.00%> (-1.28%) ⬇️
...leport/v2/templates/transforms/SourceWriterFn.java 85.34% <100.00%> (+0.12%) ⬆️
...v2/templates/dbutils/dml/CassandraTypeHandler.java 84.73% <85.71%> (ø)
...v2/templates/transforms/SourceWriterTransform.java 0.00% <0.00%> (ø)
...eport/v2/templates/transforms/AssignShardIdFn.java 79.59% <66.66%> (+1.86%) ⬆️
...2/templates/dbutils/dml/CassandraDMLGenerator.java 84.96% <80.00%> (-5.10%) ⬇️
...rt/v2/templates/dbutils/dml/MySQLDMLGenerator.java 88.98% <72.13%> (-8.80%) ⬇️
...cloud/teleport/v2/templates/SpannerToSourceDb.java 0.00% <0.00%> (ø)

... and 11 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@Deep1998 Deep1998 marked this pull request as ready for review July 9, 2025 17:51
@Deep1998 Deep1998 requested a review from a team as a code owner July 9, 2025 17:51
@Deep1998 Deep1998 marked this pull request as draft July 9, 2025 17:52
@Deep1998 Deep1998 force-pushed the rev-cp-2 branch 11 times, most recently from 7ce8030 to 6cd4f20 Compare July 15, 2025 06:39
@Deep1998 Deep1998 added the improvement Making existing code better label Jul 15, 2025
@Deep1998 Deep1998 marked this pull request as ready for review July 16, 2025 16:34
Copy link
Copy Markdown
Contributor

@bharadwaj-aditya bharadwaj-aditya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This changes are very spread out and hard to quantify or review because of the fan out. It would be cleaner to introduce an interface and swap between session file and alternates. It will keep the implementation and testing clean

Copy link
Copy Markdown
Contributor

@bharadwaj-aditya bharadwaj-aditya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving as an intermediate PR before the interface changes are merged to simplify this code.

@Deep1998 Deep1998 merged commit ce12473 into GoogleCloudPlatform:main Jul 21, 2025
21 checks passed
@Deep1998 Deep1998 deleted the rev-cp-2 branch July 21, 2025 09:52
MnkyGns pushed a commit to MnkyGns/DataflowTemplates that referenced this pull request Feb 12, 2026
…oudPlatform#2529)

* Use tmp cql session builder

* Refactor code to avoid session file for anything other than name mappings

* Rebase onto sourceddl branch with resolved comments

* Move schema utils to test folder and resolve comments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

improvement Making existing code better size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants