Skip to content

feat(Spanner): Live Migration: Remove generated columns from change events during transformation and add tests for this behavior#3255

Merged
pratickchokhani merged 1 commit intoGoogleCloudPlatform:mainfrom
pratickchokhani:live-generated-column
Feb 20, 2026
Merged

feat(Spanner): Live Migration: Remove generated columns from change events during transformation and add tests for this behavior#3255
pratickchokhani merged 1 commit intoGoogleCloudPlatform:mainfrom
pratickchokhani:live-generated-column

Conversation

@pratickchokhani
Copy link
Contributor

@pratickchokhani pratickchokhani commented Jan 26, 2026

If a primary is Generated Column within Spanner and the change stream event is DELETE, then DML will be used
Else Mutation is used.

For DML, we include all the non-generated columns in the where clause. This is so because we do not have the list of dependent columns.

If column is Generated Column in Spanner, then the column is omitted from both DML or Mutations.

@pratickchokhani pratickchokhani added the improvement Making existing code better label Jan 26, 2026
@pratickchokhani pratickchokhani changed the title Live generated column feat(Spanner): Live Migration: Remove generated columns from change events during transformation and add tests for this behavior Jan 26, 2026
@codecov
Copy link

codecov bot commented Jan 26, 2026

Codecov Report

❌ Patch coverage is 79.74684% with 16 lines in your changes missing coverage. Please review.
✅ Project coverage is 55.28%. Comparing base (d116909) to head (385cdaa).
⚠️ Report is 13 commits behind head on main.

Files with missing lines Patch % Lines
...ort/v2/templates/SpannerTransactionWriterDoFn.java 11.11% 7 Missing and 1 partial ⚠️
...rt/v2/templates/datastream/ChangeEventContext.java 91.83% 0 Missing and 4 partials ⚠️
...igrations/convertors/ChangeEventTypeConvertor.java 71.42% 3 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #3255      +/-   ##
============================================
+ Coverage     51.05%   55.28%   +4.22%     
+ Complexity     5212     2291    -2921     
============================================
  Files           994      492     -502     
  Lines         60723    28947   -31776     
  Branches       6657     3076    -3581     
============================================
- Hits          31000    16002   -14998     
+ Misses        27543    11966   -15577     
+ Partials       2180      979    -1201     
Components Coverage Δ
spanner-templates 71.96% <79.74%> (+1.11%) ⬆️
spanner-import-export ∅ <ø> (∅)
spanner-live-forward-migration 79.85% <79.74%> (+<0.01%) ⬆️
spanner-live-reverse-replication 77.33% <80.00%> (-0.04%) ⬇️
spanner-bulk-migration 87.99% <80.00%> (-0.01%) ⬇️
gcs-spanner-dv 85.73% <80.00%> (-0.03%) ⬇️
Files with missing lines Coverage Δ
...eport/v2/templates/spanner/ShadowTableCreator.java 89.36% <100.00%> (+0.23%) ⬆️
...ations/convertors/ChangeEventSpannerConvertor.java 85.24% <100.00%> (+2.08%) ⬆️
...rt/v2/templates/datastream/ChangeEventContext.java 92.47% <91.83%> (+1.16%) ⬆️
...igrations/convertors/ChangeEventTypeConvertor.java 88.05% <71.42%> (-1.95%) ⬇️
...ort/v2/templates/SpannerTransactionWriterDoFn.java 50.43% <11.11%> (-1.37%) ⬇️

... and 520 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@pratickchokhani pratickchokhani force-pushed the live-generated-column branch 3 times, most recently from 95666a2 to 928a20e Compare February 6, 2026 08:48
@pull-request-size pull-request-size bot added size/L and removed size/XL labels Feb 6, 2026
@pratickchokhani pratickchokhani force-pushed the live-generated-column branch 2 times, most recently from dc66812 to 8c90808 Compare February 9, 2026 10:04
@pull-request-size pull-request-size bot added size/XL and removed size/L labels Feb 9, 2026
@pratickchokhani pratickchokhani force-pushed the live-generated-column branch 17 times, most recently from e7cdb2d to 34ec316 Compare February 11, 2026 08:13
@pratickchokhani pratickchokhani marked this pull request as ready for review February 11, 2026 10:47
@pratickchokhani pratickchokhani requested a review from a team as a code owner February 11, 2026 10:47
@pratickchokhani pratickchokhani force-pushed the live-generated-column branch 2 times, most recently from cd4c9b0 to 26dbfd4 Compare February 17, 2026 11:24
Copy link
Contributor

@shreyakhajanchi shreyakhajanchi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added some comments. Please add details of manual test run in the description for completeness.

@pratickchokhani pratickchokhani force-pushed the live-generated-column branch 5 times, most recently from 785f60e to 4a1f40c Compare February 18, 2026 10:45
…nsformation and add tests for this behavior.
Copy link
Contributor

@shreyakhajanchi shreyakhajanchi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@pratickchokhani pratickchokhani merged commit 76ecc67 into GoogleCloudPlatform:main Feb 20, 2026
28 of 29 checks passed
joy91227 pushed a commit to joy91227/DataflowTemplates that referenced this pull request Mar 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

improvement Making existing code better size/XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants