[SourceDB to Spanner] Fetch size Automation

[sm745052]

No description provided.

[gemini-code-assist]

Summary of Changes

Hello @sm745052, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the SourceDB to Spanner migration pipelines by automating the JDBC fetch size configuration. Previously, the fetch size might have been a fixed value or manually configured, potentially leading to suboptimal performance or resource issues. The new implementation dynamically calculates an appropriate fetch size by considering the Dataflow worker's machine type (CPU and memory) and the estimated size of rows being read from the source database. This automation aims to improve the efficiency and stability of data transfer, adapting to varying data characteristics and worker configurations.

Highlights

• Automated JDBC Fetch Size Calculation: Introduces a new FetchSizeCalculator to dynamically determine the optimal JDBC fetch size for data reads, based on Dataflow worker machine type (CPU and memory) and estimated row size from the source database.
• Enhanced Row Size Estimation: Extends the DialectAdapter and JdbcValueMappingsProvider interfaces with methods for estimating the size of a row for various database column types. Concrete implementations for MySQL and PostgreSQL are updated to provide these detailed size estimations.
• Centralized Dataflow Worker Machine Type Utility: Replaces the previous DataflowWorkerMachineTypeValidator with a more comprehensive DataflowWorkerMachineTypeUtils. This new utility can retrieve worker memory and core information directly from the Compute Engine API, cache it for efficiency, and perform machine specification validations.
• Seamless Integration and Refactoring: The automated fetch size calculation is integrated into the JdbcIoWrapper used by SourceDB to Spanner migration templates. This involves updating configuration classes (JdbcIOWrapperConfig, TableConfig, SourceTableSchema) and refactoring references to the new machine type utility across multiple template files.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[codecov]

Codecov Report

❌ Patch coverage is 94.90617% with 19 lines in your changes missing coverage. Please review.
✅ Project coverage is 51.84%. Comparing base (f8677e9) to head (94822c8).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...grations/utils/DataflowWorkerMachineTypeUtils.java	91.17%	5 Missing and 1 partial ⚠️
...ud/teleport/v2/options/OptionsToConfigBuilder.java	76.19%	4 Missing and 1 partial ⚠️
...oud/teleport/v2/templates/DataStreamToSpanner.java	0.00%	1 Missing ⚠️
.../jdbc/dialectadapter/mysql/MysqlTimeConverter.java	97.50%	1 Missing ⚠️
...dbc/rowmapper/provider/MysqlJdbcValueMappings.java	98.52%	0 Missing and 1 partial ⚠️
...owmapper/provider/PostgreSQLJdbcValueMappings.java	98.86%	0 Missing and 1 partial ⚠️
...jdbc/uniformsplitter/UniformSplitterDBAdapter.java	0.00%	1 Missing ⚠️
...niformsplitter/range/BoundaryExtractorFactory.java	90.00%	0 Missing and 1 partial ⚠️
...cloud/teleport/v2/templates/SourceDbToSpanner.java	0.00%	1 Missing ⚠️
...cloud/teleport/v2/templates/SpannerToSourceDb.java	0.00%	1 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##               main    #3260       +/-   ##
=============================================
+ Coverage     33.75%   51.84%   +18.09%     
- Complexity      624     5859     +5235     
=============================================
  Files           247     1016      +769     
  Lines         16088    61716    +45628     
  Branches       1611     6795     +5184     
=============================================
+ Hits           5430    31999    +26569     
- Misses        10188    27492    +17304     
- Partials        470     2225     +1755     

Components	Coverage Δ
spanner-templates	71.61% <94.90%> (∅)
spanner-import-export	69.05% <ø> (∅)
spanner-live-forward-migration	80.02% <89.85%> (∅)
spanner-live-reverse-replication	77.33% <89.85%> (∅)
spanner-bulk-migration	88.31% <95.41%> (∅)
gcs-spanner-dv	86.68% <91.17%> (∅)

Files with missing lines	Coverage Δ
.../reader/io/jdbc/dialectadapter/DialectAdapter.java	100.00% <100.00%> (ø)
...jdbc/dialectadapter/mysql/MysqlDialectAdapter.java	99.65% <100.00%> (ø)
.../reader/io/jdbc/iowrapper/FetchSizeCalculator.java	100.00% <100.00%> (ø)
...source/reader/io/jdbc/iowrapper/JdbcIoWrapper.java	93.23% <100.00%> (ø)
.../io/jdbc/iowrapper/config/JdbcIOWrapperConfig.java	97.05% <100.00%> (ø)
...e/reader/io/jdbc/iowrapper/config/TableConfig.java	100.00% <100.00%> (ø)
.../source/reader/io/jdbc/rowmapper/JdbcMappings.java	100.00% <100.00%> (ø)
.../uniformsplitter/transforms/RangeBoundaryDoFn.java	91.17% <100.00%> (ø)
.../v2/source/reader/io/schema/SourceTableSchema.java	100.00% <ø> (ø)
...loud/teleport/v2/templates/PipelineController.java	49.33% <100.00%> (ø)
... and 10 more

... and 767 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.