feat: add IcebergToPostgres YAML blueprint #3129
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
Summary of ChangesHello @wladabutenko, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request delivers a new Dataflow Flex Template, specified in YAML, designed to perform batch data transfers from Apache Iceberg tables to PostgreSQL databases. This template provides a robust and configurable solution for users needing to move data between these two popular data storage systems, complete with thorough documentation and an integration test to validate its operation. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
derrickaw
left a comment
derrickaw
left a comment
There was a problem hiding this comment.
LGTM - Get @tarun-google review as well. Thanks.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3129 +/- ##
============================================
+ Coverage 50.66% 50.68% +0.02%
- Complexity 5053 5069 +16
============================================
Files 974 974
Lines 59828 59872 +44
Branches 6527 6536 +9
============================================
+ Hits 30309 30349 +40
- Misses 27377 27379 +2
- Partials 2142 2144 +2
🚀 New features to boost your workflow:
|
yaml/src/test/java/com/google/cloud/teleport/templates/yaml/IcebergToPostgresYamlIT.java
Show resolved
Hide resolved
…) parameter to ReadFromIceberg transform in IcebergToPostgres.yaml- Updated IcebergToPostgresYaml.java interface with filter parameter between drop and keep- Adjusted order numbers for subsequent parameters- Added filter parameter to iceberg_options.yaml with example
|
LGTM. I can merge once integration tests are passing |
…tform#3131) Bumps [org.junit.jupiter:junit-jupiter-api](https://github.com/junit-team/junit-framework) from 6.0.0 to 6.0.1. - [Release notes](https://github.com/junit-team/junit-framework/releases) - [Commits](junit-team/junit-framework@r6.0.0...r6.0.1) --- updated-dependencies: - dependency-name: org.junit.jupiter:junit-jupiter-api dependency-version: 6.0.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…ndLiveSpannerFT (GoogleCloudPlatform#3132) * Correcting BulkAndLiveSpannerFT * Adding the worker machine type parameter
…m#3111) https://b.corp.google.com/issues/468909414 #### RCA - reference.conf file using to construct cassandra configuration comes from org/apache/cassandra which is at version 4.18.1 but all other related classes (for eg. TypedDriverOption) come from com/datastax/oss which is at version 4.17.0. This mismatch is causing newer added options/parameters to fail. #### Fix: com.datastax.oss version upgrade to 4.18.1 - this is actually invalid ever since datastax moved to apache so we now have to use the new library org.apache.cassandra with updated version 4.18.1
…er migration from PostgreSQL (GoogleCloudPlatform#3103) * Add check on max STRING length and also run for PG dialect Note, this also updates the length of the BYTES row that we migrate because the previous value did not actually exceed the limit (it ended up being 7MiB, less than the 10MiB max). This now writes a value that's actually larger than the limit and verifies that the row is NOT saved. * Add check on max row size for both dialects Currently ignored because it causes OOMs on low-provisioned VMs/DBs * Fix spotless * Add integration tests to check string and cell max size limits when migrating from Postgres * Add integration tests to check primary key and index max size limits when migrating from Postgres * Add integration tests to check row max size limits when migrating from Postgres
* Support Float as primary key for Mysql source * add reference to mysql docs for the Float index * Add new Source IndexType float that map to java Float.class * Fix test for Indexes discovery (Float has its own index type now, not mapped a IndexType.OTHER) * Add TODOs for support Boundary.isSplittable on Approximate values (Float, Double) * Add integration tests * fix * - Add DecimalStepSize property to SourceColumnIndexInfo which is then passed to PartitionColumn. This is to allow the Boundary splitter to know what decimal precision is used by the source column. - Update Boundary.isSplittable to support FLOAT by doing delta based compare between 2 values. Either use a small default delta step size or the one provided the source db column. - Update MysqlDialectAdapter to pass the Numeric-scale schema value of Float column as DecimalStepSize to SourceColumnIndexInfo * syntax improvement * comment and clean diff * Add unit tests for BoundaryExtractorFactory Float.isSpittable custom behavior (with DecimalsStep paramater) * formatting * fix year index unit test * Add test for MysqlDialectAdapter, index NUMERIC_SCALE_COL * comment * Move default decimalStep from Boundary to MysqlDialectAdapter * fix PartitionColumn decimalStep preconditions * Add tests of Boundary.isMergable with float values (check that it works with approximate delta based comparisons) * Formating with mvn:spotless * build bigdecimal step default from String instead of float (to avoid approximation) * Fix tests * Address PR feedback --------- Co-authored-by: Nicolas Emond <[email protected]>
tarun-google
merged commit 94a24b1
into
GoogleCloudPlatform:main
7 participants
IcebergToPostgres YAML Batch template that reads data from Iceberg tables and writes records to PostgreSQL databases.