Add integration tests to live spanner migration for checking PostgreSQL data type support#3152
Conversation
Summary of ChangesHello @nmemond, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request significantly enhances the testing framework for the 'DataStreamToSpanner' Flex template by introducing dedicated integration tests for PostgreSQL data type migration. The tests are designed to validate how various PostgreSQL data types are mapped and transferred to both standard and PostgreSQL dialect Spanner databases during a live migration. This ensures the reliability and correctness of data transformations, with a pragmatic approach to handle currently unsupported types without blocking the test suite. Highlights
🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console. Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here. You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension. Footnotes
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3152 +/- ##
============================================
+ Coverage 51.03% 55.18% +4.14%
+ Complexity 5173 2288 -2885
============================================
Files 978 492 -486
Lines 60490 28862 -31628
Branches 6638 3058 -3580
============================================
- Hits 30871 15927 -14944
+ Misses 27455 11962 -15493
+ Partials 2164 973 -1191
🚀 New features to boost your workflow:
|
...st/java/com/google/cloud/teleport/v2/templates/PostgreSQLDatastreamToSpannerDataTypesIT.java
Outdated
Show resolved
Hide resolved
|
Failure in asserts |
...st/java/com/google/cloud/teleport/v2/templates/PostgreSQLDatastreamToSpannerDataTypesIT.java
Outdated
Show resolved
Hide resolved
nmemond
force-pushed
the
pg-data-type-tests-live
branch
from
1505c37 to
b720ad6
Compare
|
There seems to be a Assert in matching
|
timatkinson-yw
force-pushed
the
pg-data-type-tests-live
branch
from
70ed130 to
3490b3c
Compare
timatkinson-yw
force-pushed
the
pg-data-type-tests-live
branch
from
3490b3c to
aea0a25
Compare
|
Support and proses issues |
|
This the update on the place |
| if (numCombinedConditions >= 3) { | ||
| conditions.add(combinedCondition); | ||
| combinedCondition = null; | ||
| numCombinedConditions = 0; |
There was a problem hiding this comment.
Why are you grouping 3 conditions with and and then using the Chained condition on the grouped ones? Any particular reason for having this pattern?
We should not use chained condition check like this. Chained condition check introduces 15s delay in between each check and with so many of them chained, the test takes a lot of time to execute and could sometimes timeout if the migration is a little slow.
I'm working on fixing the already checked in tests with this wrong pattern -
#3380
There was a problem hiding this comment.
The nice thing about the Chained condition is that it avoids re-running conditions that have already passed, but of course the downside is as you mentioned: regardless of whether a condition passes or fails, there's a 15 second delay before the next check runs.
The combination of Chained and and was an effort to strike a balance between the two, given there are quite a few conditions. Maybe 3 was too small of a grouping, but I do think the idea itself is sound. Otherwise, there's a chance we end up having to re-run dozens of conditions over and over because a later condition is failing. In the worst case, we re-run every condition except the last one (if that's the one that's failing). Using a fixed group size, we can decide how many conditions will re-run in that worst case: for a group size of X, in the worst case we will re-run X-1 passing conditions every check.
There was a problem hiding this comment.
I don't think that re-running conditions which have already passed is a problem here. The tables don't contain a lot of rows and the count query should return within milliseconds. Even if we have 100s of tables, querying all of them is not that much inefficient operation. On the other hand waiting for 15s seconds between each query is bad and it is adding up a lot of delay here. Please don't use this pattern in your further PRs and use "and" across all the conditions.
…QL data type support (GoogleCloudPlatform#3152) * Add data types test for live migration from PostgreSQL to Spanner (both dialects) * Use static resource manager instead to allow access to CDC log * Fix expected value in test * Fix spotless * Ignore columns which don't seem to migrate consistently (to avoid flaky test) * Properly ignore the columns... * Ignore columns which don't seem to migrate consistently (to avoid flaky tests) * Ignore columns which don't seem to migrate consistently (to avoid flaky tests) --------- Co-authored-by: Atkinson <tim.atkinson@improving.com>
5 participants
This adds integration tests to check data type mappings when doing a live migration from PostgreSQL to Spanner for both dialects.
Note that some of the type mappings fail to migrate as expected. The checks for those are still included for completeness, but they're ignored to avoid failing the tests.