[Fix IT] Fix CassandraAllDataTypesIT failing test #3111

aasthabharill · 2025-12-15T11:33:05Z

https://b.corp.google.com/issues/468909414

RCA

reference.conf file using to construct cassandra configuration comes from org/apache/cassandra which is at version 4.18.1 but all other related classes (for eg. TypedDriverOption) come from com/datastax/oss which is at version 4.17.0. This mismatch is causing newer added options/parameters to fail.

Fix:

com.datastax.oss version upgrade to 4.18.1 - this is actually invalid ever since datastax moved to apache so we now have to use the new library org.apache.cassandra with updated version 4.18.1

Debugging:

When i checked the reference.conf file in the java driver core we build (/.m2/repository/org/apache/cassandra/java-driver-core/4.18.1/java-driver-core-4.18.1.jar!/reference.conf) it had the line preferred-remote-dcs = [""].
This reference.conf is automatically merged with our specified .conf file (Documentation: "Since application.conf inherits from reference.conf, you only need to redeclare what you override")
We strictly check if all the options specified in the final configuration are "valid" configurations by using the class TypedDriverOption (path in jar: /Users/aasthabharill/.m2/repository/com/datastax/oss/java-driver-core/4.17.0/java-driver-core-4.17.0.jar!/com/datastax/oss/driver/api/core/config/TypedDriverOption.class)
Problem:
- All the updated versions (specifically 4.18) of the java driver are in apache/cassandra (mentioned on datastax website)
- apache/cassandra is actually used to get the reference.conf file we're using which has the preferred-remote-dcs = [""] field.
- but we're using TypedDriverOption of datastax version 4.17 which did not have preferred-remote-dcs in its reference.conf
- Because we're using the above outdated TypedDriverOption to get a list of all the "valid" options - we will keep getting errors as and when new options are added.

gemini-code-assist · 2025-12-15T11:33:20Z

Summary of Changes

Hello @aasthabharill, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request resolves a failing integration test by enhancing the robustness of the Cassandra driver configuration loading process. Previously, encountering an unknown Cassandra option would lead to an IllegalArgumentException and a crash. With this change, the system will now gracefully log a warning for unrecognized options and proceed, preventing unnecessary failures and improving application stability.

Highlights

Graceful Error Handling: The Cassandra driver configuration loader now logs a warning and ignores unknown Cassandra options instead of throwing an IllegalArgumentException.
Test Update: The CassandraDriverConfigLoaderTest has been updated to reflect the new behavior, asserting that unsupported options are now ignored rather than causing an exception.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

codecov · 2025-12-15T11:48:25Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 50.67%. Comparing base (312301e) to head (07a0dd7).
⚠️ Report is 12 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #3111      +/-   ##
============================================
+ Coverage     50.51%   50.67%   +0.16%     
- Complexity     5046     5053       +7     
============================================
  Files           973      974       +1     
  Lines         59744    59828      +84     
  Branches       6512     6527      +15     
============================================
+ Hits          30177    30318     +141     
+ Misses        27438    27367      -71     
- Partials       2129     2143      +14

Components	Coverage Δ
spanner-templates	`70.81% <0.00%> (+0.01%)`	⬆️
spanner-import-export	`68.98% <ø> (ø)`
spanner-live-forward-migration	`80.04% <0.00%> (-0.07%)`	⬇️
spanner-live-reverse-replication	`77.73% <ø> (-0.02%)`	⬇️
spanner-bulk-migration	`88.28% <ø> (+0.01%)`	⬆️
see 10 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

.../java/com/google/cloud/teleport/v2/spanner/migrations/utils/CassandraDriverConfigLoader.java

manitgupta

Let’s wait for all the Cassandra tests to pass before merging.

…m#3111) https://b.corp.google.com/issues/468909414 #### RCA - reference.conf file using to construct cassandra configuration comes from org/apache/cassandra which is at version 4.18.1 but all other related classes (for eg. TypedDriverOption) come from com/datastax/oss which is at version 4.17.0. This mismatch is causing newer added options/parameters to fail. #### Fix: com.datastax.oss version upgrade to 4.18.1 - this is actually invalid ever since datastax moved to apache so we now have to use the new library org.apache.cassandra with updated version 4.18.1

* Add IcebergToPostgres YAML template * Add filter parameter to Iceberg read configuration- Added filter (str) parameter to ReadFromIceberg transform in IcebergToPostgres.yaml- Updated IcebergToPostgresYaml.java interface with filter parameter between drop and keep- Adjusted order numbers for subsequent parameters- Added filter parameter to iceberg_options.yaml with example * Fix catalog name to hadoop_catalog in IcebergToPostgresYamlIT * Initialize Iceberg namespace before table creation * Fix file formatting * build(deps): bump org.junit.jupiter:junit-jupiter-api (#3131) Bumps [org.junit.jupiter:junit-jupiter-api](https://github.com/junit-team/junit-framework) from 6.0.0 to 6.0.1. - [Release notes](https://github.com/junit-team/junit-framework/releases) - [Commits](junit-team/junit-framework@r6.0.0...r6.0.1) --- updated-dependencies: - dependency-name: org.junit.jupiter:junit-jupiter-api dependency-version: 6.0.1 dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * [SourceDbToSpanner] Reducing the dlqRetryMinutes and Correcting BulkAndLiveSpannerFT (#3132) * Correcting BulkAndLiveSpannerFT * Adding the worker machine type parameter * [Fix IT] Fix CassandraAllDataTypesIT failing test (#3111) https://b.corp.google.com/issues/468909414 #### RCA - reference.conf file using to construct cassandra configuration comes from org/apache/cassandra which is at version 4.18.1 but all other related classes (for eg. TypedDriverOption) come from com/datastax/oss which is at version 4.17.0. This mismatch is causing newer added options/parameters to fail. #### Fix: com.datastax.oss version upgrade to 4.18.1 - this is actually invalid ever since datastax moved to apache so we now have to use the new library org.apache.cassandra with updated version 4.18.1 * Add integration tests to verify table and index limits for bulk spanner migration from PostgreSQL (#3103) * Add check on max STRING length and also run for PG dialect Note, this also updates the length of the BYTES row that we migrate because the previous value did not actually exceed the limit (it ended up being 7MiB, less than the 10MiB max). This now writes a value that's actually larger than the limit and verifies that the row is NOT saved. * Add check on max row size for both dialects Currently ignored because it causes OOMs on low-provisioned VMs/DBs * Fix spotless * Add integration tests to check string and cell max size limits when migrating from Postgres * Add integration tests to check primary key and index max size limits when migrating from Postgres * Add integration tests to check row max size limits when migrating from Postgres * Support Float as primary key for Mysql source (#3016) * Support Float as primary key for Mysql source * add reference to mysql docs for the Float index * Add new Source IndexType float that map to java Float.class * Fix test for Indexes discovery (Float has its own index type now, not mapped a IndexType.OTHER) * Add TODOs for support Boundary.isSplittable on Approximate values (Float, Double) * Add integration tests * fix * - Add DecimalStepSize property to SourceColumnIndexInfo which is then passed to PartitionColumn. This is to allow the Boundary splitter to know what decimal precision is used by the source column. - Update Boundary.isSplittable to support FLOAT by doing delta based compare between 2 values. Either use a small default delta step size or the one provided the source db column. - Update MysqlDialectAdapter to pass the Numeric-scale schema value of Float column as DecimalStepSize to SourceColumnIndexInfo * syntax improvement * comment and clean diff * Add unit tests for BoundaryExtractorFactory Float.isSpittable custom behavior (with DecimalsStep paramater) * formatting * fix year index unit test * Add test for MysqlDialectAdapter, index NUMERIC_SCALE_COL * comment * Move default decimalStep from Boundary to MysqlDialectAdapter * fix PartitionColumn decimalStep preconditions * Add tests of Boundary.isMergable with float values (check that it works with approximate delta based comparisons) * Formating with mvn:spotless * build bigdecimal step default from String instead of float (to avoid approximation) * Fix tests * Address PR feedback --------- Co-authored-by: Nicolas Emond <[email protected]> * Update README with filter parameter documentation --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Ulada Butsenka <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: darshan-sj <[email protected]> Co-authored-by: aasthabharill <[email protected]> Co-authored-by: Nicolas Emond <[email protected]> Co-authored-by: Nicolas Riche <[email protected]>

pull-request-size bot added the size/M label Dec 15, 2025

aasthabharill added the bug-fix label Dec 15, 2025

pull-request-size bot added size/S and removed size/M labels Dec 15, 2025

VardhanThigle reviewed Dec 16, 2025

View reviewed changes

.../java/com/google/cloud/teleport/v2/spanner/migrations/utils/CassandraDriverConfigLoader.java Show resolved Hide resolved

pull-request-size bot added size/XXL and removed size/S labels Dec 17, 2025

aasthabharill force-pushed the aastha-cassandrait-fix branch from 1852cb2 to 7775dc1 Compare December 17, 2025 09:25

pull-request-size bot added size/S size/M and removed size/XXL size/S labels Dec 17, 2025

aasthabharill added 3 commits December 17, 2025 14:59

initial changes

672a6ed

spotless

f412be7

logging

eec12a5

aasthabharill force-pushed the aastha-cassandrait-fix branch from dfe98ea to eec12a5 Compare December 17, 2025 09:30

pull-request-size bot added size/XS and removed size/M labels Dec 17, 2025

remove ignore flag

b3cf227

pull-request-size bot added size/M and removed size/XS labels Dec 17, 2025

aasthabharill added 3 commits December 17, 2025 16:26

change version

33785ca

remove some log statements

625c5d1

package update

bfca3d0

pull-request-size bot added size/XL and removed size/M labels Dec 18, 2025

aasthabharill added 2 commits December 18, 2025 21:48

specify worker machine type

36a7636

remove logs

07a0dd7

aasthabharill marked this pull request as ready for review December 19, 2025 04:37

aasthabharill requested a review from a team as a code owner December 19, 2025 04:38

aasthabharill requested review from VardhanThigle, darshan-sj, manitgupta and sm745052 December 19, 2025 04:38

manitgupta approved these changes Dec 19, 2025

View reviewed changes

aasthabharill merged commit 44ad23c into GoogleCloudPlatform:main Dec 19, 2025
21 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Fix IT] Fix CassandraAllDataTypesIT failing test #3111

[Fix IT] Fix CassandraAllDataTypesIT failing test #3111

Uh oh!

aasthabharill commented Dec 15, 2025 •

edited

Loading

Uh oh!

gemini-code-assist bot commented Dec 15, 2025

Uh oh!

codecov bot commented Dec 15, 2025 •

edited

Loading

Uh oh!

Uh oh!

manitgupta left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

[Fix IT] Fix CassandraAllDataTypesIT failing test #3111

[Fix IT] Fix CassandraAllDataTypesIT failing test #3111

Uh oh!

Conversation

aasthabharill commented Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

RCA

Fix:

Debugging:

Uh oh!

gemini-code-assist bot commented Dec 15, 2025

Summary of Changes

Highlights

Footnotes

Uh oh!

codecov bot commented Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

manitgupta left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aasthabharill commented Dec 15, 2025 •

edited

Loading

codecov bot commented Dec 15, 2025 •

edited

Loading