Fix the integration test and workflow for Object Storage adapter #3232

KodaiD · 2025-11-28T00:48:07Z

Description

This PR fixes the integration test and workflow for S3 and Cloud Storage adapter.

Since S3 and Cloud Storage have a strict request rate limit, integration tests for those adapters fail, currently. To address this issue, this PR fixes the test workflow to execute those tests in separate buckets.

Additionally, since Cloud Storage has a stricter request rate limit allowing one write per second, this PR introduces some waits for some write-heavy spots.

The test is verified in https://github.com/scalar-labs/scalardb/actions/runs/19738491550.

Related issues and/or PRs

Changes made

Updated the GitHub Actions workflow file to execute test groups in separate buckets.
Updated some Consensus Commit-related test cases, which trigger many writes, to include waits.

Checklist

I have commented my code, particularly in hard-to-understand areas.
I have updated the documentation to reflect the changes.
I have considered whether similar issues could occur in other products, components, or modules if this PR is for bug fixes.
Any remaining open issues linked to this PR are documented and up-to-date (Jira, GitHub, etc.).
Tests (unit, integration, etc.) have been added for the changes.
My changes generate no new warnings.
Any dependent changes in other PRs have been merged and published.

Additional notes (optional)

N/A

Release notes

N/A

KodaiD · 2025-11-28T00:58:31Z

.github/workflows/object-storage-adapter-check.yaml

  AWS_SECRET_ACCESS_KEY: ${{ secrets.S3_SECRET_ACCESS_KEY }}
-  S3_REGION: ap-northeast-1
-  S3_BUCKET_NAME: scalardb-test-bucket
+  S3_REGION: us-east-1


This is a change without confidence. I don't know where the runner is located, but I think it's probably not Japan, so I selected the US region.

KodaiD · 2025-11-28T01:02:08Z

...ava/com/scalar/db/storage/objectstorage/ConsensusCommitIntegrationTestWithObjectStorage.java

+  @Override
+  @BeforeEach
+  public void setUp() throws Exception {
+    admin.truncateTable(namespace, TABLE);
+    admin.truncateCoordinatorTables();
+    if (ObjectStorageEnv.isCloudStorage()) {
+      Uninterruptibles.sleepUninterruptibly(2, TimeUnit.SECONDS);
+    }
+  }
+
+  @AfterEach
+  void tearDown() {
+    if (ObjectStorageEnv.isCloudStorage()) {
+      Uninterruptibles.sleepUninterruptibly(2, TimeUnit.SECONDS);
+    }
+  }


Two seconds of sleep resolves the rate limit issue. One second of sleep was not enough when I tried.

KodaiD · 2025-11-28T01:30:38Z

...ava/com/scalar/db/storage/objectstorage/ObjectStorageConditionalMutationIntegrationTest.java

-  @Override
-  protected int getThreadNum() {
-    return 3;
-  }


Since writes are performed for separate partitions in this test (i.e., conflict does not occur), this override is removed to gain more concurrency.

KodaiD · 2025-11-28T01:34:34Z

...db/storage/objectstorage/SingleCrudOperationTransactionIntegrationTestWithObjectStorage.java

+        if (ObjectStorageEnv.isCloudStorage()) {
+          Uninterruptibles.sleepUninterruptibly(1, TimeUnit.SECONDS);
+        }


Those lines are required to avoid the rate limit constraint. We can only write an item once per second.

KodaiD · 2025-11-28T01:36:03Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces fixes for integration tests on S3 and Cloud Storage by working around their rate limits. The changes involve adding waits in write-heavy tests and modifying the test workflow. The approach is sound, but there are opportunities to improve maintainability by reducing code duplication and eliminating magic numbers. I've also identified a potential issue where a thread-limiting override was removed, which might reintroduce the very rate-limiting problems this PR aims to solve.

Copilot

Pull request overview

This PR addresses integration test failures for S3 and Cloud Storage adapters caused by strict request rate limits. The solution splits tests into separate buckets per test group and adds rate-limiting waits for Cloud Storage's strict 1-write-per-second limit.

Key changes:

Expanded the GitHub Actions workflow matrix from 2 test modes to 14 distinct test groups, each using a separate S3/Cloud Storage bucket to avoid rate limit conflicts
Added Cloud Storage-specific sleeps (1 second per write operation, 2 seconds in setUp/tearDown) to handle the strict rate limits
Made visibility changes to the base test class to enable subclass customization for Cloud Storage

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
`.github/workflows/object-storage-adapter-check.yaml`	Reorganized test execution into 14 distinct test groups, each using separate buckets with descriptive suffixes; updated S3 region and bucket naming
`ConsensusCommitSpecificIntegrationTestBase.java`	Changed visibility of fields (`recoveryExecutor`, `groupCommitter`) and methods (`setUp`, `truncateTables`) from private to protected to support subclass overrides
`ConsensusCommitSpecificIntegrationTestWithObjectStorage.java`	Added Cloud Storage-specific sleeps in setUp/tearDown methods and duplicated cleanup logic from parent
`ConsensusCommitIntegrationTestWithObjectStorage.java`	Added Cloud Storage-specific sleeps in setUp/tearDown methods
`SingleCrudOperationTransactionIntegrationTestWithObjectStorage.java`	Overrode `populateRecords()` to add 1-second sleeps between writes for Cloud Storage rate limiting
`ObjectStorageSinglePartitionKeyIntegrationTest.java`	Added `getColumnWithMaxValue()` override to use 'Z' characters instead of 0xFF for Cloud Storage TEXT max values
`ObjectStorageSingleClusteringKeyScanIntegrationTest.java`	Added `getColumnWithMaxValue()` override to use 'Z' characters instead of 0xFF for Cloud Storage TEXT max values
`ObjectStorageMultiplePartitionKeyIntegrationTest.java`	Added `getColumnWithMaxValue()` override to use 'Z' characters instead of 0xFF for Cloud Storage TEXT max values
`ObjectStorageMultipleClusteringKeyScanIntegrationTest.java`	Added `getColumnWithMaxValue()` override to use 'Z' characters instead of 0xFF for Cloud Storage TEXT max values
`ObjectStorageConditionalMutationIntegrationTest.java`	Removed custom thread count override (reverted to base class default)

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

...java/com/scalar/db/storage/objectstorage/ObjectStorageSinglePartitionKeyIntegrationTest.java

...scalar/db/storage/objectstorage/ConsensusCommitSpecificIntegrationTestWithObjectStorage.java

...ava/com/scalar/db/storage/objectstorage/ConsensusCommitIntegrationTestWithObjectStorage.java

...scalar/db/storage/objectstorage/ConsensusCommitSpecificIntegrationTestWithObjectStorage.java

...ava/com/scalar/db/storage/objectstorage/ConsensusCommitIntegrationTestWithObjectStorage.java

komamitsu

LGTM! 👍

Torch3333 · 2025-11-28T06:58:37Z

.github/workflows/object-storage-adapter-check.yaml

+          - label: storage_others
+            tests_filter: '--tests "**.ObjectStorageCaseSensitivity**" --tests "**.ObjectStorageColumnValue**" --tests "**.ObjectStorageCrossPartition**" --tests "**.ObjectStorageIntegrationTest" --tests "**.ObjectStorageJapanese**" --tests "**.ObjectStorageMutationAtomicity**" --tests "**.ObjectStorageWithReservedKeyword**"'
+            bucket_suffix: storage
+            group_commit_enabled: false


I have one concern: when adding a new integration test class, we must not forget to add the object storage implementation class to this list; otherwise, the new test won't be run by the CI.

Basically, we would need a way to run the test for this others category that runs any tests not already included in any of the other categories.
One way to achieve this is to use the Junit annotation EnabledIfSystemProperty. Using Gradle test task filtering might be possible too.

For example, using the @EnabledIfSystemProperty annotation, we can:

annotate the class of category A with @EnabledIfSystemProperty("runCategoryA")
./gradlew integrationTestObjectStorage -DrunCategoryA

annotate the class of category B with @EnabledIfSystemProperty("runCategoryB")
./gradlew integrationTestObjectStorage -DrunCategoryB

leave the tests of the others category without annotation that would be run by default.
./gradlew integrationTestObjectStorage

What do you think ?

KodaiD added 7 commits November 25, 2025 09:59

Fix workflow

960d6af

Fix workflow and test

ff63b5a

Fix test

4119ccf

Fix workflow and test

94af094

Add waits

517675a

Prepare more buckets

0c3da51

Fix test

ee5da61

KodaiD self-assigned this Nov 28, 2025

KodaiD commented Nov 28, 2025

View reviewed changes

gemini-code-assist bot reviewed Nov 28, 2025

View reviewed changes

KodaiD marked this pull request as ready for review November 28, 2025 01:39

Copilot AI review requested due to automatic review settings November 28, 2025 01:39

Copilot started reviewing on behalf of KodaiD November 28, 2025 01:39 View session

Copilot finished reviewing on behalf of KodaiD November 28, 2025 01:42

Copilot AI reviewed Nov 28, 2025

View reviewed changes

KodaiD added 2 commits November 28, 2025 11:55

Apply suggestions from code review

f5a9b9d

Merge remote-tracking branch 'origin' into fix-object-storage-adapter-ci

c10957a

KodaiD requested review from a team, Torch3333, brfrn169, feeblefakie and komamitsu and removed request for a team November 28, 2025 02:57

komamitsu approved these changes Nov 28, 2025

View reviewed changes

Torch3333 reviewed Nov 28, 2025

View reviewed changes

Fix the integration test and workflow for Object Storage adapter #3232

Are you sure you want to change the base?

Fix the integration test and workflow for Object Storage adapter #3232

Uh oh!

Conversation

KodaiD commented Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related issues and/or PRs

Changes made

Checklist

Additional notes (optional)

Release notes

Uh oh!

KodaiD Nov 28, 2025

Choose a reason for hiding this comment

Uh oh!

KodaiD Nov 28, 2025

Choose a reason for hiding this comment

Uh oh!

KodaiD Nov 28, 2025

Choose a reason for hiding this comment

Uh oh!

KodaiD Nov 28, 2025

Choose a reason for hiding this comment

Uh oh!

KodaiD commented Nov 28, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

komamitsu left a comment

Choose a reason for hiding this comment

Uh oh!

Torch3333 Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

KodaiD commented Nov 28, 2025 •

edited

Loading

Torch3333 Nov 28, 2025 •

edited

Loading