Add support for Cloud Storage #3179

KodaiD · 2025-11-18T01:19:43Z

Description

This PR adds Cloud Storage support to the Object Storage adapter.

Related issues and/or PRs

Changes made

Added an ObjectStorageWrapper implementation for Cloud Storage.
Added unit tests.
Added a GitHub workflow for integration tests.

Checklist

I have commented my code, particularly in hard-to-understand areas.
I have updated the documentation to reflect the changes.
I have considered whether similar issues could occur in other products, components, or modules if this PR is for bug fixes.
Any remaining open issues linked to this PR are documented and up-to-date (Jira, GitHub, etc.).
Tests (unit, integration, etc.) have been added for the changes.
My changes generate no new warnings.
Any dependent changes in other PRs have been merged and published.

Additional notes (optional)

N/A

Release notes

Added Google Cloud Storage adapter.

gemini-code-assist · 2025-11-18T02:50:42Z

It looks like you might be trying to invoke a command. If you'd like me to perform a code review, please use /gemini review. If you'd like a summary of the pull request, please use /gemini summary. You can also type /gemini help to see a list of all available commands.

KodaiD · 2025-11-18T02:52:40Z

...ation-test/java/com/scalar/db/storage/objectstorage/ObjectStorageWrapperIntegrationTest.java

-    assertThatCode(() -> wrapper.update(objectKey, "some-object", "some-version"))
+    assertThatCode(() -> wrapper.update(objectKey, "some-object", "123456789"))


Since Cloud Storage assumes long for their versions, text that is not compatible with long causes the following error:

java.lang.NumberFormatException: For input string: "wrong-version"

KodaiD · 2025-11-18T02:54:34Z

...com/scalar/db/storage/objectstorage/ObjectStorageWrapperLargeObjectWriteIntegrationTest.java

    }

-    char[] charArray = new char[(int) parallelUploadThresholdInBytes];
+    char[] charArray = new char[(int) parallelUploadThresholdInBytes + 1];


The object size has been changed to ensure that the multipart upload was triggered.

KodaiD · 2025-11-18T02:56:02Z

core/src/main/java/com/scalar/db/storage/objectstorage/cloudstorage/CloudStorageWrapper.java

+
+@ThreadSafe
+public class CloudStorageWrapper implements ObjectStorageWrapper {
+  public static final int BATCH_DELETE_SIZE_LIMIT = 100;


You should not include more than 100 calls in a single batch request.

Reference: https://cloud.google.com/storage/docs/batch

KodaiD · 2025-11-18T02:58:37Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces support for Google Cloud Storage as a backend for the Object Storage adapter. The changes include adding a new CloudStorageWrapper, its configuration, and corresponding unit and integration tests. The implementation is comprehensive and follows the existing patterns for other object storage providers. I've found a few issues, including a potential NullPointerException if credentials aren't provided, incorrect exception handling in batch deletes, a copy-paste error in the CloudStorageProvider, and a minor issue in the build configuration. Overall, this is a great addition, and with these fixes, it will be a robust implementation.

core/src/main/java/com/scalar/db/storage/objectstorage/cloudstorage/CloudStorageProvider.java

core/src/main/java/com/scalar/db/storage/objectstorage/cloudstorage/CloudStorageWrapper.java

build.gradle

KodaiD · 2025-11-18T04:16:35Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces support for Google Cloud Storage to the Object Storage adapter, which is a significant and well-executed enhancement. The changes include a new CloudStorageWrapper implementation, its corresponding configuration class, and a service provider. A notable improvement is the refactoring of the ObjectStorageConfig interface, which enhances abstraction and has been consistently applied to the existing Blob Storage and S3 implementations. Error handling has been improved, particularly in the close() methods of ObjectStorage and ObjectStorageAdmin. The addition of comprehensive unit and integration tests for the new Cloud Storage functionality is commendable. Overall, the pull request is of high quality. I have one suggestion to refine the exception handling in the CloudStorageWrapper constructor for better clarity.

core/src/main/java/com/scalar/db/storage/objectstorage/cloudstorage/CloudStorageWrapper.java

Copilot

Pull Request Overview

This PR adds Google Cloud Storage support as a new object storage adapter for ScalarDB, implementing the ObjectStorageWrapper interface to enable Cloud Storage as a backend storage option alongside existing adapters (S3, Azure Blob Storage).

Key Changes:

Implemented CloudStorageWrapper with full CRUD operations using Google Cloud Storage API
Added configuration support via CloudStorageConfig using DatabaseConfig properties
Extended ObjectStorageWrapper interface to support exceptions in close() method and added documentation for deleteByPrefix atomicity constraints

Reviewed Changes

Copilot reviewed 22 out of 23 changed files in this pull request and generated 11 comments.

Show a summary per file

File	Description
CloudStorageWrapper.java	Core implementation of Cloud Storage adapter with get, insert, update, delete operations using Google Cloud Storage client
CloudStorageConfig.java	Configuration class mapping DatabaseConfig properties to Cloud Storage settings (project ID, bucket, credentials)
CloudStorageProvider.java	Service provider implementation for Cloud Storage adapter discovery
CloudStorageErrorCode.java	Enum defining Cloud Storage HTTP error codes for error handling
CloudStorageWrapperTest.java	Comprehensive unit tests for CloudStorageWrapper operations
CloudStorageConfigTest.java	Unit tests for configuration validation and property loading
ObjectStorageWrapper.java	Updated interface with close() exception handling and deleteByPrefix documentation
ObjectStorageWrapperException.java	Added constructor accepting message-only for improved exception handling
PreconditionFailedException.java	Added constructor accepting message-only for simpler exception creation
ObjectStorageConfig.java	Removed getUsername() method from interface (moved to implementation classes)
S3Config.java	Moved getUsername() from interface override to implementation-specific method
BlobStorageConfig.java	Moved getUsername() from interface override to implementation-specific method
ObjectStorageWrapperFactory.java	Added factory support for CloudStorageWrapper instantiation
ObjectStorageUtils.java	Added utility support for CloudStorageConfig creation
ObjectStorage.java	Updated close() to handle ObjectStorageWrapperException with logging
ObjectStorageAdmin.java	Updated close() to handle ObjectStorageWrapperException with logging
ObjectStorageWrapperIntegrationTest.java	Added Cloud Storage support and test data adjustments for numeric version strings
ObjectStorageWrapperLargeObjectWriteIntegrationTest.java	Added Cloud Storage parallel upload configuration and test data size adjustment
ObjectStorageEnv.java	Added isCloudStorage() helper for test environment detection
core/build.gradle	Added google-cloud-storage dependency and formatting improvements
build.gradle	Added googleCloudStorageVersion property (2.60.0)
.github/workflows/object-storage-adapter-check.yaml	Added Cloud Storage integration test workflow and fixed S3 artifact naming
META-INF/services/com.scalar.db.api.DistributedStorageProvider	Registered CloudStorageProvider for service discovery

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

.github/workflows/object-storage-adapter-check.yaml

...com/scalar/db/storage/objectstorage/ObjectStorageWrapperLargeObjectWriteIntegrationTest.java

core/src/main/java/com/scalar/db/storage/objectstorage/cloudstorage/CloudStorageWrapper.java

core/src/main/java/com/scalar/db/storage/objectstorage/ObjectStorageWrapper.java

core/src/main/java/com/scalar/db/storage/objectstorage/cloudstorage/CloudStorageConfig.java

core/src/main/java/com/scalar/db/storage/objectstorage/cloudstorage/CloudStorageProvider.java

core/src/main/java/com/scalar/db/storage/objectstorage/cloudstorage/CloudStorageWrapper.java

KodaiD · 2025-11-18T06:15:17Z

core/src/main/java/com/scalar/db/storage/objectstorage/cloudstorage/CloudStorageConfig.java

+    return projectId;
+  }
+
+  public Credentials getCredentials() {


We should consider a more secure approach for production. I will address it in another PR.

KodaiD · 2025-11-18T06:20:42Z

core/build.gradle

-        systemProperties(System.getProperties().findAll{it.key.toString().startsWith("scalardb")})
+        systemProperties(System.getProperties().findAll { it.key.toString().startsWith("scalardb") })


The formatter adds white spaces. Since other tasks in this file also use this style, I've kept this change.

KodaiD added 2 commits November 18, 2025 09:58

Add Cloud Storage adapter

d8ce1b0

Resolve conflicts

eaee9cc

KodaiD self-assigned this Nov 18, 2025

KodaiD added the enhancement New feature or request label Nov 18, 2025

KodaiD commented Nov 18, 2025

View reviewed changes

gemini-code-assist bot reviewed Nov 18, 2025

View reviewed changes

Fix

2ef8e05

gemini-code-assist bot reviewed Nov 18, 2025

View reviewed changes

core/src/main/java/com/scalar/db/storage/objectstorage/cloudstorage/CloudStorageWrapper.java Outdated Show resolved Hide resolved

KodaiD marked this pull request as ready for review November 18, 2025 04:29

Copilot AI review requested due to automatic review settings November 18, 2025 04:29

Copilot started reviewing on behalf of KodaiD November 18, 2025 04:30 View session

Copilot finished reviewing on behalf of KodaiD November 18, 2025 04:32

Copilot AI reviewed Nov 18, 2025

View reviewed changes

Fix based on review

e4fc125

KodaiD requested review from a team, Torch3333, brfrn169, feeblefakie and komamitsu and removed request for a team November 18, 2025 06:12

KodaiD commented Nov 18, 2025

View reviewed changes

		assertThatCode(() -> wrapper.update(objectKey, "some-object", "some-version"))
		assertThatCode(() -> wrapper.update(objectKey, "some-object", "123456789"))

		systemProperties(System.getProperties().findAll{it.key.toString().startsWith("scalardb")})
		systemProperties(System.getProperties().findAll { it.key.toString().startsWith("scalardb") })

Add support for Cloud Storage #3179

Are you sure you want to change the base?

Add support for Cloud Storage #3179

Conversation

KodaiD commented Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Related issues and/or PRs

Changes made

Checklist

Additional notes (optional)

Release notes

Uh oh!

gemini-code-assist bot commented Nov 18, 2025

Uh oh!

KodaiD Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

KodaiD Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

KodaiD Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

KodaiD commented Nov 18, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

KodaiD commented Nov 18, 2025

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

KodaiD Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

KodaiD Nov 18, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

KodaiD commented Nov 18, 2025 •

edited

Loading