GCS client library migration in Java SDK - part 1 #36876

shunping · 2025-11-22T05:27:41Z

This PR lays the groundwork for the GCS client library migration in the Java SDK. It refactors the existing GCS client implementation to prepare for a new implementation while maintaining backward compatibility.

Key changes include:

The current GCS client implementation has been moved to a new legacy module.
The existing module's public APIs have been mostly preserved and now delegate calls to the new legacy module.
Code and tests have been cleaned up:
- References to the gcsio module, which will be deprecated, have been removed from GcsUtil.java.
- Some references in GcsUtilLegacy.java and GcsOptions.java will be removed in a future PR.
The open API in the legacy module has been rerouted to include previously implemented but unused IO request count metrics, as discussed in [BEAM-11980] Java GCS - Implement IO Request Count metrics #15394 (comment).

… of gcsio in GcsUtil

shunping · 2025-11-22T13:09:42Z

...oogle-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtil.java

 import com.google.api.services.storage.model.StorageObject;
 import com.google.auth.Credentials;
-import com.google.auto.value.AutoValue;
-import com.google.cloud.hadoop.gcsio.CreateObjectOptions;


References to com.google.cloud.hadoop.gcsio.* and com.google.cloud.hadoop.util.* have been removed from GcsUtil.java. They are kept in GcsUtilLegacy.java though during the migration process.

shunping · 2025-11-22T13:12:13Z

...oogle-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtil.java

-  }
-
-  @VisibleForTesting
-  void setCloudStorageImpl(GoogleCloudStorage g) {


While the two setCloudStorageImpl functions are no longer surfaced for testing in GcsUtil.java, we still keep them in GcsUtilLegacy.java.

shunping · 2025-11-22T13:15:51Z

...oogle-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtil.java

    }
  }

-  GoogleCloudStorage createGoogleCloudStorage(


This function is also removed from GcsUtil.java, but will remain in GcsUtilLegacy.java.

shunping · 2025-11-22T13:20:52Z

...cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilLegacy.java

+   * @return a SeekableByteChannel that can read the object data
+   */
+  public SeekableByteChannel open(GcsPath path) throws IOException {
+    return open(path, this.googleCloudStorageOptions.getReadChannelOptions());


Comparing with the previous implementation

beam/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtil.java

Lines 593 to 600 in b20ccbf

public SeekableByteChannel open(GcsPath path) throws IOException {

String bucket = path.getBucket();

SeekableByteChannel channel =

googleCloudStorage.open(

new StorageResourceId(path.getBucket(), path.getObject()),

this.googleCloudStorageOptions.getReadChannelOptions());

return wrapInCounting(channel, bucket);

}

,

we will invoke the open() function below to include the IO request counters similar to the create() functions below.

shunping · 2025-11-22T13:26:37Z

cc'ed @clairemcginty

Context: gcsio is sunsetting (https://s.apache.org/beam-gcsutil-modernization), and we are on the way to migrate to the GCS client library for Beam Java SDK.

…functions.

shunping · 2025-11-22T21:33:26Z

...e-cloud-platform-core/src/test/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilTest.java

 import org.apache.beam.runners.core.metrics.MonitoringInfoMetricName;
 import org.apache.beam.sdk.extensions.gcp.auth.TestCredential;
 import org.apache.beam.sdk.extensions.gcp.options.GcsOptions;
-import org.apache.beam.sdk.extensions.gcp.util.GcsUtil.BatchInterface;


BatchInterface and RewriteOp are only used in Tests, so it is fine to get rid of them in GcsUtil and keep them in GcsUtilLegacy.

github-actions · 2025-11-23T00:33:56Z

Assigning reviewers:

R: @Abacn for label java.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

github-actions · 2025-11-30T12:14:48Z

Reminder, please take a look at this pr: @Abacn

github-actions · 2025-12-03T12:17:13Z

Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:

R: @chamikaramj for label java.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

Abacn · 2025-12-05T16:49:53Z

...cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/options/GcsOptions.java

  @Description(
      "The GoogleCloudStorageReadOptions instance that should be used to read from Google Cloud Storage.")
-  @Default.InstanceFactory(GcsUtil.GcsReadOptionsFactory.class)
+  @Default.InstanceFactory(GcsUtilLegacy.GcsReadOptionsFactory.class)


How about "GcsUtilV1" (and latest version of the class still as GcsUtil)? this follows other branched classes naming convention, and in case they both exist in Beam code base for extended time and a "GcsUtilV3" appear in the future (hopefuly won't be the case).

Abacn · 2025-12-05T16:52:18Z

...cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtilLegacy.java

+@SuppressWarnings({
+  "nullness" // TODO(https://github.com/apache/beam/issues/20497)
+})
+public class GcsUtilLegacy {


shall we make it package private?

Abacn · 2025-12-05T17:04:17Z

...oogle-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtil.java

          storageBuilder.getHttpRequestInitializer(),
          gcsOptions.getExecutorService(),
-          hasExperiment(options, "use_grpc_for_gcs"),
+          org.apache.beam.sdk.options.ExperimentalOptions.hasExperiment(


nit: import hasExperiment

Abacn · 2025-12-05T17:07:29Z

...oogle-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/util/GcsUtil.java


-  LinkedList<RewriteOp> makeRewriteOps(
+  @VisibleForTesting
+  @SuppressWarnings("JdkObsolete")


please add comments for "SuppressWarnings". Here it is // for LinkedList

Also this appears only in higher version of Java. That's likely why it is not captured before

github-actions · 2025-12-13T12:14:51Z

Reminder, please take a look at this pr: @chamikaramj

github-actions · 2025-12-17T12:17:27Z

Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:

R: @ahmedabu98 for label java.

Available commands:

stop reviewer notifications - opt out of the automated review tooling
remind me after tests pass - tag the comment author after tests pass
waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

Abacn · 2025-12-17T15:57:28Z

waiting on author

shunping added 7 commits November 21, 2025 10:02

Make a copy of GcsUtil.java as GcsUtilLegacy.java

f25cb2d

Change references from GcsUtil to GcsUtilLegacy in GcsUtilLegacy.java

7e325ab

Change public apis in GcsUtil.java to call the GcsUtilLegacy.java

eddfddc

Add mocking GcsUtilLegacy for tests and remove unnecessary references…

b57e603

… of gcsio in GcsUtil

Possible breaking change: removed unused public functions.

a78c983

Get rid of the last bit (read option) gcsio references in GcsUtil.java.

62f8d66

Route the code path of open reading files to include api count metric.

8d0bbaa

github-actions bot added java extensions gcp labels Nov 22, 2025

Spotless check

70fcc3f

shunping commented Nov 22, 2025

View reviewed changes

Remove BatchInterface and RewriteOp from GcsUtil.java. Simplify some …

de37131

…functions.

shunping commented Nov 22, 2025

View reviewed changes

shunping marked this pull request as ready for review November 23, 2025 00:14

github-actions bot added the Next Action: Reviewers label Nov 23, 2025

github-actions bot added the slow-review label Nov 30, 2025

github-actions bot added reassigned-reviewers and removed slow-review labels Dec 3, 2025

shunping requested a review from Abacn December 3, 2025 20:59

Abacn reviewed Dec 5, 2025

View reviewed changes

github-actions bot added the slow-review label Dec 13, 2025

github-actions bot removed the slow-review label Dec 17, 2025

github-actions bot added Next Action: Author and removed Next Action: Reviewers labels Dec 17, 2025

	public SeekableByteChannel open(GcsPath path) throws IOException {
	String bucket = path.getBucket();
	SeekableByteChannel channel =
	googleCloudStorage.open(
	new StorageResourceId(path.getBucket(), path.getObject()),
	this.googleCloudStorageOptions.getReadChannelOptions());
	return wrapInCounting(channel, bucket);
	}

GCS client library migration in Java SDK - part 1 #36876

Are you sure you want to change the base?

GCS client library migration in Java SDK - part 1 #36876

Uh oh!

Conversation

shunping commented Nov 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shunping Nov 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shunping Nov 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shunping Nov 22, 2025

Choose a reason for hiding this comment

Uh oh!

shunping Nov 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shunping commented Nov 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

shunping Nov 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Nov 23, 2025

Uh oh!

github-actions bot commented Nov 30, 2025

Uh oh!

github-actions bot commented Dec 3, 2025

Uh oh!

Abacn Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

Abacn Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

Abacn Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

Abacn Dec 5, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Dec 13, 2025

Uh oh!

github-actions bot commented Dec 17, 2025

Uh oh!

Abacn commented Dec 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

shunping commented Nov 22, 2025 •

edited

Loading

shunping Nov 22, 2025 •

edited

Loading

shunping Nov 22, 2025 •

edited

Loading

shunping Nov 22, 2025 •

edited

Loading

shunping commented Nov 22, 2025 •

edited

Loading

shunping Nov 22, 2025 •

edited

Loading