Excessive ALTS handshake logs from gke-metadata-server when using Java GCS client with gRPC

[takaaki7]

Description

We are observing an extremely high volume of informational logs (approximately 500,000 to 1,000,000 per day) from the gke-metadata-server container on our GKE cluster.

This appears to be triggered by our Java application that uses the google-cloud-storage library with the gRPC transport option to interact with Google Cloud Storage. Our application handles a high volume of requests, writing to GCS approximately 1 million times per day, with peaks of up to 1,000 writes per second.

The excessive logging is causing a significant increase in costs for our monitoring and logging tool (Datadog).

The most frequent log messages are related to ALTS session handshakes:

[rpc-id:b0b2453b42e5f6b2] Closing ALTS session from pod "ml-test-common-historical-1".
[rpc-id:b0b2453b42e5f6b2] Use node identity as channel identity to connect to service: "storage.googleapis.com"
[rpc-id:b0b2453b42e5f6b2] Starting ALTS session from pod "ml-test-common-historical-1" running as test-common/ml-test-common-app to act as ml-test-common@evaluation-mila.iam.gserviceaccount.com
[rpc-id:b0b2453b42e5f6b2] Processing ALTS handshake request from "10.80.12.18:53498".

Given the high traffic, we suspect that the client is creating new ALTS sessions too frequently, rather than reusing existing ones, which results in this verbose logging.

Environment

• google-cloud-storage version: 2.55.0
• Transport: gRPC (StorageOptions.grpc())
• Environment: Google Kubernetes Engine (GKE)
• GKE Cluster Version: 1.33.4-gke.1036000
• Workload Identity Enabled: Yes
• Java Version: 24

Code Snippet

Our Storage client is initialized as a singleton and reused throughout the application's lifecycle. The following code demonstrates our usage pattern.

// This class is a singleton
class MyClass {
  final Storage storage;

  public MyClass() {
    // Client is initialized once using gRPC transport
    this.storage = StorageOptions.grpc()
        .setProjectId("your-gcp-project-id")
        .build()
        .getService();
  }

  @Override
  public void saveData(String dataId, ByteBuffer data) throws Exception {
    BlobInfo blobInfo = toBlobInfo(dataId);
    Blob blob = this.storage.create(blobInfo, BlobTargetOption.doesNotExist());
    try (WriteChannel wc = blob.writer()){
      wc.write(data);
    }
  }
}

Investigation So Far

We opened a support case with the Google Kubernetes Engine team. Their recommendation was to "adjust the client library's connection pool to prevent frequent session creation."

However, it is not clear from the google-cloud-storage or google-api-grpc documentation how to configure or tune the underlying gRPC connection pool for this purpose.

[BenWhitehead]

When using the gRPC transport to connect to GCS[1], a full client side load balancer is maintained. The specific backends connected to changes over time based on the load those servers experience. This client side load balancer will initialize connections to the backends returned from the service discovery subsystem (xDS), so they are ready to use when an RPC needs to be serviced. The service discovery subsystem is the one that returns the set of servers, and refreshes on a periodic basis.

ALTS is the transport security used between the client and servers instead of regular TLS.

To reduce logging related to the xDS and ALTS subsystems you can set the following logging levels in logback.xml (or equivelent logging config)

  <logger name="io.grpc.googleapis.GoogleCloudToProdNameResolver" level="warn"/>
  <logger name="io.grpc.internal.JndiResourceResolverFactory$JndiResourceResolver" level="warn"/>
  <logger name="io.grpc.util.MultiChildLoadBalancer" level="warn"/>
  <logger name="io.grpc.xds.WeightedRoundRobinLoadBalancer" level="warn"/>
  <logger name="io.grpc.xds.XdsCredentialsRegistry" level="warn"/>
  <logger name="io.grpc.xds.XdsLogger" level="warn"/>

• [1] https://cloud.google.com/storage/docs/enable-grpc-api
• [2] https://cloud.google.com/docs/security/encryption-in-transit/application-layer-transport-security

[takaaki7]

We run this app on GKE. gke-metadata-server who logs this is system resource, and we cannot modify logging configuration...

I suspect that this issue might be resolved by adjusting the client library's session pool settings.
Are there any configurations for this?

[BenWhitehead]

Thanks for clarifying where the logging originates.

Though, as noted previously the refreshing and subset of servers returned is a live thing returned by xDS and is not a bootstrap configurable item.

https://docs.datadoghq.com/agent/logs/advanced_log_collection/?tab=kubernetes#exclude-at-match seems to provide a suggestion on how these log messages might be excluded from datadog ingestion.

[BenWhitehead]

A colleague mentioned to me today, you might be able to apply the filtering at the GKE level. Here's what they shared with me:

For the 'ad hoc' log exclusion filter to filter out the ALTS handshake logs, they can try using https://cloud.google.com/kubernetes-engine/docs/how-to/exclude-logs#add-filter , by which they can filter out logs by something like the query below:

resource.labels.namespace_name="kube-system"
resource.type="k8s_container"
resource.labels.container_name="gke-metadata-server"
sourceLocation.file="altsproxy.go"