Skip to content

Support GCS vended credentials in Iceberg REST catalog#28423

Open
kaveti wants to merge 2 commits intotrinodb:masterfrom
kaveti:gcp-vended-credentials-support
Open

Support GCS vended credentials in Iceberg REST catalog#28423
kaveti wants to merge 2 commits intotrinodb:masterfrom
kaveti:gcp-vended-credentials-support

Conversation

@kaveti
Copy link

@kaveti kaveti commented Feb 23, 2026

Description

The Iceberg REST catalog spec allows the catalog server to return
file IO properties in the load-table response. These properties can
include cloud-specific credentials and configuration that override
the connector's static settings on a per-table basis as "vended credentials."

Trino already supports vended S3 credentials (access key, secret key,
session token), but GCS properties from the REST catalog response are
currently ignored. This means users connecting to a REST catalog that
vends GCS credentials (e.g., Tabular, Polaris, or custom catalog
implementations) cannot use those credentials for GCS-backed tables.

For example, a REST catalog server managing tables across multiple GCP
projects might vend different gcs.project-id and gcs.oauth2.token
values per table. Without this change, Trino would always use the
statically configured credentials, which may not have access to all
tables.

This change adds support for the following Iceberg GCP properties
from vended credentials:

  • gcs.oauth2.token / gcs.oauth2.token-expires-at — OAuth2 access token
  • gcs.project-id — GCP project ID
  • gcs.service.host — custom storage endpoint
  • gcs.no-auth — skip authentication (e.g., public buckets)
  • gcs.user-project — requester-pays billing project
  • gcs.encryption-key / gcs.decryption-key — client-side encryption keys

These properties are extracted from fileIoProperties in
[IcebergRestCatalogFileSystemFactory] and passed through
ConnectorIdentity extra credentials to the GCS filesystem layer:

  • [GcsStorageFactory] applies no-auth (via NoCredentials) and
    user-project (via setQuotaProjectId) on the StorageOptions
    builder. no-auth takes priority over an OAuth token when both
    are present.
  • [GcsFileSystemFactory] extracts encryption/decryption keys and
    passes them as defaults to [GcsFileSystem], which applies them
    to [newInputFile] and [newOutputFile] operations. Explicit
    encryption keys provided via [newEncryptedInputFile] /
    [newEncryptedOutputFile] still take precedence.

Additional context and related issues

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text:

## Section
* Fix some things. ({issue}`24518 `)

Fixes #24518

@cla-bot cla-bot bot added the cla-signed label Feb 23, 2026
@github-actions github-actions bot added the iceberg Iceberg connector label Feb 23, 2026
@kaveti kaveti force-pushed the gcp-vended-credentials-support branch from 9ffaa0f to 14c29f7 Compare February 23, 2026 14:58
@wendigo wendigo requested review from ebyhr and findinpath February 24, 2026 11:17
Copy link
Contributor

@findinpath findinpath left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your contribution - please focus on adding integration tests for the functionality.

@kaveti kaveti force-pushed the gcp-vended-credentials-support branch 2 times, most recently from 9783451 to 078d848 Compare February 27, 2026 09:54
@kaveti
Copy link
Author

kaveti commented Feb 27, 2026

thank you reviewing the PR. @findinpath i have addressed all your review comments. could you please take a look again.

@kaveti kaveti force-pushed the gcp-vended-credentials-support branch 2 times, most recently from e234f58 to 18beb6a Compare March 2, 2026 10:21
@kekwan
Copy link
Contributor

kekwan commented Mar 4, 2026

Much needed PR for GCS. Thanks @kaveti. PR LGTM, looks in-line with S3 implementation.

@kaveti kaveti force-pushed the gcp-vended-credentials-support branch from 26201f0 to d9b6b17 Compare March 5, 2026 08:48
- Rewrite TestIcebergGcsVendingRestCatalog as TestIcebergGcsVendingRestCatalogConnectorSmokeTest
  extending BaseIcebergConnectorSmokeTest with real GCS credentials
- Create IcebergGcsRestCatalogBackendContainer for GCS-specific REST catalog backend
- Rename IcebergRestCatalogBackendContainer to IcebergS3RestCatalogBackendContainer
- Remove old test class and GcsCredentialVendingCatalogAdapter
- Remove default encryption/decryption key mechanism from GcsFileSystem and GcsFileSystemFactory
  to align with S3FileSystem and AzureFileSystem (only per-call newEncrypted* methods)
- Remove encryption key constants from GcsFileSystemConstants
- Remove encryption key mapping from IcebergRestCatalogFileSystemFactory
- Add new GCS vending test to cloud-tests profile in pom.xml
@kaveti kaveti force-pushed the gcp-vended-credentials-support branch from d9b6b17 to d382c1f Compare March 5, 2026 10:25
@kaveti
Copy link
Author

kaveti commented Mar 5, 2026

@findinpath could you please review again. i have addressed your review comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla-signed iceberg Iceberg connector

Development

Successfully merging this pull request may close these issues.

Iceberg REST Catalog: Support for vended credentials for GCP

3 participants