Iceberg Azure credential vending

[smaheshwar-pltr]

Description

// Work in progress

Support Azure credential vending with Iceberg REST catalog.

Additional context and related issues

Closes #23238

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
( ) Release notes are required. Please propose a release note for me.
( ) Release notes are required, with the following suggested text:

## Section
* Fix some things. ({issue}`issuenumber`)

Summary by Sourcery

Enable Azure credential vending by extracting SAS tokens from file IO properties for Iceberg REST catalog and injecting them into the Azure filesystem via a new AzureVendedAuth wrapper.

New Features:

• Support Azure SAS token vending in Iceberg REST catalog filesystem factory
• Introduce AzureVendedAuth in AzureFileSystemFactory to apply per-account vended SAS tokens with fallback to default authentication

[cla-bot]

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Sreesh Maheshwar.
This is most likely caused by a git client misconfiguration; please make sure to:

1. check if your git client is configured with an email to sign commits git config --list | grep email
2. If not, set it up using git config --global user.email [email protected]
3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

[sourcery-ai]

Reviewer's Guide

This PR extends Iceberg REST catalog support to vend Azure SAS token credentials alongside existing S3 credential vending by unifying the credential selection logic, propagating vended credentials through the Azure filesystem, and introducing a new AzureVendedAuth implementation.

Class diagram for updated Azure authentication classes

classDiagram
    class AzureAuth {
        <<interface>>
        +void setAuth(String storageAccount, BlobContainerClientBuilder builder)
        +void setAuth(String storageAccount, DataLakeServiceClientBuilder builder)
    }
    class AzureAuthAccessKey
    class AzureAuthDefault
    class AzureAuthOauth
    class AzureVendedAuth {
        +Map<String, String> accountSasTokens
        +AzureAuth fallbackAuth
        +AzureVendedAuth(Map<String, String> accountSasTokens, AzureAuth fallbackAuth)
        +void setAuth(String storageAccount, BlobContainerClientBuilder builder)
        +void setAuth(String storageAccount, DataLakeServiceClientBuilder builder)
    }
    AzureAuth <|.. AzureAuthAccessKey
    AzureAuth <|.. AzureAuthDefault
    AzureAuth <|.. AzureAuthOauth
    AzureAuth <|.. AzureVendedAuth
    AzureVendedAuth --> AzureAuth : fallbackAuth

Loading

Class diagram for AzureFileSystemFactory authentication selection

classDiagram
    class AzureFileSystemFactory {
        +TrinoFileSystem create(ConnectorIdentity identity)
        -static AzureAuth withVendedAuth(ConnectorIdentity identity, AzureAuth defaultAuth)
    }
    class AzureAuth
    class AzureVendedAuth
    AzureFileSystemFactory --> AzureAuth
    AzureFileSystemFactory --> AzureVendedAuth

Loading

File-Level Changes

Change	Details	Files
Unify vended credential logic in IcebergRestCatalogFileSystemFactory and add Azure SAS token support	Replaced S3-only credential block with a builder handling both S3 access keys and Azure SAS tokens Added getAzureCredentials helper to extract per-account SAS tokens from properties Adjusted identity creation to use overridden credentials when available	plugin/trino-iceberg/src/main/java/io/trino/plugin/iceberg/catalog/rest/IcebergRestCatalogFileSystemFactory.java
Propagate vended credentials into Azure filesystem factory	Wrapped default AzureAuth with withVendedAuth in AzureFileSystemFactory.create Implemented withVendedAuth to choose between fallback and vended credentials from ConnectorIdentity	lib/trino-filesystem-azure/src/main/java/io/trino/filesystem/azure/AzureFileSystemFactory.java
Introduce AzureVendedAuth implementation and related constants	Extended AzureAuth sealed interface to permit AzureVendedAuth Added AzureVendedAuth class to apply SAS token or fallback auth per storage account Defined AzureFileSystemConstants.EXTRA_SAS_TOKEN_PROPERTY_PREFIX for SAS token keys	lib/trino-filesystem-azure/src/main/java/io/trino/filesystem/azure/AzureAuth.java lib/trino-filesystem-azure/src/main/java/io/trino/filesystem/azure/AzureVendedAuth.java lib/trino-filesystem-azure/src/main/java/io/trino/filesystem/azure/AzureFileSystemConstants.java

---

Tips and commands

Interacting with Sourcery

• Trigger a new review: Comment @sourcery-ai review on the pull request.
• Continue discussions: Reply directly to Sourcery's review comments.
• Generate a GitHub issue from a review comment: Ask Sourcery to create an
  issue from a review comment by replying to it. You can also reply to a
  review comment with @sourcery-ai issue to create an issue from it.
• Generate a pull request title: Write @sourcery-ai anywhere in the pull
  request title to generate a title at any time. You can also comment
  @sourcery-ai title on the pull request to (re-)generate the title at any time.
• Generate a pull request summary: Write @sourcery-ai summary anywhere in
  the pull request body to generate a PR summary at any time exactly where you
  want it. You can also comment @sourcery-ai summary on the pull request to
  (re-)generate the summary at any time.
• Generate reviewer's guide: Comment @sourcery-ai guide on the pull
  request to (re-)generate the reviewer's guide at any time.
• Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
  pull request to resolve all Sourcery comments. Useful if you've already
  addressed all the comments and don't want to see them anymore.
• Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
  request to dismiss all existing Sourcery reviews. Especially useful if you
  want to start fresh with a new review - don't forget to comment
  @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

• Enable or disable review features such as the Sourcery-generated pull request
  summary, the reviewer's guide, and others.
• Change the review language.
• Add, remove or edit custom review instructions.
• Adjust other review settings.

Getting Help

• Contact our support team for questions or feedback.
• Visit our documentation for detailed guides and information.
• Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

[cla-bot]

Thank you for your pull request and welcome to our community. We could not parse the GitHub identity of the following contributors: Sreesh Maheshwar.
This is most likely caused by a git client misconfiguration; please make sure to:

1. check if your git client is configured with an email to sign commits git config --list | grep email
2. If not, set it up using git config --global user.email [email protected]
3. Make sure that the git commit email is configured in your GitHub account settings, see https://github.com/settings/emails

[smaheshwar-pltr]

Introducing this new auth means that with this PR,

trino/lib/trino-filesystem-azure/src/main/java/io/trino/filesystem/azure/AzureFileSystem.java

Lines 482 to 486 in d20c663

	return switch (azureAuth) {
	case AzureAuthOauth _ -> oauth2PresignedUri(location, ttl, Optional.empty());
	case AzureAuthAccessKey _ -> accessKeyPresignedUri(location, ttl, Optional.empty());
	default -> throw new UnsupportedOperationException("Unsupported azure auth: " + azureAuth);
	};

throws.

Maybe this is fine for now, as I see it's not supported even for default auth?

[smaheshwar-pltr]

Also highlighting that I'm not adding this type to

trino/lib/trino-filesystem-azure/src/main/java/io/trino/filesystem/azure/AzureFileSystemConfig.java

Lines 27 to 32 in d20c663

	public enum AuthType
	{
	ACCESS_KEY,
	OAUTH,
	DEFAULT,
	}

I think that this should be an internal auth type that cannot enabled by config. Though I appreciate this design of an internal, IRC-centred auth may be controversial.

[smaheshwar-pltr]

Setting sasToken with fallback in these flows aims to mirror https://github.com/apache/iceberg/blob/acd3e4657d2f4e6bfe4d092fa88dd362bb50fedc/azure/src/main/java/org/apache/iceberg/azure/AzureProperties.java#L160-L181

[ebyhr]

Please add a test.

[smaheshwar-pltr]

@ebyhr yes, sounds good! (Currently, still a WIP and completely untested)

While I have you here, may you elaborate on what sort of tests you'd like for this PR? I see TestIcebergTrinoRestCatalogConnectorSmokeTest and TestIcebergPolarisCatalogConnectorSmokeTest but not sure how doable wiring up such a test suite with an Azure storage IRC is.

[smaheshwar-pltr]

Ping on the above @ebyhr 🙏

[eladitzhakian]

This can be very useful for us. How can I help pushing this?

[chenjian2664]

@smaheshwar-pltr Hi, since you’re adding credential vending support for the Iceberg REST catalog, the test can refer to the ICEBERG_REST or ICEBERG_AZURE group in the product tests.
You may need to add an Iceberg REST test with Azure and enable vending without configuring a sas token to showcase the behavior

[cla-bot]

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

[cla-bot]

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

[cla-bot]

Thank you for your pull request and welcome to the Trino community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. Continue to work with us on the review and improvements in this PR, and submit the signed CLA to [email protected]. Photos, scans, or digitally-signed PDF files are all suitable. Processing may take a few days. The CLA needs to be on file before we merge your changes. For more information, see https://github.com/trinodb/cla

[smaheshwar-pltr]

Thanks, @chenjian2664. I feel like we should be able to wire up the REST fixture with the Azure storage used in CI to get something similar to TestIcebergVendingRestCatalogConnectorSmokeTest that can test this. You'd basically have something like (using syntax from #27417)

public IcebergAzureRestCatalogBackendContainer(
        Optional<Network> network,
        String warehouseLocation,
        String accountName,
        String domainName,
        String sasToken)
{
    super(
            "tabulario/iceberg-rest:1.6.0",
            "iceberg-rest",
            ImmutableSet.of(8181),
            ImmutableMap.of(),
            toCatalogEnvVars(ImmutableMap.of(
                    "include-credentials", "true",
                    "warehouse", warehouseLocation,
                    "io-impl", "org.apache.iceberg.azure.adlsv2.ADLSFileIO",
                    "adls.sas-token." + accountName + "." + domainName, sasToken)),
            network,
            5);
}

Can dig into this, though may be slow due to maybe having to iterate with CI.

[smaheshwar-pltr]

@ebyhr yes, sounds good! (Currently, still a WIP and completely untested)

(Update from me is that I did get around to smoke testing this PR with a remote catalog and it works)

Edit: I only tested the read path, from trying to wire up integration tests in #27427, I realised that the write path doesn't work because of the HNS check done with a SAS token.