Managed Identity Access not working for delta lake access

[SebastianSchroeder]

I am trying to read a delta lake hosted on an azure storage account into duckdb (1.4.1). I am running the code inside a docker container as a container app job with two managed identities configured. The code is as follows:

def retrieve_data(delta_table_file_path: str) -> DataFrame:
    storage_account = "mystorageaccount"
    container = "mycontainer"
    data_path = f"abfss://{container}@{storage_account}.dfs.core.windows.net/{delta_table_file_path}"
    client_id = os.getenv("SECOND_MANAGED_IDENTITY_ID")

    with duckdb.connect() as connection:
        connection.install_extension("delta")
        connection.load_extension("delta")
        connection.install_extension("azure")
        connection.load_extension("azure")

        connection.execute(f"""
            CREATE SECRET secret1 (
                TYPE AZURE,
                PROVIDER MANAGED_IDENTITY,
                ACCOUNT_NAME '{storage_account}',
                CLIENT_ID '{client_id}'
            );
        """)

        return connection.execute(f"SELECT * FROM delta_scan('{data_path}')").pl()

I am getting the following error on execution (replaced actual paths with their names in the code):

_duckdb.IOException: IO Error: DeltaKernel ObjectStoreError (8): Error interacting with object store: The operation lacked valid authentication credentials for path delta_table_file_path/_delta_log/_last_checkpoint: Error performing GET https://mystorageaccount.blob.core.windows.net/mycontainer/delta_table_file_path/_delta_log/_last_checkpoint in 60.863775ms - Server returned non-2xx status code: 401 Unauthorized: NoAuthenticationInformationServer failed to authenticate the request. Please refer to the information in the www-authenticate header.

It looks like the authentication credentials are not passed correctly. Am I doing something wrong or is this a bug? I also tried to connect using az instead of abfss with no success.

[benfleis]

Hi @SebastianSchroeder, I personally have little experience with delta lake in this kind of configuration, but a couple diagnostic follow ups:

1. Could you set the environment variable: AZURE_LOG_LEVEL=verbose around your py code and paste the results. (They should already be redacted of secrets, but have a look to be sure.)

2. Could you also try CREATE SECRET ( TYPE AZURE, PROVIDER credential_chain )` to allow the default managed identity to authenticate, and share the result? I am curious whether that succeeds to authenticate, even if it fails to authorize. (With verbose logs would be even better!)

I'll also try to find somebody more experienced with Delta Lake to see whether there may be something else at play.

[SebastianSchroeder]

Thank you for your swift response. I added the env variable but that didn't produce any additional logging even though the global log level is set to debug.

I tried the credential_chain provider and got the following error with it:

_duckdb.IOException: IO Error: DeltaKernel ObjectStoreError (8): Error interacting with object store: Generic MicrosoftAzure error: Error performing token request: Error performing GET http://169.254.169.254/metadata/identity/oauth2/token?api-version=2019-08-01&resource=https%3A%2F%2Fstorage.azure.com in 12.818628413s, after 10 retries, max_retries: 10, retry_timeout: 180s - HTTP error: error sending request

An additional info: We used the container app job to access the delta lake via the deltalake library and that worked fine with the managed identity.

[benfleis]

Re the env variable, it was a first hope -- the Azure SDK gives pretty good logs with that, but some tools/wrappers swallow them.

The second error you show suggests something in your container/config -- if it fails to GET at the azure magic IP, there's probably a network issue there.

Can you confirm a working MI in this container some other way? I am trying to rule out container/config issues first, since I had originally tested the specific "second identity" case we're looking at, and it worked correctly in my setup based on the ID I specified.

[SebastianSchroeder]

I just verified that the access via the managed identity works when using this code:

    credential = ManagedIdentityCredential(client_id=client_id)
    service_client = DataLakeServiceClient(f"https://{account_name}.dfs.core.windows.net", credential=credential)

    filesystem_client = service_client.get_file_system_client(filesystem_name)
    directory_client = filesystem_client.get_directory_client(directory_name)

    paths = directory_client.get_paths()
    for path in paths:
        logger.info(path.name)

The logging for the authentication was this:

[2025-10-27 11:51:10,941] {managed_identity.py:151} INFO - ManagedIdentityCredential will use App Service managed identity with client_id: REDACTED
[2025-10-27 11:51:10,943] {managed_identity.py:512} DEBUG - Obtaining token via managed identity on Azure App Service
[2025-10-27 11:51:10,943] {_universal.py:511} INFO - Request URL: 'http://localhost:12356/msi/token?api-version=REDACTED&resource=REDACTED&client_id=REDACTED'

It looks like the wrong identity url is being accessed when using duckdb. A reddit post pointed me to this azure documentation: https://learn.microsoft.com/en-us/azure/app-service/overview-managed-identity?tabs=portal%2Cdotnet#rest-endpoint-reference. When I print the IDENTITY_ENDPOINT env var its value is http://localhost:12356/msi/token.

[benfleis]

Hi @SebastianSchroeder,

To make sure I understand, let me summarize first.

On a single azure VM / MI eligible compute platform with multiple available client IDs:

• a specific (or all?) client ID works with py code above, via the ID URL localhost:12356/msi/token
• the same client ID fails with DuckDB py code that brings in Azure & Delta extensions, with a very different looking ID call to http://169.254.169.254/metadata/identity/oauth2/token?...

The IP above (169.254.169.254) is an azure specific private IP (which I've also seen when testing identity bits), see Azure Instance Metadata Service. AFAICT this is expected for VMs, whereas the localhost:12356 is used for app service, container apps and other fabric compute.

DuckDB Azure isn't explicitly setting the Identity endpoint; it should all be transparently managed by the underlying Azure SDK which we use.

So, one more question and request:

1. Are both examples running on the same VM (or container app, etc.)? Is it possible that the credentials/ACLs/RBACs/etc., are configured differently between the two?

2. Is it possible to demonstrate Authentication & Authorization on a duckdb CLI with the azure plugin, and avoid python & delta for the test? While I see no reason to suspect change in behavior, I do know that different tools can use different pathways. Is that easily managed?

[SebastianSchroeder]

Hi @benfleis ,

I tried to pinpoint where exactly the problem lies and only accessed a single parquet file without the delta extension:

    storage_account = "mystorage"
    data_path = f"abfss://foo@{storage_account}.dfs.core.windows.net/bar/{delta_table_file_name}/foobar.snappy.parquet"
    client_id = os.getenv("MY_CLIENT_ID")

    with duckdb.connect() as connection:
        connection.install_extension("azure")
        connection.load_extension("azure")

        connection.execute(f"""
             CREATE SECRET secret1 (
                TYPE AZURE,
                PROVIDER MANAGED_IDENTITY,
                ACCOUNT_NAME '{storage_account}',
                CLIENT_ID '{client_id}'
            );
        """)

        return connection.execute(f"SELECT * FROM '{data_path}'").pl()

That fails with the following error:

_duckdb.InvalidInputException: Invalid Input Error: Unsupported provider type managed_identity for azure

When I use the chained credential provider:

    with duckdb.connect() as connection:
        connection.install_extension("azure")
        connection.load_extension("azure")

        connection.execute(f"""
             CREATE SECRET secret1 (
                TYPE AZURE,
                PROVIDER credential_chain,
                ACCOUNT_NAME '{storage_account}'
            );
        """)

        return connection.execute(f"SELECT * FROM '{data_path}'").pl()

I get the following error:

_duckdb.IOException: IO Error: AzureBlobStorageFileSystem could not open file: 'abfss://foo@{storage_account}.dfs.core.windows.net/bar/{delta_table_file_name}/foobar.snappy.parquet', unknown error occurred, this could mean the credentials used were wrong. Original error message: 'Failed to get token from ChainedTokenCredential.

To answer your first question. All examples were run on the same container app job with the same env variables and user assigned identities. I'll try the duckdb cli and azure plugin if we cannot find a solution but maybe the new insights with point to the cause of the error.

[benfleis]

Thanks for digging in @SebastianSchroeder, that last one solved it!

Unsupported provider type managed_identity for azure

was the key, and I can see that explicit MI was (incorrectly) only enabled for az://, not for abfss://.

I'll fix and ping you here once it's in. Are you capable of testing a manual/nightly build?

[SebastianSchroeder]

That is great news, thank you! I should be able to test a manual build, let me know when it's ready.

[benfleis]

Hi @SebastianSchroeder ,
the PR #132 just landed. Can you build/test it and see if you're all set?

[SebastianSchroeder]

Hello @benfleis ,
thank you for the fast fix! I can now successfully access a single parquet file on azure blob storage using duckdb.

However, when I try to access the whole delta lake using the delta extension I still get an authentication error:

_duckdb.IOException: IO Error: DeltaKernel ObjectStoreError (8): Error interacting with object store: The operation lacked valid authentication credentials for path foo/bar.delta/_delta_log/_last_checkpoint: Error performing GET https://mystorageaccount.blob.core.windows.net/<container>/foo/bar.delta/_delta_log/_last_checkpoint in 54.69545ms - Server returned non-2xx status code: 401 Unauthorized: <?xml version="1.0" encoding="utf-8"?><Error><Code>NoAuthenticationInformation</Code><Message>Server failed to authenticate the request. Please refer to the information in the www-authenticate header.

Should I address that on the duckdb-delta project or is that a problem of duckdb-azure?

[benfleis]

Hi @SebastianSchroeder, I would definitely suggest posting it there with a backlink to this. My knowledge on our delta module is still limited, and I am surprised that it wouldn't just work after getting this piece to land...

Out of curiosity, have you had success with Delta+Azure and other auth methods (eg service principal or connection strings)? If you can easily test these manually for comparison, it would be useful to check whether it's a MI specific failure, or general auth/delta issue.

[SebastianSchroeder]

Hello @benfleis ,
I tried accessing the delta lake with the current azure and delta extension using a connection string and that worked fine. It seems to be that there is still a managed identity related problem when using the delta extension.

Maybe it has something to do with the delta extension accessing the checkpoints via https as shown in the error message here?