delta_scan ignores GCS secrets and uses anonymous access

[ik8]

Problem

delta_scan ignores properly configured GCS secrets and attempts anonymous access, while read_parquet with the same secrets works correctly.

Environment

• DuckDB version: 1.4.1
• duckdb-delta extension: Latest
• Platform: macOS
• Authentication: Google Cloud Application Default Credentials + Bearer Token

Working vs Not Working

✅ WORKS: read_parquet with GCS secret

import duckdb
import subprocess

# Get access token
result = subprocess.run(['gcloud', 'auth', 'application-default', 'print-access-token'], 
                       capture_output=True, text=True)
access_token = result.stdout.strip()

con = duckdb.connect()
con.execute("INSTALL httpfs")
con.execute("LOAD httpfs")
con.execute("DROP SECRET IF EXISTS gcs_secret")
con.execute(f"CREATE SECRET gcs_secret (TYPE GCS, bearer_token '{access_token}')")

# This works perfectly
result = con.execute("SELECT count(*) FROM read_parquet('gs://bucket/path/*.parquet')").fetchone()
print(f"Success: {result[0]} rows")

❌ FAILS: delta_scan with same GCS secret

# Same setup as above, but using delta_scan
con.execute("INSTALL delta")
con.execute("LOAD delta")

# This fails with anonymous access error
result = con.execute("SELECT count(*) FROM delta_scan('gs://bucket/path/delta-table')").fetchone()

Error Message

IO Error: DeltaKernel ObjectStoreError (8): Error interacting with object store: 
The operation lacked the necessary privileges to complete for path [...]: 
Error performing GET https://storage.googleapis.com/[...] - Server returned 
non-2xx status code: 403 Forbidden: Access denied. Anonymous caller does not 
have storage.objects.get access to the Google Cloud Storage object.

Key Observations

1. GCS secret is created successfully - verified with SELECT * FROM duckdb_secrets()
2. Individual file access works - read_parquet uses the secret correctly
3. Delta table structure is valid - gsutil ls gs://bucket/path/_delta_log/ shows proper Delta log files
4. Authentication is valid - gsutil and gcloud commands work fine
5. delta_scan attempts anonymous access - ignores the configured secret

Related Issues

• Similar to duckdb 1.1 delta reader ignore azure secret  #83 (Azure secrets ignored by delta_scan)
• Different from register_filesystem has no effect after v1.1.0 upgrade #87 (which is about filesystem registration)

Expected Behavior

delta_scan should use the configured GCS secret for authentication, just like read_parquet does.

[sumanthssr]

seeing same issue with AKS, read_parquet is working where delta_scan throwing 403 error, any update on this issue?

[blieusong]

Hi, I have the same issue. We cannot create HMAC keys so GCS secret support would be a welcome feature for the delta extension.

D CREATE SECRET (TYPE GCS, bearer_token 'xxx');
┌─────────┐
│ Success │
│ boolean │
├─────────┤
│ true    │
└─────────┘
D SELECT * FROM delta_scan('gs://my-beautiful-bucket/my_table/delta');
IO Error:
DeltaKernel ObjectStoreError (8): Error interacting with object store: The operation lacked the necessary privileges to complete for path my_table/delta/_delta_log/_last_checkpoint: Error performing GET https://storage.googleapis.com/my-beautiful-bucket/my_table/delta/_delta_log/_last_checkpoint in 169.550931ms - Server returned non-2xx status code: 403 Forbidden: <?xml version='1.0' encoding='UTF-8'?><Error><Code>AccessDenied</Code><Message>Access denied.</Message><Details>Anonymous caller does not have storage.objects.get access to the Google Cloud Storage object. Permission 'storage.objects.get' denied on resource (or it may not exist).</Details></Error>

• duckdb version 1.4.4
• duckdb delta extension version 50de511

Cheers