S3 secrets are ignored for the s3a:// prefix

[Miosp]

When using custom S3 (e.g. MinIO - with ENDPOINT defined) and reading data from locations starting with s3a://, the secrets seem to be not used and default access is issued.

With a setup:

CREATE OR REPLACE SECRET secret (
      TYPE s3,
      PROVIDER config,
      ENDPOINT 'minio:9000',
      KEY_ID 'minioadmin',
      SECRET 'minioadmin',
      REGION 'us-east-1',
      URL_STYLE 'path',
      USE_SSL false
  );
--
┌─────────┐
│ Success │
│ boolean │
├─────────┤
│ true    │
└─────────┘
--
SELECT * FROM delta_scan('s3a://warehouse-data/example.db/users');
--
IO Error:
DeltaKernel ObjectStoreError (8): Error interacting with object store: Generic S3 error: Error performing PUT http://169.254.169.254/latest/api/token in 13.612171702s, after 10 retries, max_retries: 10, retry_timeout: 180s  - HTTP error: error sending request

LINE 1: SELECT * FROM delta_scan('s3a://warehouse-data/example.db/users');
                      ^
--
SELECT * FROM delta_scan('s3://warehouse-data/example.db/users');
┌─────────┬─────────────────────┬───────────┬───────────────────┬─────────────┐
│ user_id │        email        │ full_name │ subscription_tier │ signup_date │
│ varchar │       varchar       │  varchar  │      varchar      │   varchar   │
├─────────┼─────────────────────┼───────────┼───────────────────┼─────────────┤
│ user_1  │ user_1@example.com  │ User 1    │ Standard          │ 2023-01-01  │
│ user_2  │ user_2@example.com  │ User 2    │ Standard          │ 2023-01-01  │
│ user_3  │ user_3@example.com  │ User 3    │ VIP               │ 2023-01-01  │
│ user_4  │ user_4@example.com  │ User 4    │ Standard          │ 2023-01-01  │
│ user_5  │ user_5@example.com  │ User 5    │ Standard          │ 2023-01-01  │
│ user_6  │ user_6@example.com  │ User 6    │ VIP               │ 2023-01-01  │
│ user_7  │ user_7@example.com  │ User 7    │ Standard          │ 2023-01-01  │
│ user_8  │ user_8@example.com  │ User 8    │ Standard          │ 2023-01-01  │
│ user_9  │ user_9@example.com  │ User 9    │ VIP               │ 2023-01-01  │
│ user_10 │ user_10@example.com │ User 10   │ Standard          │ 2023-01-01  │
├─────────┴─────────────────────┴───────────┴───────────────────┴─────────────┤
│ 10 rows                                                           5 columns │
└─────────────────────────────────────────────────────────────────────────────┘

When defining global secrets with s3_region, s3_access_key_id..., the s3a:// prefix started to work.

This is mainly an issue when dealing with custom catalogs, like the experimental Unity catalog extension, which will interact with tables created from Hadoop ecosystem, which use s3a for their reasons.

I'm not sure whether this is an issue with the delta extension, but in my experience it happened only there, as with the parquet reader it all worked.