Skip to content

Endpoint with custom port requires legacy configuration #129

@james-aus

Description

@james-aus

When working with some self-hosted S3 providers (such as Garage), it can be useful to set a custom port.

Using what I think is the recommended way of doing things, I can create the default secret and try to use it like so:

create or replace secret (type s3, provider config, key_id 'abc123', secret 'abc123', region 'garage',
        endpoint 'somehost:3900', url_style 'path', use_ssl 'false';
select count(*) from read_parquet(['s3://bucket/Example/somefile.parquet']);
IO Error:
Could not establish connection error for HTTP HEAD to 'http://somehost:3900/bucket/Example/somefile.parquet'

LINE 1: select count(*) from read_parquet(['s3://bucket/Example/somefile...
                             ^

Digging deeper, I could see no attempt to connect in my logs and I believe that the port number is not being split from the hostname -- but I'm not sure of that because I'm unfamiliar with the AWS SDK and haven't spent a lot of time reading the code.

I did notice however, that running drop secret __default_s3 (or restarting the DuckDB CLI -- just unconfiguring what we've just done) then configuring instead:

set s3_region = 'garage';
set s3_endpoint = 'somehost:3900';
set s3_use_ssl = false;
set s3_url_style = 'path';
set s3_access_key_id = 'abc123';
set s3_secret_access_key = 'abc123';

Allows the previous count(*) command to work correctly.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions