AWS S3 HTTP 400 Bad Request Error - Exception Message Improvement

[brianwyka]

Occasionally (for some reason which we do not know), we get an HTTP 400 error when using DuckDB with httpfs extension and connecting to AWS S3.

HTTP Error: HTTP GET error on 'https://bucket.s3.us-east-1.amazonaws.com/path/year%3D2025/month%3D7/file2.snappy.parquet' (HTTP 400)  Bad Request - this can be caused by the S3 region being set incorrectly. * Provided region is "us-east-1"

Here is the error handling in the code:
https://github.com/duckdb/duckdb-httpfs/blob/main/extension/httpfs/s3fs.cpp#L772

In AWS S3, 400 Bad Request can be for any number of reasons:
https://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html#ErrorCodeList

I'm looking for an improvement to the error message that gets logged to indicate the reason for the HTTP 400 so that further troubleshooting can be performed.

In our use case, we are using DuckDB secret manager for connecting and authenticating, and we set the region and endpoint explicitly...

    CREATE TEMPORARY SECRET s_123 (
        TYPE S3,
        KEY_ID '...',
        SECRET '...',
        SESSION_TOKEN '...',
        REGION 'us-east-1',
        ENDPOINT 's3.us-east-1.amazonaws.com',
        SCOPE 'https://bucket.s3.us-east-1.amazonaws.com/path/',
        URL_STYLE 'vhost',
        USE_SSL true
    );

The request was made to multiple parquet files, to add more context:

SELECT * FROM read_parquet(['s3://bucket/path/year%3D2025/month%3D7/file1.snappy.parquet','s3://bucket/path/year%3D2025/month%3D7/file2.snappy.parquet','s3://bucket/path/year%3D2025/month%3D7/file3.snappy.parquet']);

[brianwyka]

I see you did some work here recently @Mytherin : cb26385