Skip to content

s3 secret with PATH does not work #153

@admin-one211

Description

@admin-one211

What happens?

CREATE SECRET minio_secret (
            TYPE s3,
            KEY_ID 'minioadmin',
            SECRET 'minioadmin',
            ENDPOINT 'minio:9000',
            USE_SSL false,
            URL_STYLE PATH
        );

The above command with minio gives following error
select * from read_parquet('s3://org1-test-bucket/data/dt=2025/data.parquet');
HTTP Error:
Unable to connect to URL "http://minio:9000/data/dt%3D2025/data.parquet": 404 (Not Found).

Exact command to create secret which works is

CREATE SECRET minio_secret (
        TYPE s3,
        KEY_ID 'minioadmin',
        SECRET 'minioadmin',
        ENDPOINT 'minio:9000',
        USE_SSL false,
        URL_STYLE "path"
    );

Note the difference in URL_STYLE --> "path" vs PATH

To Reproduce

Run minio container with docker-compose.yml

version: "3.9"
services:
  minio:
    image: minio/minio
    ports:
      - "9000:9000" # MinIO API port
      - "9001:9001" # MinIO Console port (default 9001 for newer versions)
    volumes:
      - minio_data:/data # Mount a named volume for persistent data storage
    environment:
      MINIO_ROOT_USER: minioadmin # Set your desired root username
      MINIO_ROOT_PASSWORD: minioadmin # Set your desired root password
      MINIO_SERVER_URL: http://localhost:9000 # Optional: Define the server URL
    command: server /data --console-address ":9001" # Start MinIO server and specify console address
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3

  create_minio_buckets:
    image: minio/mc
    depends_on:
      - minio
    restart: on-failure
    entrypoint: >
      /bin/sh -c "
      sleep 2;
      /usr/bin/mc alias set dockerminio http://minio:9000 minioadmin minioadmin;
      /usr/bin/mc mb dockerminio/org1-test-bucket;
      /usr/bin/mc mb dockerminio/org1-test-bucket/data;
      /usr/bin/mc mb dockerminio/org2-test-bucket;
      /usr/bin/mc mb dockerminio/org2-test-bucket/data;
      exit 0;
      "

docker-compose up -d

SELECT version();
┌─────────────┐
│ "version"() │
│   varchar   │
├─────────────┤
│ v1.4.1      │
└─────────────┘

CREATE SECRET minio_secret (
            TYPE s3,
            KEY_ID 'minioadmin',
            SECRET 'minioadmin',
            ENDPOINT 'localhost:9000',
            USE_SSL false,
            URL_STYLE PATH
        );
COPY
      (SELECT * FROM VALUES('john', 'US', 1000),('jack', 'UK', 10000) AS my_values(name, country, amount))
      TO 's3://org1-test-bucket/data/dt=2025/data.parquet'
      (FORMAT parquet);
HTTP Error:
Unable to connect to URL http://minio:9000/data/dt%3D2025/data.parquet: Not Found (HTTP code 404)

OS:

mac

DuckDB Version:

1.4.1

DuckDB Client:

cli

Hardware:

No response

Full Name:

gagan taneja

Affiliation:

one211.com

Did you include all relevant configuration (e.g., CPU architecture, Linux distribution) to reproduce the issue?

  • Yes, I have

Did you include all code required to reproduce the issue?

  • Yes, I have

Did you include all relevant data sets for reproducing the issue?

No - Other reason (please specify in the issue body)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions