[Need Help!] dockercompose - MinIO with duckdb on Cube

[ClorisYT]

I encountered an issue deploying the Cube.js service using Docker Compose on a virtual machine (VM). The service fails to start correctly, and no error logs indicate the root cause of the failure.

Environment Details:
Service: Cube.js (cubejs/cube:latest)
Environment: Virtual Machine (VM)

services:
  cube:
    image: cubejs/cube:latest
    ports:
      - 4001:4000

    environment:
      - CUBEJS_DEV_MODE=true
  
      - CUBEJS_DB_TYPE=duckdb
      - CUBEJS_DB_DUCKDB_S3_REGION='us-east-1'
      - CUBEJS_DB_DUCKDB_S3_ACCESS_KEY_ID=xxxx
      - CUBEJS_DB_DUCKDB_S3_SECRET_ACCESS_KEY=xxx
      - CUBEJS_DB_DUCKDB_S3_ENDPOINT=http://xxxxx
      - CUBEJS_DB_DUCKDB_S3_USE_SSL=false
      - CUBEJS_DB_TYPE=duckdb

    volumes:
      - ./cubestore:/cube/data

I am trying to retrieve data from MinIO using DuckDB as an in-memory database, and then display the data schema on Cube.
I have only tested placing a parquet file in my MinIO bucket, but I can't seem to start the service.

So I need someone to help me deal with it or advise me. Thank you.

[igorlukanin]

Does the solution from this issue help?

[ClorisYT]

Does the solution from this issue help?

Thank you for your response.
However, this issue discussion didn’t help, as I had already reviewed that solution but couldn't retrieve data from my MinIO instance. As I mentioned, only one Parquet file is in the bucket, yet Cube cannot read it. I suspect this is an issue with DuckDB, but even though I am very familiar with it, I haven't found a solution.🥲

For now, the only workaround is to write a Python script that uses an in-memory DuckDB instance to read data from MinIO, export it as a CSV file, and then load it into PostgreSQL. Finally, I configured the PostgreSQL environment in Docker Compose.

Let me know if you have any other suggestions. Thanks!

[ClorisYT]

I'm sharing an update regarding my progress on accessing MinIO files through DuckDB. I overlooked the correct approach earlier but found a solution demonstrated in the Cube.js official data model demo video.

To make it work, I configured the following environment variables in the docker-compose file:

• CUBEJS_DB_DUCKDB_S3_REGION
• CUBEJS_DB_DUCKDB_S3_ACCESS_KEY_ID
• CUBEJS_DB_DUCKDB_S3_SECRET_ACCESS_KEY
• CUBEJS_DB_DUCKDB_S3_ENDPOINT
• CUBEJS_DB_DUCKDB_S3_USE_SSL

Then, in the Cube.js data model YAML file, before defining dimensions and measures, I added:

cubes:  
  - name: xxx  
    sql: SELECT * FROM 's3://{bucket}//.csv'  
    public: false

This follows DuckDB’s syntax for accessing files in a MinIO bucket and successfully reads the data.

However, I encountered an HTTPS-related issue when installing the DuckDB httpfs extension. It is a long-standing unresolved issue (it is still open in their GitHub discussions).

[igorlukanin]

However, I encountered an HTTPS-related issue when installing the DuckDB httpfs extension. It is a long-standing unresolved issue (it is still open in their GitHub discussions).

Could you please link it here?

[ClorisYT]

However, I encountered an HTTPS-related issue when installing the DuckDB httpfs extension. It is a long-standing unresolved issue (it is still open in their GitHub discussions).

Could you please link it here?

https://github.com/duckdb/duckdb/issues/9378，here actually.