Skip to content

feat(duckdb-driver): Add support for using default credential provider chain for duckdb s3 access #9679

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jun 18, 2025
Merged

feat(duckdb-driver): Add support for using default credential provider chain for duckdb s3 access #9679

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
d2b587b
use default credential provider chain if access key and secret key ar…
GabeInDevOps Nov 7, 2024
0ec4ad1
Merge branch 'feat/duckdb-credential-provider-chain' of github.com:Ga…
KSDaemon Jun 13, 2025
e4a8ecf
some polishment
KSDaemon Jun 13, 2025
ee9ba7b
update docs
KSDaemon Jun 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 17 additions & 16 deletions docs/pages/product/configuration/data-sources/duckdb.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,22 +59,23 @@ deployment][ref-demo-deployment] in Cube Cloud.

## Environment Variables

| Environment Variable | Description | Possible Values | Required |
| --------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------- | :------: |
| `CUBEJS_DB_DUCKDB_MEMORY_LIMIT` | The maximum memory limit for DuckDB. Equivalent to `SET memory_limit=<MEMORY_LIMIT>`. Default is 75% of available RAM | A valid memory limit | ❌ | ✅ |
| `CUBEJS_DB_DUCKDB_SCHEMA` | The [default search schema][link-duckdb-configuration-ref] | A valid schema name | ❌ | ✅ |
| `CUBEJS_DB_DUCKDB_MOTHERDUCK_TOKEN` | The service token to use for connections to MotherDuck | A valid [MotherDuck service token][motherduck-docs-svc-token] | ❌ | ✅ |
| `CUBEJS_DB_DUCKDB_DATABASE_PATH` | The database filepath to use for connection to a local database. | A valid duckdb database file path | ❌ | ✅ |
| `CUBEJS_DB_DUCKDB_S3_ACCESS_KEY_ID` | The Access Key ID to use for database connections | A valid Access Key ID | ❌ | ✅ |
| `CUBEJS_DB_DUCKDB_S3_SECRET_ACCESS_KEY` | The Secret Access Key to use for database connections | A valid Secret Access Key | ❌ | ✅ |
| `CUBEJS_DB_DUCKDB_S3_ENDPOINT` | The S3 endpoint | A valid [S3 endpoint][duckdb-docs-s3-import] | ❌ | ✅ |
| `CUBEJS_DB_DUCKDB_S3_REGION` | The [region of the bucket][duckdb-docs-s3-import] | A valid AWS region | ❌ | ✅ |
| `CUBEJS_DB_DUCKDB_S3_USE_SSL` | Use SSL for connection | A boolean | ❌ | ❌ |
| `CUBEJS_DB_DUCKDB_S3_URL_STYLE` | To choose the S3 URL style(vhost or path) | 'vhost' or 'path' | ❌ | ❌ |
| `CUBEJS_DB_DUCKDB_S3_SESSION_TOKEN` | The token for the S3 session | A valid Session Token | ❌ | ✅ |
| `CUBEJS_DB_DUCKDB_EXTENSIONS` | A comma-separated list of DuckDB extensions to install and load | A comma-separated list of DuckDB extensions | ❌ | ✅ |
| `CUBEJS_DB_DUCKDB_COMMUNITY_EXTENSIONS` | A comma-separated list of DuckDB community extensions to install and load | A comma-separated list of DuckDB community extensions | ❌ | ✅ |
| `CUBEJS_CONCURRENCY` | The number of [concurrent queries][ref-data-source-concurrency] to the data source | A valid number | ❌ |
| Environment Variable | Description | Possible Values | Required |
|--------------------------------------------|-----------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------|:--------:|
| `CUBEJS_DB_DUCKDB_MEMORY_LIMIT` | The maximum memory limit for DuckDB. Equivalent to `SET memory_limit=<MEMORY_LIMIT>`. Default is 75% of available RAM | A valid memory limit | ❌ |
| `CUBEJS_DB_DUCKDB_SCHEMA` | The [default search schema][link-duckdb-configuration-ref] | A valid schema name | ❌ |
| `CUBEJS_DB_DUCKDB_MOTHERDUCK_TOKEN` | The service token to use for connections to MotherDuck | A valid [MotherDuck service token][motherduck-docs-svc-token] | ❌ |
| `CUBEJS_DB_DUCKDB_DATABASE_PATH` | The database filepath to use for connection to a local database. | A valid duckdb database file path | ❌ |
| `CUBEJS_DB_DUCKDB_S3_ACCESS_KEY_ID` | The Access Key ID to use for database connections | A valid Access Key ID | ❌ |
| `CUBEJS_DB_DUCKDB_S3_SECRET_ACCESS_KEY` | The Secret Access Key to use for database connections | A valid Secret Access Key | ❌ |
| `CUBEJS_DB_DUCKDB_S3_ENDPOINT` | The S3 endpoint | A valid [S3 endpoint][duckdb-docs-s3-import] | ❌ |
| `CUBEJS_DB_DUCKDB_S3_REGION` | The [region of the bucket][duckdb-docs-s3-import] | A valid AWS region | ❌ |
| `CUBEJS_DB_DUCKDB_S3_USE_SSL` | Use SSL for connection | A boolean | ❌ |
| `CUBEJS_DB_DUCKDB_S3_URL_STYLE` | To choose the S3 URL style(vhost or path) | `vhost` or `path` | ❌ |
| `CUBEJS_DB_DUCKDB_S3_SESSION_TOKEN` | The token for the S3 session | A valid Session Token | ❌ |
| `CUBEJS_DB_DUCKDB_EXTENSIONS` | A comma-separated list of DuckDB extensions to install and load | A comma-separated list of DuckDB extensions | ❌ |
| `CUBEJS_DB_DUCKDB_COMMUNITY_EXTENSIONS` | A comma-separated list of DuckDB community extensions to install and load | A comma-separated list of DuckDB community extensions | ❌ |
| `CUBEJS_DB_DUCKDB_S3_USE_CREDENTIAL_CHAIN` | A flag to use credentials chain for secrets for S3 connections | `true`, `false`. Defaults to `false` | ❌ |
| `CUBEJS_CONCURRENCY` | The number of [concurrent queries][ref-data-source-concurrency] to the data source | A valid number | ❌ |

[ref-data-source-concurrency]: /product/configuration/concurrency#data-source-concurrency

Expand Down
8 changes: 8 additions & 0 deletions docs/pages/product/configuration/reference/environment-variables.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -375,6 +375,14 @@ A comma-separated list of DuckDB community extensions to install and load.
| ----------------------------------------------------- | ---------------------- | --------------------- |
| A comma-separated list of DuckDB community extensions | N/A | N/A |

## `CUBEJS_DB_DUCKDB_S3_USE_CREDENTIAL_CHAIN`

A flag to use credentials chain for secrets for S3 connections.

| Possible Values | Default in Development | Default in Production |
| ----------------------------------------------------- | ---------------------- | --------------------- |
| `true`, `false` | `false` | `false` |

## `CUBEJS_DB_ELASTIC_APIKEY_ID`

The [ID of the API key from elastic.co][elastic-docs-api-keys]. Required when
Expand Down
31 changes: 31 additions & 0 deletions packages/cubejs-backend-shared/src/env.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1784,6 +1784,7 @@ const variables: Record<string, (...args: any) => any> = {
}
return [];
},

duckdbCommunityExtensions: ({
dataSource
}: {
Expand All @@ -1797,6 +1798,36 @@ const variables: Record<string, (...args: any) => any> = {
}
return [];
},

duckdbS3UseCredentialChain: ({
dataSource
}: {
dataSource: string,
}) => {
const val = process.env[
keyByDataSource('CUBEJS_DB_DUCKDB_S3_USE_CREDENTIAL_CHAIN', dataSource)
];

if (val) {
if (val.toLocaleLowerCase() === 'true') {
return true;
} else if (val.toLowerCase() === 'false') {
return false;
} else {
throw new TypeError(
`The ${
keyByDataSource(
'CUBEJS_DB_DUCKDB_S3_USE_CREDENTIAL_CHAIN',
dataSource,
)
} must be either 'true' or 'false'.`
);
}
} else {
return false;
}
},

/** ***************************************************************
* Presto/Trino Driver *
**************************************************************** */
Expand Down
13 changes: 13 additions & 0 deletions packages/cubejs-duckdb-driver/src/DuckDBDriver.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ export type DuckDBDriverConfiguration = {
initSql?: string,
motherDuckToken?: string,
schema?: string,
duckdbS3UseCredentialChain?: boolean,
};

type InitPromise = {
Expand Down Expand Up @@ -164,6 +165,18 @@ export class DuckDBDriver extends BaseDriver implements DriverInterface {
}
}

const useCredentialChain = this.config.duckdbS3UseCredentialChain || getEnv('duckdbS3UseCredentialChain', this.config);
if (useCredentialChain) {
try {
await execAsync('CREATE SECRET (TYPE S3, PROVIDER \'CREDENTIAL_CHAIN\')');
} catch (e) {
if (this.logger) {
console.error('DuckDB - error on creating S3 credential chain secret', { e });
}
throw e;
}
}

// Install & load extensions if configured in env variable.
const officialExtensions = getEnv('duckdbExtensions', this.config);
await this.installExtensions(officialExtensions, execAsync);
Expand Down
Loading