Skip to content

Commit 4202501

Browse files
KSDaemonmarianore-muttdata
KSDaemon
authored and
marianore-muttdata
committed
feat(snowflake-driver): Ability to use encrypted private keys for auth (cube-js#9371)
* feat(snowflake-driver): Ability to use encrypted private keys for auth * update snowflake docs * add snowflake-encrypted-pk to ci drivers tests * fix * add missing secrets
1 parent 06ad0e6 commit 4202501

File tree

7 files changed

+16435
-23
lines changed

7 files changed

+16435
-23
lines changed

‎.github/workflows/drivers-tests.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -213,6 +213,7 @@ jobs:
213213
redshift
214214
redshift-export-bucket-s3
215215
snowflake
216+
snowflake-encrypted-pk
216217
snowflake-export-bucket-s3
217218
snowflake-export-bucket-azure
218219
snowflake-export-bucket-azure-via-storage-integration
@@ -242,6 +243,7 @@ jobs:
242243
- redshift
243244
- redshift-export-bucket-s3
244245
- snowflake
246+
- snowflake-encrypted-pk
245247
- snowflake-export-bucket-s3
246248
- snowflake-export-bucket-azure
247249
- snowflake-export-bucket-azure-via-storage-integration
@@ -340,6 +342,8 @@ jobs:
340342
# Snowflake
341343
DRIVERS_TESTS_CUBEJS_DB_SNOWFLAKE_USER: ${{ secrets.DRIVERS_TESTS_CUBEJS_DB_SNOWFLAKE_USER }}
342344
DRIVERS_TESTS_CUBEJS_DB_SNOWFLAKE_PASS: ${{ secrets.DRIVERS_TESTS_CUBEJS_DB_SNOWFLAKE_PASS }}
345+
DRIVERS_TESTS_CUBEJS_DB_SNOWFLAKE_PRIVATE_KEY: ${{ secrets.DRIVERS_TESTS_CUBEJS_DB_SNOWFLAKE_PRIVATE_KEY }}
346+
DRIVERS_TESTS_CUBEJS_DB_SNOWFLAKE_PRIVATE_KEY_PASS: ${{ secrets.DRIVERS_TESTS_CUBEJS_DB_SNOWFLAKE_PRIVATE_KEY_PASS }}
343347
with:
344348
max_attempts: 3
345349
retry_on: error

‎docs/pages/product/configuration/data-sources/snowflake.mdx‎

Lines changed: 24 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -12,16 +12,16 @@ redirect_from:
1212
- [The region][snowflake-docs-regions] for the [Snowflake][snowflake] warehouse
1313
- The username/password for the [Snowflake][snowflake] account
1414

15-
## Snowflake quoted identifiers
15+
## Snowflake quoted identifiers
1616

17-
Due to an issue in snowflakes opinion about quoted identifers we set a session value to override
17+
Due to an issue in snowflakes opinion about quoted identifers we set a session value to override
1818
snowflake defaults for users that have set an account value for: QUOTED_IDENTIFIERS_IGNORE_CASE
1919
you can learn more about this here: https://docs.snowflake.com/en/sql-reference/identifiers-syntax#double-quoted-identifiers
2020

2121
## Setup
2222

2323
<WarningBox>
24-
If you're having Network error and Snowflake can't be reached please make sure you tried
24+
If you're having Network error and Snowflake can't be reached please make sure you tried
2525
[format 2 for an account id][snowflake-format-2].
2626
</WarningBox>
2727

@@ -65,21 +65,25 @@ if [dedicated infrastructure][ref-dedicated-infra] is used. Check out the
6565

6666
## Environment Variables
6767

68-
| Environment Variable | Description | Possible Values | Required |
69-
| ----------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | :------: |
70-
| `CUBEJS_DB_SNOWFLAKE_ACCOUNT` | The Snowflake account identifier to use when connecting to the database | [A valid Snowflake account ID][snowflake-docs-account-id] | ✅ |
71-
| `CUBEJS_DB_SNOWFLAKE_REGION` | The Snowflake region to use when connecting to the database | [A valid Snowflake region][snowflake-docs-regions] | ❌ |
72-
| `CUBEJS_DB_SNOWFLAKE_WAREHOUSE` | The Snowflake warehouse to use when connecting to the database | [A valid Snowflake warehouse][snowflake-docs-warehouse] in the account | ✅ |
73-
| `CUBEJS_DB_SNOWFLAKE_ROLE` | The Snowflake role to use when connecting to the database | [A valid Snowflake role][snowflake-docs-roles] in the account | ❌ |
74-
| `CUBEJS_DB_SNOWFLAKE_CLIENT_SESSION_KEEP_ALIVE` | If `true`, [keep the Snowflake connection alive indefinitely][snowflake-docs-connection-options] | `true`, `false` | ❌ |
75-
| `CUBEJS_DB_NAME` | The name of the database to connect to | A valid database name | ✅ |
76-
| `CUBEJS_DB_USER` | The username used to connect to the database | A valid database username | ✅ |
77-
| `CUBEJS_DB_PASS` | The password used to connect to the database | A valid database password | ✅ |
78-
| `CUBEJS_DB_SNOWFLAKE_AUTHENTICATOR` | The type of authenticator to use with Snowflake. Use `SNOWFLAKE` with username/password, or `SNOWFLAKE_JWT` with key pairs. Defaults to `SNOWFLAKE` | `SNOWFLAKE`, `SNOWFLAKE_JWT` | ❌ |
79-
| `CUBEJS_DB_SNOWFLAKE_PRIVATE_KEY_PATH` | The path to the private RSA key folder | A valid path to the private RSA key | ❌ |
80-
| `CUBEJS_DB_SNOWFLAKE_PRIVATE_KEY_PASS` | The password for the private RSA key. Only required for encrypted keys | A valid password for the encrypted private RSA key | ❌ |
81-
| `CUBEJS_DB_MAX_POOL` | The maximum number of concurrent database connections to pool. Default is `20` | A valid number | ❌ |
82-
| `CUBEJS_CONCURRENCY` | The number of [concurrent queries][ref-data-source-concurrency] to the data source | A valid number | ❌ |
68+
| Environment Variable | Description | Possible Values | Required |
69+
| ---------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------- | :------: |
70+
| `CUBEJS_DB_SNOWFLAKE_ACCOUNT` | The Snowflake account identifier to use when connecting to the database | [A valid Snowflake account ID][snowflake-docs-account-id] | ✅ |
71+
| `CUBEJS_DB_SNOWFLAKE_REGION` | The Snowflake region to use when connecting to the database | [A valid Snowflake region][snowflake-docs-regions] | ❌ |
72+
| `CUBEJS_DB_SNOWFLAKE_WAREHOUSE` | The Snowflake warehouse to use when connecting to the database | [A valid Snowflake warehouse][snowflake-docs-warehouse] in the account | ✅ |
73+
| `CUBEJS_DB_SNOWFLAKE_ROLE` | The Snowflake role to use when connecting to the database | [A valid Snowflake role][snowflake-docs-roles] in the account | ❌ |
74+
| `CUBEJS_DB_SNOWFLAKE_CLIENT_SESSION_KEEP_ALIVE` | If `true`, [keep the Snowflake connection alive indefinitely][snowflake-docs-connection-options] | `true`, `false` | ❌ |
75+
| `CUBEJS_DB_NAME` | The name of the database to connect to | A valid database name | ✅ |
76+
| `CUBEJS_DB_USER` | The username used to connect to the database | A valid database username | ✅ |
77+
| `CUBEJS_DB_PASS` | The password used to connect to the database | A valid database password | ✅ |
78+
| `CUBEJS_DB_SNOWFLAKE_AUTHENTICATOR` | The type of authenticator to use with Snowflake. Use `SNOWFLAKE` with username/password, or `SNOWFLAKE_JWT` with key pairs. Defaults to `SNOWFLAKE` | `SNOWFLAKE`, `SNOWFLAKE_JWT`, `OAUTH` | ❌ |
79+
| `CUBEJS_DB_SNOWFLAKE_PRIVATE_KEY` | The content of the private RSA key | Content of the private RSA key (encrypted or not) | ❌ |
80+
| `CUBEJS_DB_SNOWFLAKE_PRIVATE_KEY_PATH` | The path to the private RSA key | A valid path to the private RSA key | ❌ |
81+
| `CUBEJS_DB_SNOWFLAKE_PRIVATE_KEY_PASS` | The password for the private RSA key. Only required for encrypted keys | A valid password for the encrypted private RSA key | ❌ |
82+
| `CUBEJS_DB_SNOWFLAKE_OAUTH_TOKEN_PATH` | The path to the valid oauth toket file | A valid path for the oauth token file | ❌ |
83+
| `CUBEJS_DB_SNOWFLAKE_HOST` | Host address to which the driver should connect | A valid hostname | ❌ |
84+
| `CUBEJS_DB_SNOWFLAKE_QUOTED_IDENTIFIERS_IGNORE_CASE` | Whether or not quoted identifiers should be case insensitive. Default is `false` | `true`, `false` | ❌ |
85+
| `CUBEJS_DB_MAX_POOL` | The maximum number of concurrent database connections to pool. Default is `20` | A valid number | ❌ |
86+
| `CUBEJS_CONCURRENCY` | The number of [concurrent queries][ref-data-source-concurrency] to the data source | A valid number | ❌ |
8387

8488
[ref-data-source-concurrency]: /product/configuration/concurrency#data-source-concurrency
8589

@@ -160,8 +164,8 @@ CUBEJS_DB_EXPORT_INTEGRATION=gcs_int
160164

161165
#### Azure
162166

163-
To use Azure Blob Storage as an export bucket, follow [the guide on
164-
using a Snowflake storage integration (Option 1)][snowflake-docs-azure].
167+
To use Azure Blob Storage as an export bucket, follow [the guide on
168+
using a Snowflake storage integration (Option 1)][snowflake-docs-azure].
165169
Take note of the integration name (`azure_int` from the example link)
166170
as you'll need it to configure Cube.
167171

‎packages/cubejs-snowflake-driver/src/SnowflakeDriver.ts‎

Lines changed: 26 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,7 @@ import {
2222
} from '@cubejs-backend/base-driver';
2323
import { formatToTimeZone } from 'date-fns-timezone';
2424
import fs from 'fs/promises';
25+
import crypto from 'crypto';
2526
import { HydrationMap, HydrationStream } from './HydrationStream';
2627

2728
const SUPPORTED_BUCKET_TYPES = ['s3', 'gcs', 'azure'];
@@ -245,8 +246,30 @@ export class SnowflakeDriver extends BaseDriver implements DriverInterface {
245246
assertDataSource('default');
246247

247248
let privateKey = getEnv('snowflakePrivateKey', { dataSource });
248-
if (privateKey && !privateKey.endsWith('\n')) {
249-
privateKey += '\n';
249+
250+
if (privateKey) {
251+
// If the private key is encrypted - we need to decrypt it before passing to
252+
// snowflake sdk.
253+
if (privateKey.includes('BEGIN ENCRYPTED PRIVATE KEY')) {
254+
const keyPasswd = getEnv('snowflakePrivateKeyPass', { dataSource });
255+
256+
if (!keyPasswd) {
257+
throw new Error(
258+
'Snowflake encrypted private key provided, but no passphrase was given.'
259+
);
260+
}
261+
262+
const privateKeyObject = crypto.createPrivateKey({
263+
key: privateKey,
264+
format: 'pem',
265+
passphrase: keyPasswd
266+
});
267+
268+
privateKey = privateKeyObject.export({
269+
format: 'pem',
270+
type: 'pkcs8'
271+
});
272+
}
250273
}
251274

252275
snowflake.configure({ logLevel: 'OFF' });
@@ -266,7 +289,7 @@ export class SnowflakeDriver extends BaseDriver implements DriverInterface {
266289
oauthTokenPath: getEnv('snowflakeOAuthTokenPath', { dataSource }),
267290
privateKeyPath: getEnv('snowflakePrivateKeyPath', { dataSource }),
268291
privateKeyPass: getEnv('snowflakePrivateKeyPass', { dataSource }),
269-
privateKey,
292+
...(privateKey ? { privateKey } : {}),
270293
exportBucket: this.getExportBucket(dataSource),
271294
resultPrefetch: 1,
272295
executionTimeout: getEnv('dbQueryTimeout', { dataSource }),

‎packages/cubejs-testing-drivers/fixtures/snowflake.json‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,15 @@
4040
"CUBEJS_DB_EXPORT_GCS_CREDENTIALS": "${DRIVERS_TESTS_CUBEJS_DB_EXPORT_GCS_CREDENTIALS}"
4141
}
4242
}
43+
},
44+
"encrypted-pk": {
45+
"cube": {
46+
"environment": {
47+
"CUBEJS_DB_SNOWFLAKE_AUTHENTICATOR": "SNOWFLAKE_JWT",
48+
"CUBEJS_DB_SNOWFLAKE_PRIVATE_KEY": "${DRIVERS_TESTS_CUBEJS_DB_SNOWFLAKE_PRIVATE_KEY}",
49+
"CUBEJS_DB_SNOWFLAKE_PRIVATE_KEY_PASS": "${DRIVERS_TESTS_CUBEJS_DB_SNOWFLAKE_PRIVATE_KEY_PASS}"
50+
}
51+
}
4352
}
4453
},
4554
"cube": {

‎packages/cubejs-testing-drivers/package.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,7 @@
4444
"snowflake-driver": "yarn test-driver -i dist/test/snowflake-driver.test.js",
4545
"snowflake-core": "yarn test-driver -i dist/test/snowflake-core.test.js",
4646
"snowflake-full": "yarn test-driver -i dist/test/snowflake-full.test.js",
47+
"snowflake-encrypted-pk-full": "yarn test-driver -i dist/test/snowflake-encrypted-pk-full.test.js",
4748
"snowflake-export-bucket-s3-full": "yarn test-driver -i dist/test/snowflake-export-bucket-s3-full.test.js",
4849
"snowflake-export-bucket-azure-full": "yarn test-driver -i dist/test/snowflake-export-bucket-azure-full.test.js",
4950
"snowflake-export-bucket-azure-via-storage-integration-full": "yarn test-driver -i dist/test/snowflake-export-bucket-azure-via-storage-integration-full.test.js",

0 commit comments

Comments
 (0)