Skip to content

Commit 8b96bc8

Browse files
paco-valdezpaco-valdez
authored
docs: Add export bucket configuration for Trino and Presto (#9932)
1 parent 88a6946 commit 8b96bc8

File tree

2 files changed

+138
-0
lines changed

2 files changed

+138
-0
lines changed

‎docs/pages/product/configuration/data-sources/presto.mdx‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,12 @@ CUBEJS_DB_SCHEMA=my_presto_schema
3434
| `CUBEJS_DB_SSL` | If `true`, enables SSL encryption for database connections from Cube | `true`, `false` | ❌ |
3535
| `CUBEJS_DB_MAX_POOL` | The maximum number of concurrent database connections to pool. Default is `8` | A valid number | ❌ |
3636
| `CUBEJS_CONCURRENCY` | The number of [concurrent queries][ref-data-source-concurrency] to the data source | A valid number | ❌ |
37+
| `CUBEJS_DB_EXPORT_BUCKET_TYPE` | The export bucket type for pre-aggregations | `s3`, `gcs` | ❌ |
38+
| `CUBEJS_DB_EXPORT_BUCKET` | The export bucket to connect to | A valid bucket URL | ❌ |
39+
| `CUBEJS_DB_EXPORT_BUCKET_AWS_KEY` | The AWS Access Key ID to use for export bucket writes | A valid AWS Access Key ID | ❌ |
40+
| `CUBEJS_DB_EXPORT_BUCKET_AWS_SECRET` | The AWS Secret Access Key to use for export bucket writes | A valid AWS Secret Access Key | ❌ |
41+
| `CUBEJS_DB_EXPORT_BUCKET_AWS_REGION` | The AWS region to use for export bucket writes | A valid AWS region | ❌ |
42+
| `CUBEJS_DB_EXPORT_GCS_CREDENTIALS` | A Base64 encoded JSON key file for connecting to Google Cloud Storage | A valid Google Cloud JSON key file, encoded as a Base64 string | ❌ |
3743

3844
[ref-data-source-concurrency]: /product/configuration/concurrency#data-source-concurrency
3945

@@ -47,16 +53,79 @@ be used in pre-aggregations when using Presto as a source database. To learn
4753
more about Presto support for approximate aggregate functions, [click
4854
here][presto-docs-approx-agg-fns].
4955

56+
## Pre-Aggregation Build Strategies
57+
58+
<InfoBox>
59+
60+
To learn more about pre-aggregation build strategies, [head
61+
here][ref-caching-using-preaggs-build-strats].
62+
63+
</InfoBox>
64+
65+
| Feature | Works with read-only mode? | Is default? |
66+
| ------------- | :------------------------: | :---------: |
67+
| Simple | ✅ | ✅ |
68+
| Export Bucket | ❌ | ❌ |
69+
70+
By default, Presto uses a [simple][self-preaggs-simple] strategy to
71+
build pre-aggregations.
72+
73+
### Simple
74+
75+
No extra configuration is required to configure simple pre-aggregation builds
76+
for Presto.
77+
78+
### Export Bucket
79+
80+
Presto supports using both [AWS S3][aws-s3] and [Google Cloud Storage][google-cloud-storage] for export bucket functionality.
81+
82+
#### AWS S3
83+
84+
<InfoBox>
85+
86+
Ensure the AWS credentials are correctly configured in IAM to allow reads and
87+
writes to the export bucket in S3.
88+
89+
</InfoBox>
90+
91+
```dotenv
92+
CUBEJS_DB_EXPORT_BUCKET_TYPE=s3
93+
CUBEJS_DB_EXPORT_BUCKET=my.bucket.on.s3
94+
CUBEJS_DB_EXPORT_BUCKET_AWS_KEY=<AWS_KEY>
95+
CUBEJS_DB_EXPORT_BUCKET_AWS_SECRET=<AWS_SECRET>
96+
CUBEJS_DB_EXPORT_BUCKET_AWS_REGION=<AWS_REGION>
97+
```
98+
99+
#### Google Cloud Storage
100+
101+
<InfoBox>
102+
103+
When using an export bucket, remember to assign the **Storage Object Admin**
104+
role to your Google Cloud credentials (`CUBEJS_DB_EXPORT_GCS_CREDENTIALS`).
105+
106+
</InfoBox>
107+
108+
```dotenv
109+
CUBEJS_DB_EXPORT_BUCKET=gs://presto-export-bucket
110+
CUBEJS_DB_EXPORT_BUCKET_TYPE=gcs
111+
CUBEJS_DB_EXPORT_GCS_CREDENTIALS=<BASE64_ENCODED_SERVICE_CREDENTIALS_JSON>
112+
```
113+
50114
## SSL
51115

52116
To enable SSL-encrypted connections between Cube and Presto, set the
53117
`CUBEJS_DB_SSL` environment variable to `true`. For more information on how to
54118
configure custom certificates, please check out [Enable SSL Connections to the
55119
Database][ref-recipe-enable-ssl].
56120

121+
[aws-s3]: https://aws.amazon.com/s3/
122+
[google-cloud-storage]: https://cloud.google.com/storage
57123
[presto]: https://prestodb.io/
58124
[presto-docs-approx-agg-fns]:
59125
https://prestodb.io/docs/current/functions/aggregate.html
126+
[ref-caching-using-preaggs-build-strats]:
127+
/product/caching/using-pre-aggregations#pre-aggregation-build-strategies
60128
[ref-recipe-enable-ssl]:
61129
/product/configuration/recipes/using-ssl-connections-to-data-source
62130
[ref-schema-ref-types-formats-countdistinctapprox]: /product/data-modeling/reference/types-and-formats#count_distinct_approx
131+
[self-preaggs-simple]: #simple

‎docs/pages/product/configuration/data-sources/trino.mdx‎

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,12 @@ CUBEJS_DB_SCHEMA=my_trino_schema
3434
| `CUBEJS_DB_SSL` | If `true`, enables SSL encryption for database connections from Cube | `true`, `false` | ❌ |
3535
| `CUBEJS_DB_MAX_POOL` | The maximum number of concurrent database connections to pool. Default is `8` | A valid number | ❌ |
3636
| `CUBEJS_CONCURRENCY` | The number of [concurrent queries][ref-data-source-concurrency] to the data source | A valid number | ❌ |
37+
| `CUBEJS_DB_EXPORT_BUCKET_TYPE` | The export bucket type for pre-aggregations | `s3`, `gcs` | ❌ |
38+
| `CUBEJS_DB_EXPORT_BUCKET` | The export bucket to connect to | A valid bucket URL | ❌ |
39+
| `CUBEJS_DB_EXPORT_BUCKET_AWS_KEY` | The AWS Access Key ID to use for export bucket writes | A valid AWS Access Key ID | ❌ |
40+
| `CUBEJS_DB_EXPORT_BUCKET_AWS_SECRET` | The AWS Secret Access Key to use for export bucket writes | A valid AWS Secret Access Key | ❌ |
41+
| `CUBEJS_DB_EXPORT_BUCKET_AWS_REGION` | The AWS region to use for export bucket writes | A valid AWS region | ❌ |
42+
| `CUBEJS_DB_EXPORT_GCS_CREDENTIALS` | A Base64 encoded JSON key file for connecting to Google Cloud Storage | A valid Google Cloud JSON key file, encoded as a Base64 string | ❌ |
3743

3844
[ref-data-source-concurrency]: /product/configuration/concurrency#data-source-concurrency
3945

@@ -47,13 +53,76 @@ be used in pre-aggregations when using Trino as a source database. To learn more
4753
about Trino support for approximate aggregate functions, [click
4854
here][trino-docs-approx-agg-fns].
4955

56+
## Pre-Aggregation Build Strategies
57+
58+
<InfoBox>
59+
60+
To learn more about pre-aggregation build strategies, [head
61+
here][ref-caching-using-preaggs-build-strats].
62+
63+
</InfoBox>
64+
65+
| Feature | Works with read-only mode? | Is default? |
66+
| ------------- | :------------------------: | :---------: |
67+
| Simple | ✅ | ✅ |
68+
| Export Bucket | ❌ | ❌ |
69+
70+
By default, Trino uses a [simple][self-preaggs-simple] strategy to
71+
build pre-aggregations.
72+
73+
### Simple
74+
75+
No extra configuration is required to configure simple pre-aggregation builds
76+
for Trino.
77+
78+
### Export Bucket
79+
80+
Trino supports using both [AWS S3][aws-s3] and [Google Cloud Storage][google-cloud-storage] for export bucket functionality.
81+
82+
#### AWS S3
83+
84+
<InfoBox>
85+
86+
Ensure the AWS credentials are correctly configured in IAM to allow reads and
87+
writes to the export bucket in S3.
88+
89+
</InfoBox>
90+
91+
```dotenv
92+
CUBEJS_DB_EXPORT_BUCKET_TYPE=s3
93+
CUBEJS_DB_EXPORT_BUCKET=my.bucket.on.s3
94+
CUBEJS_DB_EXPORT_BUCKET_AWS_KEY=<AWS_KEY>
95+
CUBEJS_DB_EXPORT_BUCKET_AWS_SECRET=<AWS_SECRET>
96+
CUBEJS_DB_EXPORT_BUCKET_AWS_REGION=<AWS_REGION>
97+
```
98+
99+
#### Google Cloud Storage
100+
101+
<InfoBox>
102+
103+
When using an export bucket, remember to assign the **Storage Object Admin**
104+
role to your Google Cloud credentials (`CUBEJS_DB_EXPORT_GCS_CREDENTIALS`).
105+
106+
</InfoBox>
107+
108+
```dotenv
109+
CUBEJS_DB_EXPORT_BUCKET=gs://trino-export-bucket
110+
CUBEJS_DB_EXPORT_BUCKET_TYPE=gcs
111+
CUBEJS_DB_EXPORT_GCS_CREDENTIALS=<BASE64_ENCODED_SERVICE_CREDENTIALS_JSON>
112+
```
113+
50114
## SSL
51115

52116
To enable SSL-encrypted connections between Cube and Trino, set the
53117
`CUBEJS_DB_SSL` environment variable to `true`. For more information on how to
54118
configure custom certificates, please check out [Enable SSL Connections to the
55119
Database][ref-recipe-enable-ssl].
56120

121+
[aws-s3]: https://aws.amazon.com/s3/
122+
[google-cloud-storage]: https://cloud.google.com/storage
123+
[ref-caching-using-preaggs-build-strats]:
124+
/product/caching/using-pre-aggregations#pre-aggregation-build-strategies
125+
[self-preaggs-simple]: #simple
57126
[trino]: https://trino.io/
58127
[trino-docs-approx-agg-fns]:
59128
https://trino.io/docs/current/functions/aggregate.html#approximate-aggregate-functions

0 commit comments

Comments
 (0)