You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/content/Caching/Running-in-Production.mdx
+43-33Lines changed: 43 additions & 33 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,25 +5,24 @@ category: Caching
5
5
menuOrder: 4
6
6
---
7
7
8
-
Cube.js makes use of two different kinds of cache:
8
+
Cube makes use of two different kinds of cache:
9
9
10
10
- Redis, for in-memory storage of query results
11
11
- Cube Store for storing pre-aggregations
12
12
13
-
In development, Cube.js uses in-memory storage on the server. In production, we
13
+
In development, Cube uses in-memory storage on the server. In production, we
14
14
**strongly** recommend running Redis as a separate service.
15
15
16
16
<WarningBox>
17
17
18
-
Cube Store [will replace
19
-
Redis][replace-redis] for in-memory cache and queue management in late
20
-
2022.
18
+
Cube Store [will replace Redis][replace-redis] for in-memory cache and queue
19
+
management in late 2022.
21
20
22
21
</WarningBox>
23
22
24
-
Cube Store is enabled by default when running Cube.js in development mode. In
23
+
Cube Store is enabled by default when running Cube in development mode. In
25
24
production, Cube Store **must** run as a separate process. The easiest way to do
26
-
this is to use the official Docker images for Cube.js and Cube Store.
25
+
this is to use the official Docker images for Cube and Cube Store.
27
26
28
27
<InfoBox>
29
28
@@ -41,13 +40,13 @@ docker run -p 3030:3030 cubejs/cubestore
41
40
<InfoBox>
42
41
43
42
Cube Store can further be configured via environment variables. To see a
44
-
complete reference, please consult the [Cube Store section of the Environment
45
-
Variables reference][ref-config-env].
43
+
complete reference, please consult the `CUBESTORE_*` environment variables in
44
+
the [Environment Variables reference][ref-config-env].
46
45
47
46
</InfoBox>
48
47
49
-
Next, run Cube.js and tell it to connect to Cube Store running on `localhost`
50
-
(on the default port `3030`):
48
+
Next, run Cube and tell it to connect to Cube Store running on `localhost` (on
49
+
the default port `3030`):
51
50
52
51
```bash
53
52
docker run -p 4000:4000 \
@@ -56,8 +55,8 @@ docker run -p 4000:4000 \
56
55
cubejs/cube
57
56
```
58
57
59
-
In the command above, we're specifying `CUBEJS_CUBESTORE_HOST` to let Cube.js
60
-
know where Cube Store is running.
58
+
In the command above, we're specifying `CUBEJS_CUBESTORE_HOST` to let Cube know
59
+
where Cube Store is running.
61
60
62
61
You can also use Docker Compose to achieve the same:
63
62
@@ -87,7 +86,8 @@ services:
87
86
88
87
## Architecture
89
88
90
-
Deep dive on Cube Store architecture can be found in [this presentation](https://docs.google.com/presentation/d/1oQ-koloag0UcL-bUHOpBXK4txpqiGl41rxhgDVrw7gw/).
89
+
Deep dive on Cube Store architecture can be found in
`CUBESTORE_WORKERS`and `CUBESTORE_META_ADDR` variables should be set with stable addresses, which should not change.
127
-
You can use stable DNS names and put load balancers in front of your worker and router instances to fulfill stable name requirements in environments where stable IP addresses can't be guaranteed.
126
+
`CUBESTORE_WORKERS`and `CUBESTORE_META_ADDR` variables should be set with
127
+
stable addresses, which should not change. You can use stable DNS names and put
128
+
load balancers in front of your worker and router instances to fulfill stable
129
+
name requirements in environments where stable IP addresses can't be guaranteed.
128
130
129
131
<InfoBox>
130
132
@@ -133,7 +135,8 @@ recommend using [partitioned pre-aggregations][ref-caching-partitioning].
133
135
134
136
</InfoBox>
135
137
136
-
A sample Docker Compose stack for the single machine setting this up might look like:
138
+
A sample Docker Compose stack for the single machine setting this up might look
139
+
like:
137
140
138
141
```yaml
139
142
version: '2.2'
@@ -188,11 +191,13 @@ services:
188
191
189
192
## Replication and High Availability
190
193
191
-
The open-source version of Cube Store doesn't support replicating any of its nodes.
192
-
The router node and every worker node should always have only one instance copy if served behind the load balancer or service address.
193
-
Replication will lead to undefined behavior of the cluster, including connection errors and data loss.
194
-
If any cluster node is down, it'll lead to a complete cluster outage.
195
-
If Cube Store replication and high availability are required, please consider using Cube Cloud.
194
+
The open-source version of Cube Store doesn't support replicating any of its
195
+
nodes. The router node and every worker node should always have only one
196
+
instance copy if served behind the load balancer or service address. Replication
197
+
will lead to undefined behavior of the cluster, including connection errors and
198
+
data loss. If any cluster node is down, it'll lead to a complete cluster outage.
199
+
If Cube Store replication and high availability are required, please consider
200
+
using Cube Cloud.
196
201
197
202
## Storage
198
203
@@ -204,14 +209,17 @@ Cube Store can only use one type of remote storage at runtime.
204
209
205
210
Cube Store makes use of a separate storage layer for storing metadata as well as
206
211
for persisting pre-aggregations as Parquet files. Cube Store [can be configured
207
-
to use either AWS S3 or Google Cloud Storage][ref-config-env-cloud-storage].
208
-
If desired, local path on the server can also be used in case all Cube Store cluster nodes are co-located on a single machine.
212
+
to use either AWS S3 or Google Cloud Storage][ref-config-env]. If desired, local
213
+
path on the server can also be used in case all Cube Store cluster nodes are
214
+
co-located on a single machine.
209
215
210
216
<WarningBox>
211
217
212
-
Cube Store requires strong consistency guarantees from underlying distributed storage.
213
-
AWS S3, Google Cloud Storage, and Azure Blob Storage (Cube Cloud only) are the only known implementations that provide strong consistency.
214
-
Using other implementations in production is discouraged and can lead to consistency and data corruption errors.
218
+
Cube Store requires strong consistency guarantees from underlying distributed
219
+
storage. AWS S3, Google Cloud Storage, and Azure Blob Storage (Cube Cloud only)
220
+
are the only known implementations that provide strong consistency. Using other
221
+
implementations in production is discouraged and can lead to consistency and
222
+
data corruption errors.
215
223
216
224
</WarningBox>
217
225
@@ -247,9 +255,12 @@ services:
247
255
248
256
### Local Storage
249
257
250
-
Separately from remote storage, Cube Store requires local scratch space to warm up partitions by downloading Parquet files before querying them.
251
-
By default, this directory should be mounted to `.cubestore/data` dir inside contained and can be configured by [CUBESTORE_DATA_DIR][ref-config-env] environment variable.
252
-
It is advised to use local SSDs for this scratch space to maximize querying performance.
258
+
Separately from remote storage, Cube Store requires local scratch space to warm
259
+
up partitions by downloading Parquet files before querying them. By default,
260
+
this directory should be mounted to `.cubestore/data` dir inside contained and
261
+
can be configured by [CUBESTORE_DATA_DIR][ref-config-env] environment variable.
262
+
It is advised to use local SSDs for this scratch space to maximize querying
263
+
performance.
253
264
254
265
### <--{"id" : "Storage"}--> AWS
255
266
@@ -271,10 +282,9 @@ default.
271
282
272
283
Cube Store currently does not have any in-built authentication mechanisms. For
273
284
this reason, we recommend running your Cube Store cluster on a network that only
@@ -483,30 +482,40 @@ refer to the database-specific documentation for more details:
483
482
-[Snowflake][ref-connect-db-snowflake]
484
483
485
484
When using cloud storage, it is important to correctly configure any data
486
-
retention policies to clean up the data in the export bucket as Cube.js does not
485
+
retention policies to clean up the data in the export bucket as Cube does not
487
486
currently manage this. For most use-cases, 1 day is sufficient.
488
487
489
488
## Streaming pre-aggregations
490
489
491
-
Streaming pre-aggregations are different from traditional pre-aggregations in the way they are being updated. Traditional pre-aggregations follow the “pull” model — Cube **pulls updates** from the data source based on some cadence and/or condition. Streaming pre-aggregations follow the “push” model — Cube **subscribes to the updates** from the data source and always keeps pre-aggregation up to date.
490
+
Streaming pre-aggregations are different from traditional pre-aggregations in
491
+
the way they are being updated. Traditional pre-aggregations follow the “pull”
492
+
model — Cube **pulls updates** from the data source based on some cadence and/or
493
+
condition. Streaming pre-aggregations follow the “push” model — Cube
494
+
**subscribes to the updates** from the data source and always keeps
495
+
pre-aggregation up to date.
492
496
493
-
You don’t need to define `refreshKey` for streaming pre-aggregations. Whether pre-aggregation is streaming or not is defined by the data source.
497
+
You don’t need to define `refreshKey` for streaming pre-aggregations. Whether
498
+
pre-aggregation is streaming or not is defined by the data source.
494
499
495
-
Currently, Cube supports only one streaming data source - [ksqlDB](/config/databases/ksqldb). All pre-aggregations where data source is ksqlDB are streaming.
500
+
Currently, Cube supports only one streaming data source -
501
+
[ksqlDB](/config/databases/ksqldb). All pre-aggregations where data source is
502
+
ksqlDB are streaming.
496
503
497
-
We’re working on supporting streaming pre-aggregations for the following data sources -
504
+
We’re working on supporting streaming pre-aggregations for the following data
505
+
sources -
498
506
499
507
- Materialize
500
508
- Flink SQL
501
509
- Spark Streaming
502
510
503
-
Please [let us know](https://cube.dev/contact) if you are interested in early access to any of these drivers or would like Cube to support any other SQL streaming engine.
511
+
Please [let us know](https://cube.dev/contact) if you are interested in early
512
+
access to any of these drivers or would like Cube to support any other SQL
|`CUBEJS_DB_SSL`| If `true`, enables SSL encryption for database connections from Cube.js|`true`, `false`|
170
+
|`CUBEJS_DB_SSL`| If `true`, enables SSL encryption for database connections from Cube|`true`, `false`|
171
171
|`CUBEJS_DB_SSL_CA`| The contents of a CA bundle in PEM format, or a path to one. For more information, check the `options.ca` property for TLS Secure Contexts [in the Node.js documentation][link-nodejs-tls-options]| A valid CA bundle or a path to one |
172
172
|`CUBEJS_DB_SSL_CERT`| The contents of an SSL certificate in PEM format, or a path to one. For more information, check the `options.cert` property for TLS Secure Contexts [in the Node.js documentation][link-nodejs-tls-options]| A valid SSL certificate or a path to one |
173
173
|`CUBEJS_DB_SSL_KEY`| The contents of a private key in PEM format, or a path to one. For more information, check the `options.key` property for TLS Secure Contexts [in the Node.js documentation][link-nodejs-tls-options]| A valid SSL private key or a path to one |
0 commit comments