Skip to content

Commit 2b60728

Browse files
josh-wongjosh-wong
authored
Add new and updated ScalarDB Analytics docs (tentatively) (#1417)
* Add new doc (tentative publication) * Add updated doc (tentative publication) * Add ScalarDB Analytics admin guide and config reference * Add redirect for ScalarDB Analytics quickstart
1 parent 57bbc0c commit 2b60728

File tree

10 files changed

+2203
-981
lines changed

10 files changed

+2203
-981
lines changed

‎docs/scalardb-analytics/administration.mdx‎

Lines changed: 798 additions & 0 deletions
Large diffs are not rendered by default.

‎docs/scalardb-analytics/configuration.mdx‎

Lines changed: 242 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,242 @@
1+
---
2+
tags:
3+
- Enterprise Option
4+
displayed_sidebar: docsEnglish
5+
---
6+
7+
# Configuration reference
8+
9+
This page provides a comprehensive reference for configuring all components of ScalarDB Analytics.
10+
11+
## Overview
12+
13+
ScalarDB Analytics consists of three main components that require configuration:
14+
15+
1. **ScalarDB Analytics server** - The server that hosts the catalog and metering services
16+
2. **CLI client** - The command-line interface for managing catalogs and data sources
17+
3. **Spark integration** - Configuration for using ScalarDB Analytics with Apache Spark
18+
19+
## ScalarDB Analytics server configuration
20+
21+
The server is configured using a standard Java properties file (e.g., `scalardb-analytics-server.properties`) that defines database connections, network settings, licensing, and optional features.
22+
23+
### Configuration properties
24+
25+
#### Metadata database configuration
26+
27+
The server requires a metadata database to store catalog information.
28+
29+
| Property | Required | Description | Default | Example |
30+
| ---------------------------------------- | -------- | ------------------------------------ | ------- | ----------------------------------------------------- |
31+
| `scalar.db.analytics.server.db.url` | Yes | JDBC URL for the metadata database | - | `jdbc:postgresql://localhost:5432/scalardb_analytics` |
32+
| `scalar.db.analytics.server.db.username` | Yes | Database user for authentication | - | `analytics_user` |
33+
| `scalar.db.analytics.server.db.password` | Yes | Database password for authentication | - | `your_secure_password` |
34+
35+
#### gRPC server configuration
36+
37+
Configure the ports for the catalog and metering services.
38+
39+
| Property | Required | Default | Description | Example |
40+
| ------------------------------------------ | -------- | ------- | ----------------------------- | ------- |
41+
| `scalar.db.analytics.server.catalog.port` | No | `11051` | Port for the catalog service | `11051` |
42+
| `scalar.db.analytics.server.metering.port` | No | `11052` | Port for the metering service | `11052` |
43+
44+
#### TLS configuration
45+
46+
Enable TLS/SSL for secure communication.
47+
48+
| Property | Required | Default | Description | Example |
49+
| ------------------------------------------------- | -------- | ------- | ----------------------------------------- | --------------------- |
50+
| `scalar.db.analytics.server.tls.enabled` | No | `false` | Enable TLS/SSL for gRPC endpoints | `true` |
51+
| `scalar.db.analytics.server.tls.cert_chain_path` | Yes\* | - | Path to the server certificate chain file | `/path/to/server.crt` |
52+
| `scalar.db.analytics.server.tls.private_key_path` | Yes\* | - | Path to the server private key file | `/path/to/server.key` |
53+
54+
\* Required when `tls.enabled` is `true`
55+
56+
#### License configuration
57+
58+
Configure your ScalarDB Analytics license.
59+
60+
| Property | Required | Description | Default | Example |
61+
| -------------------------------------------------------------- | -------- | ---------------------------------------------- | ------- | ------------------------------ |
62+
| `scalar.db.analytics.server.licensing.license_key` | Yes | Your ScalarDB Analytics license key | - | Contact Scalar for license |
63+
| `scalar.db.analytics.server.licensing.license_check_cert_pem` | Yes\* | License verification certificate as PEM string | - | Contact Scalar for certificate |
64+
| `scalar.db.analytics.server.licensing.license_check_cert_path` | Yes\* | Path to license verification certificate file | - | `/path/to/cert.pem` |
65+
66+
\* Either `license_check_cert_pem` or `license_check_cert_path` must be specified
67+
68+
#### Metering storage configuration
69+
70+
Configure storage for metering data.
71+
72+
| Property | Required | Default | Description | Example |
73+
| ------------------------------------------------------------- | -------- | ---------- | ------------------------------------------------------------------------------------------------ | ------------------------------------------ |
74+
| `scalar.db.analytics.server.metering.storage.provider` | Yes | - | Storage provider for metering data (`filesystem`, `aws-s3`, `azureblob`, `google-cloud-storage`) | `filesystem` |
75+
| `scalar.db.analytics.server.metering.storage.containerName` | No | `metering` | Container/bucket name for cloud storage | `my-metering-bucket` |
76+
| `scalar.db.analytics.server.metering.storage.path` | Yes\* | - | Local directory path (for `filesystem` provider only) | `/var/scalardb-analytics/metering` |
77+
| `scalar.db.analytics.server.metering.storage.accessKeyId` | Yes\*\* | - | Access key ID for cloud storage providers | `AKIAIOSFODNN7EXAMPLE` |
78+
| `scalar.db.analytics.server.metering.storage.secretAccessKey` | Yes\*\* | - | Secret access key for cloud storage providers | `wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY` |
79+
| `scalar.db.analytics.server.metering.storage.prefix` | No | - | Optional prefix for all storage paths | `production/` |
80+
81+
\* Required when provider is `filesystem`
82+
\*\* Required for cloud storage providers (`aws-s3`, `azureblob`, `google-cloud-storage`)
83+
84+
## CLI client configuration
85+
86+
The CLI client requires connection settings to communicate with the ScalarDB Analytics server using a Java properties file (e.g., `client.properties`).
87+
88+
### Configuration properties
89+
90+
#### Server connection configuration
91+
92+
| Property | Required | Default | Description | Example |
93+
| ------------------------------------------------- | -------- | ------- | ------------------------------------------------------- | ----------------------- |
94+
| `scalar.db.analytics.client.server.host` | Yes | - | Hostname or IP address of the ScalarDB Analytics server | `analytics.example.com` |
95+
| `scalar.db.analytics.client.server.catalog.port` | No | `11051` | Port number for the catalog service | `11051` |
96+
| `scalar.db.analytics.client.server.metering.port` | No | `11052` | Port number for the metering service | `11052` |
97+
98+
#### TLS configuration
99+
100+
| Property | Required | Default | Description | Example |
101+
| ---------------------------------------------------------- | -------- | ------- | ----------------------------------------------------------------------- | ------------------- |
102+
| `scalar.db.analytics.client.server.tls.enabled` | No | `false` | Enable TLS/SSL for server connections | `true` |
103+
| `scalar.db.analytics.client.server.tls.ca_root_cert_path` | Yes\* | - | Path to the CA certificate file for verifying server certificates | `/path/to/cert.pem` |
104+
| `scalar.db.analytics.client.server.tls.override_authority` | No | - | Override the server authority for TLS verification (useful for testing) | `test.example.com` |
105+
106+
\* Required when `tls.enabled` is `true`
107+
108+
## Spark integration configuration
109+
110+
To use ScalarDB Analytics with Apache Spark, configure your Spark application by adding the necessary settings to your Spark configuration file (`spark-defaults.conf`).
111+
112+
### Configuration properties
113+
114+
#### Core Spark configuration
115+
116+
| Property | Required | Description | Default | Example |
117+
| ---------------------- | -------- | ----------------------------------------------------- | ------- | ------------------------------------------------------------------ |
118+
| `spark.jars.packages` | Yes | Maven coordinates for ScalarDB Analytics dependencies | - | `com.scalar-labs:scalardb-analytics-spark-all-3.5_2.12:3.16.2` |
119+
| `spark.extraListeners` | Yes | Register the ScalarDB Analytics metering listener | - | `com.scalar.db.analytics.spark.metering.ScalarDbAnalyticsListener` |
120+
121+
For `spark.jars.packages`, replace:
122+
123+
- `<SPARK_VERSION>` with your Spark version (e.g., `3.5`)
124+
- `<SCALA_VERSION>` with your Scala version (e.g., `2.12`)
125+
- `<SCALARDB_ANALYTICS_VERSION>` with the ScalarDB Analytics version (e.g., `3.16.2`)
126+
127+
#### Catalog configuration
128+
129+
| Property | Required | Description | Default | Value |
130+
| ---------------------------------- | -------- | ------------------------------------------------------ | ------- | ---------------------------------------------------------------- |
131+
| `spark.sql.catalog.<catalog-name>` | Yes | Register the ScalarDB Analytics catalog implementation | - | `com.scalar.db.analytics.spark.catalog.ScalarDBAnalyticsCatalog` |
132+
133+
#### Server connection settings
134+
135+
| Property | Required | Default | Description | Example |
136+
| ------------------------------------------------------- | -------- | ------- | ------------------------------------------------------- | ----------- |
137+
| `spark.sql.catalog.<catalog-name>.server.host` | Yes | - | Hostname or IP address of the ScalarDB Analytics server | `localhost` |
138+
| `spark.sql.catalog.<catalog-name>.server.catalog.port` | No | `11051` | Port number for the catalog service | `11051` |
139+
| `spark.sql.catalog.<catalog-name>.server.metering.port` | No | `11052` | Port number for the metering service | `11052` |
140+
141+
#### TLS/SSL settings
142+
143+
| Property | Required | Default | Description | Example |
144+
| ---------------------------------------------------------------- | -------- | ------- | ----------------------------------------------------------------- | ------------------- |
145+
| `spark.sql.catalog.<catalog-name>.server.tls.enabled` | No | `false` | Enable TLS/SSL for server connections | `true` |
146+
| `spark.sql.catalog.<catalog-name>.server.tls.ca_root_cert_path` | Yes\* | - | Path to the CA certificate file for verifying server certificates | `/path/to/cert.pem` |
147+
| `spark.sql.catalog.<catalog-name>.server.tls.override_authority` | No | - | Override the server authority for TLS verification | `test.example.com` |
148+
149+
\* Required when `tls.enabled` is `true`
150+
151+
Replace `<catalog-name>` with your chosen catalog name (e.g., `analytics`).
152+
153+
## Configuration examples
154+
155+
### Basic development configuration
156+
157+
#### Server configuration (`scalardb-analytics-server.properties`)
158+
159+
```properties
160+
# Metadata database
161+
scalar.db.analytics.server.db.url=jdbc:postgresql://localhost:5432/scalardb_analytics
162+
scalar.db.analytics.server.db.username=dev_user
163+
scalar.db.analytics.server.db.password=dev_password
164+
165+
# License
166+
scalar.db.analytics.server.licensing.license_key=YOUR_DEV_LICENSE_KEY
167+
scalar.db.analytics.server.licensing.license_check_cert_path=/path/to/license_cert.pem
168+
169+
# Metering storage (filesystem for development)
170+
scalar.db.analytics.server.metering.storage.provider=filesystem
171+
scalar.db.analytics.server.metering.storage.path=/tmp/scalardb-analytics-metering
172+
```
173+
174+
#### Client configuration (`client.properties`)
175+
176+
```properties
177+
scalar.db.analytics.client.server.host=localhost
178+
```
179+
180+
#### Spark configuration (`spark-defaults.conf`)
181+
182+
```properties
183+
spark.jars.packages com.scalar-labs:scalardb-analytics-spark-all-3.5_2.12:3.16.2
184+
spark.extraListeners com.scalar.db.analytics.spark.metering.ScalarDbAnalyticsListener
185+
spark.sql.catalog.analytics com.scalar.db.analytics.spark.catalog.ScalarDBAnalyticsCatalog
186+
spark.sql.catalog.analytics.server.host localhost
187+
```
188+
189+
### Production configuration with TLS
190+
191+
#### Server configuration (`scalardb-analytics-server.properties`)
192+
193+
```properties
194+
# Metadata database
195+
scalar.db.analytics.server.db.url=jdbc:postgresql://db.internal:5432/scalardb_analytics_prod
196+
scalar.db.analytics.server.db.username=analytics_prod
197+
scalar.db.analytics.server.db.password=your_secure_password
198+
199+
# gRPC ports
200+
scalar.db.analytics.server.catalog.port=11051
201+
scalar.db.analytics.server.metering.port=11052
202+
203+
# TLS
204+
scalar.db.analytics.server.tls.enabled=true
205+
scalar.db.analytics.server.tls.cert_chain_path=/path/to/server.crt
206+
scalar.db.analytics.server.tls.private_key_path=/path/to/server.key
207+
208+
# License
209+
scalar.db.analytics.server.licensing.license_key=YOUR_LICENSE_KEY
210+
scalar.db.analytics.server.licensing.license_check_cert_pem=-----BEGIN CERTIFICATE-----\nMIID...certificate content...\n-----END CERTIFICATE-----
211+
212+
# Metering storage (S3)
213+
scalar.db.analytics.server.metering.storage.provider=aws-s3
214+
scalar.db.analytics.server.metering.storage.containerName=analytics-metering
215+
scalar.db.analytics.server.metering.storage.accessKeyId=AKIAIOSFODNN7EXAMPLE
216+
scalar.db.analytics.server.metering.storage.secretAccessKey=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
217+
scalar.db.analytics.server.metering.storage.prefix=prod/
218+
```
219+
220+
#### Client configuration (`client.properties`)
221+
222+
```properties
223+
scalar.db.analytics.client.server.host=analytics.example.com
224+
scalar.db.analytics.client.server.tls.enabled=true
225+
scalar.db.analytics.client.server.tls.ca_root_cert_path=/path/to/cert.pem
226+
```
227+
228+
#### Spark configuration (`spark-defaults.conf`)
229+
230+
```properties
231+
spark.jars.packages com.scalar-labs:scalardb-analytics-spark-all-3.5_2.12:3.16.2
232+
spark.extraListeners com.scalar.db.analytics.spark.metering.ScalarDbAnalyticsListener
233+
spark.sql.catalog.analytics com.scalar.db.analytics.spark.catalog.ScalarDBAnalyticsCatalog
234+
spark.sql.catalog.analytics.server.host analytics.example.com
235+
spark.sql.catalog.analytics.server.tls.enabled true
236+
spark.sql.catalog.analytics.server.tls.ca_root_cert_path /path/to/cert.pem
237+
```
238+
239+
## Next steps
240+
241+
- [Run analytical queries](run-analytical-queries.mdx) - Start running queries with your configuration
242+
- [Deployment guide](deployment.mdx) - Deploy ScalarDB Analytics in production

0 commit comments

Comments
 (0)