Skip to content

Commit 1aae337

Browse files
josh-wongjosh-wong
committed
AUTO: Sync ScalarDB docs in English to docs site repo
1 parent a11ce71 commit 1aae337

File tree

3 files changed

+529
-54
lines changed

3 files changed

+529
-54
lines changed

‎docs/scalardb-analytics/_README.mdx‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
---
2+
tags:
3+
- Enterprise Option
4+
displayed_sidebar: docsEnglish
5+
---
6+
7+
# ScalarDB Analytics
8+
9+
import WarningLicenseKeyContact from '/src/components/en-us/_warning-license-key-contact.mdx';
10+
11+
**ScalarDB Analytics** is the analytical component of ScalarDB. Similar to ScalarDB, it unifies diverse data sources - ranging from RDBMSs like PostgreSQL and MySQL to NoSQL databases such as Cassandra and DynamoDB - into a single logical database. While ScalarDB focuses on operational workloads with strong transactional consistency across multiple databases, ScalarDB Analytics is optimized for analytical workloads. It supports a wide range of queries, including complex joins, aggregations, and window functions. ScalarDB Analytics operates seamlessly on both ScalarDB-managed data sources and non-ScalarDB-managed ones, enabling advanced analytical queries across various datasets.
12+
13+
The current version of ScalarDB Analytics leverages **Apache Spark** as its execution engine. It provides a unified view of ScalarDB-managed and non-ScalarDB-managed data sources by utilizing a Spark custom catalog. Using ScalarDB Analytics, you can treat tables from these data sources as native Spark tables. This allows you to execute arbitrary Spark SQL queries seamlessly. For example, you can join a table stored in Cassandra with a table in PostgreSQL to perform a cross-database analysis with ease.
14+
15+
<WarningLicenseKeyContact product="ScalarDB Analytics with Spark" />
16+
17+
## Further reading
18+
19+
This section provides links to various ScalarDB Analytics–related documentation.
20+
21+
### Getting started
22+
23+
* [Getting Started with ScalarDB Analytics](./quickstart.mdx) - A quick tutorial to set up ScalarDB Analytics and run federated queries
24+
25+
### Key documentation
26+
27+
* [Overview](./overview.mdx) - Understand ScalarDB Analytics architecture and features
28+
* [Deploy ScalarDB Analytics](./deployment.mdx) - Deploy on Amazon EMR, Databricks, and other platforms
29+
* [Run Analytical Queries](./run-analytical-queries.mdx) - Execute queries across multiple databases
30+
* [Administration Guide](./administration.mdx) - Manage catalogs and data sources
31+
* [Configuration Reference](./configuration.mdx) - Configure Spark and data sources
32+
33+
### Technical details
34+
35+
* [Design Document](./design.mdx) - Deep dive into the technical architecture
36+
* [Version Compatibility](./run-analytical-queries.mdx#version-compatibility) - Supported Spark and Scala versions

‎docs/scalardb-analytics/design.mdx‎

Lines changed: 284 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -86,16 +86,294 @@ ScalarDB Analytics supports a wide range of data types across different data sou
8686
- `DURATION`
8787
- `INTERVAL`
8888

89-
These data types are used across all data sources and provide a unified type system for querying heterogeneous databases.
89+
### Catalog information mappings by data source
9090

91-
### Data source integration
91+
When registering a data source to ScalarDB Analytics, the catalog information of the data source, that is, namespaces, tables, and columns, are resolved and registered to the universal data catalog. To resolve the catalog information of the data source, a particular object on the data sources side are mapped to the universal data catalog object. This mapping is consists of two parts: catalog-level mappings and data-type mappings. In the following sections, we describe how ScalarDB Analytics maps the catalog level and data type from each data source into the universal data catalog.
9292

93-
When registering a data source to ScalarDB Analytics, two types of mappings occur:
93+
#### Catalog-level mappings
9494

95-
1. **Catalog structure mapping**: The data source's catalog information (namespaces, tables, and columns) is resolved and mapped to the universal data catalog structure
96-
2. **Data type mapping**: Native data types from each data source are mapped to the universal data types listed above
95+
The catalog-level mappings are the mappings of the namespace names, table names, and column names from the data sources to the universal data catalog. To see the catalog-level mappings in each data source, select a data source.
9796

98-
These mappings ensure compatibility and consistency across different database systems. For detailed information about how specific databases are mapped, see [Catalog metadata reference](administration.mdx#catalog-metadata-reference) in the administration guide.
97+
<Tabs groupId="data-source" queryString>
98+
<TabItem value="scalardb" label="ScalarDB" default>
99+
The catalog information of ScalarDB is automatically resolved by ScalarDB Analytics. The catalog-level objects are mapped as follows:
100+
101+
- The ScalarDB namespace is mapped to the namespace. Therefore, the namespace of the ScalarDB data source is always single level, consisting of only the namespace name.
102+
- The ScalarDB table is mapped to the table.
103+
- The ScalarDB column is mapped to the column.
104+
105+
</TabItem>
106+
107+
<TabItem value="postgresql" label="PostgreSQL" default>
108+
The catalog information of PostgreSQL is automatically resolved by ScalarDB Analytics. The catalog-level objects are mapped as follows:
109+
110+
- The PostgreSQL schema is mapped to the namespace. Therefore, the namespace of the PostgreSQL data source is always single level, consisting of only the schema name.
111+
- Only user-defined schemas are mapped to namespaces. The following system schemas are ignored:
112+
- `information_schema`
113+
- `pg_catalog`
114+
- The PostgreSQL table is mapped to the table.
115+
- The PostgreSQL column is mapped to the column.
116+
117+
</TabItem>
118+
<TabItem value="mysql" label="MySQL">
119+
The catalog information of MySQL is automatically resolved by ScalarDB Analytics. The catalog-level objects are mapped as follows:
120+
121+
- The MySQL database is mapped to the namespace. Therefore, the namespace of the MySQL data source is always single level, consisting of only the database name.
122+
- Only user-defined databases are mapped to namespaces. The following system databases are ignored:
123+
- `mysql`
124+
- `sys`
125+
- `information_schema`
126+
- `performance_schema`
127+
- The MySQL table is mapped to the table.
128+
- The MySQL column is mapped to the column.
129+
130+
</TabItem>
131+
<TabItem value="oracle" label="Oracle">
132+
The catalog information of Oracle is automatically resolved by ScalarDB Analytics. The catalog-level objects are mapped as follows:
133+
134+
- The Oracle schema is mapped to the namespace. Therefore, the namespace of the Oracle data source is always single level, consisting of only schema name.
135+
- Only user-defined schemas are mapped to namespaces. The following system schemas are ignored:
136+
- `ANONYMOUS`
137+
- `APPQOSSYS`
138+
- `AUDSYS`
139+
- `CTXSYS`
140+
- `DBSNMP`
141+
- `DGPDB_INT`
142+
- `DBSFWUSER`
143+
- `DVF`
144+
- `DVSYS`
145+
- `GGSYS`
146+
- `GSMADMIN_INTERNAL`
147+
- `GSMCATUSER`
148+
- `GSMROOTUSER`
149+
- `GSMUSER`
150+
- `LBACSYS`
151+
- `MDSYS`
152+
- `OJVMSYS`
153+
- `ORDDATA`
154+
- `ORDPLUGINS`
155+
- `ORDSYS`
156+
- `OUTLN`
157+
- `REMOTE_SCHEDULER_AGENT`
158+
- `SI_INFORMTN_SCHEMA`
159+
- `SYS`
160+
- `SYS$UMF`
161+
- `SYSBACKUP`
162+
- `SYSDG`
163+
- `SYSKM`
164+
- `SYSRAC`
165+
- `SYSTEM`
166+
- `WMSYS`
167+
- `XDB`
168+
- `DIP`
169+
- `MDDATA`
170+
- `ORACLE_OCM`
171+
- `XS$NULL`
172+
173+
</TabItem>
174+
<TabItem value="sql-server" label="SQL Server">
175+
The catalog information of SQL Server is automatically resolved by ScalarDB Analytics. The catalog-level objects are mapped as follows:
176+
177+
- The SQL Server database and schema are mapped to the namespace together. Therefore, the namespace of the SQL Server data source is always two-level, consisting of the database name and the schema name.
178+
- Only user-defined databases are mapped to namespaces. The following system databases are ignored:
179+
- `sys`
180+
- `guest`
181+
- `INFORMATION_SCHEMA`
182+
- `db_accessadmin`
183+
- `db_backupoperator`
184+
- `db_datareader`
185+
- `db_datawriter`
186+
- `db_ddladmin`
187+
- `db_denydatareader`
188+
- `db_denydatawriter`
189+
- `db_owner`
190+
- `db_securityadmin`
191+
- Only user-defined schemas are mapped to namespaces. The following system schemas are ignored:
192+
- `master`
193+
- `model`
194+
- `msdb`
195+
- `tempdb`
196+
- The SQL Server table is mapped to the table.
197+
- The SQL Server column is mapped to the column.
198+
199+
</TabItem>
200+
<TabItem value="dynamodb" label="DynamoDB">
201+
Since DynamoDB is schema-less, you need to specify the catalog information explicitly when registering a DynamoDB data source by using the following format JSON:
202+
203+
```json
204+
{
205+
"namespaces": [
206+
{
207+
"name": "<NAMESPACE_NAME>",
208+
"tables": [
209+
{
210+
"name": "<TABLE_NAME>",
211+
"columns": [
212+
{
213+
"name": "<COLUMN_NAME>",
214+
"type": "<COLUMN_TYPE>"
215+
},
216+
...
217+
]
218+
},
219+
...
220+
]
221+
},
222+
...
223+
]
224+
}
225+
```
226+
227+
In the specified JSON, you can use any arbitrary namespace names, but the table names must match the table names in DynamoDB and column name and type must match field names and types in DynamoDB.
228+
229+
</TabItem>
230+
</Tabs>
231+
232+
#### Data-type mappings
233+
234+
The native data types of the underlying data sources are mapped to the data types in ScalarDB Analytics. To see the data-type mappings in each data source, select a data source.
235+
236+
<Tabs groupId="data-source" queryString>
237+
<TabItem value="scalardb" label="ScalarDB" default>
238+
| **ScalarDB Data Type** | **ScalarDB Analytics Data Type** |
239+
|:------------------------------|:---------------------------------|
240+
| `BOOLEAN` | `BOOLEAN` |
241+
| `INT` | `INT` |
242+
| `BIGINT` | `BIGINT` |
243+
| `FLOAT` | `FLOAT` |
244+
| `DOUBLE` | `DOUBLE` |
245+
| `TEXT` | `TEXT` |
246+
| `BLOB` | `BLOB` |
247+
| `DATE` | `DATE` |
248+
| `TIME` | `TIME` |
249+
| `TIMESTAMP` | `TIMESTAMP` |
250+
| `TIMESTAMPTZ` | `TIMESTAMPTZ` |
251+
</TabItem>
252+
<TabItem value="postgresql" label="PostgreSQL" default>
253+
| **PostgreSQL Data Type** | **ScalarDB Analytics Data Type** |
254+
|:------------------------------|:---------------------------------|
255+
| `integer` | `INT` |
256+
| `bigint` | `BIGINT` |
257+
| `real` | `FLOAT` |
258+
| `double precision` | `DOUBLE` |
259+
| `smallserial` | `SMALLINT` |
260+
| `serial` | `INT` |
261+
| `bigserial` | `BIGINT` |
262+
| `char` | `TEXT` |
263+
| `varchar` | `TEXT` |
264+
| `text` | `TEXT` |
265+
| `bpchar` | `TEXT` |
266+
| `boolean` | `BOOLEAN` |
267+
| `bytea` | `BLOB` |
268+
| `date` | `DATE` |
269+
| `time` | `TIME` |
270+
| `time with time zone` | `TIME` |
271+
| `time without time zone` | `TIME` |
272+
| `timestamp` | `TIMESTAMP` |
273+
| `timestamp with time zone` | `TIMESTAMPTZ` |
274+
| `timestamp without time zone` | `TIMESTAMP` |
275+
</TabItem>
276+
<TabItem value="mysql" label="MySQL">
277+
| **MySQL Data Type** | **ScalarDB Analytics Data Type** |
278+
|:-----------------------|:---------------------------------|
279+
| `bit` | `BOOLEAN` |
280+
| `bit(1)` | `BOOLEAN` |
281+
| `bit(x)` if *x >= 2* | `BLOB` |
282+
| `tinyint` | `SMALLINT` |
283+
| `tinyint(1)` | `BOOLEAN` |
284+
| `boolean` | `BOOLEAN` |
285+
| `smallint` | `SMALLINT` |
286+
| `smallint unsigned` | `INT` |
287+
| `mediumint` | `INT` |
288+
| `mediumint unsigned` | `INT` |
289+
| `int` | `INT` |
290+
| `int unsigned` | `BIGINT` |
291+
| `bigint` | `BIGINT` |
292+
| `float` | `FLOAT` |
293+
| `double` | `DOUBLE` |
294+
| `real` | `DOUBLE` |
295+
| `char` | `TEXT` |
296+
| `varchar` | `TEXT` |
297+
| `text` | `TEXT` |
298+
| `binary` | `BLOB` |
299+
| `varbinary` | `BLOB` |
300+
| `blob` | `BLOB` |
301+
| `date` | `DATE` |
302+
| `time` | `TIME` |
303+
| `datetime` | `TIMESTAMP` |
304+
| `timestamp` | `TIMESTAMPTZ` |
305+
</TabItem>
306+
<TabItem value="oracle" label="Oracle">
307+
| **Oracle Data Type** | **ScalarDB Analytics Data Type** |
308+
|:-----------------------------------|:---------------------------------|
309+
| `NUMBER` if *scale = 0* | `BIGINT` |
310+
| `NUMBER` if *scale > 0* | `DOUBLE` |
311+
| `FLOAT` if *precision ≤ 53* | `DOUBLE` |
312+
| `BINARY_FLOAT` | `FLOAT` |
313+
| `BINARY_DOUBLE` | `DOUBLE` |
314+
| `CHAR` | `TEXT` |
315+
| `NCHAR` | `TEXT` |
316+
| `VARCHAR2` | `TEXT` |
317+
| `NVARCHAR2` | `TEXT` |
318+
| `CLOB` | `TEXT` |
319+
| `NCLOB` | `TEXT` |
320+
| `BLOB` | `BLOB` |
321+
| `BOOLEAN` | `BOOLEAN` |
322+
| `DATE` | `DATE` |
323+
| `TIMESTAMP` | `TIMESTAMPTZ` |
324+
| `TIMESTAMP WITH TIME ZONE` | `TIMESTAMPTZ` |
325+
| `TIMESTAMP WITH LOCAL TIME ZONE` | `TIMESTAMP` |
326+
| `RAW` | `BLOB` |
327+
</TabItem>
328+
<TabItem value="sql-server" label="SQL Server">
329+
| **SQL Server Data Type** | **ScalarDB Analytics Data Type** |
330+
|:---------------------------|:---------------------------------|
331+
| `bit` | `BOOLEAN` |
332+
| `tinyint` | `SMALLINT` |
333+
| `smallint` | `SMALLINT` |
334+
| `int` | `INT` |
335+
| `bigint` | `BIGINT` |
336+
| `real` | `FLOAT` |
337+
| `float` | `DOUBLE` |
338+
| `float(n)` if *n ≤ 24* | `FLOAT` |
339+
| `float(n)` if *n ≥ 25* | `DOUBLE` |
340+
| `binary` | `BLOB` |
341+
| `varbinary` | `BLOB` |
342+
| `char` | `TEXT` |
343+
| `varchar` | `TEXT` |
344+
| `nchar` | `TEXT` |
345+
| `nvarchar` | `TEXT` |
346+
| `ntext` | `TEXT` |
347+
| `text` | `TEXT` |
348+
| `date` | `DATE` |
349+
| `time` | `TIME` |
350+
| `datetime` | `TIMESTAMP` |
351+
| `datetime2` | `TIMESTAMP` |
352+
| `smalldatetime` | `TIMESTAMP` |
353+
| `datetimeoffset` | `TIMESTAMPTZ` |
354+
</TabItem>
355+
<TabItem value="dynamodb" label="DynamoDB">
356+
| **DynamoDB Data Type** | **ScalarDB Analytics Data Type** |
357+
|:-------------------------|:---------------------------------|
358+
| `Number` | `BYTE` |
359+
| `Number` | `SMALLINT` |
360+
| `Number` | `INT` |
361+
| `Number` | `BIGINT` |
362+
| `Number` | `FLOAT` |
363+
| `Number` | `DOUBLE` |
364+
| `Number` | `DECIMAL` |
365+
| `String` | `TEXT` |
366+
| `Binary` | `BLOB` |
367+
| `Boolean` | `BOOLEAN` |
368+
369+
:::warning
370+
371+
It is important to ensure that the field values of `Number` types are parsable as a specified data type for ScalarDB Analytics. For example, if a column that corresponds to a `Number`-type field is specified as an `INT` type, its value must be an integer. If the value is not an integer, an error will occur when running a query.
372+
373+
:::
374+
375+
</TabItem>
376+
</Tabs>
99377

100378
## Query engine
101379

0 commit comments

Comments
 (0)