diff --git a/versioned_docs/version-3.X/_develop-run-analytical-queries-overview.mdx b/versioned_docs/version-3.X/_develop-run-analytical-queries-overview.mdx new file mode 100644 index 00000000..e2be771f --- /dev/null +++ b/versioned_docs/version-3.X/_develop-run-analytical-queries-overview.mdx @@ -0,0 +1,14 @@ +--- +tags: + - Community + - Enterprise Option +displayed_sidebar: docsEnglish +--- + +# Run Analytical Queries Overview + +In this sub-category, you can learn how to set up and configure ScalarDB Analytics, an analytics component of ScalarDB. Then, you can run analytical queries over ScalarDB-managed databases, which are updated through ScalarDB transactions, and non-ScalarDB-managed databases. + +To learn how to run analytical queries, see the following guides: + +- [Run Analytical Queries on Sample Data by Using ScalarDB Analytics with PostgreSQL](scalardb-samples/scalardb-analytics-postgresql-sample/README.mdx) diff --git a/versioned_docs/version-3.X/add-scalardb-to-your-build.mdx b/versioned_docs/version-3.X/add-scalardb-to-your-build.mdx new file mode 100644 index 00000000..976e68d8 --- /dev/null +++ b/versioned_docs/version-3.X/add-scalardb-to-your-build.mdx @@ -0,0 +1,41 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Add ScalarDB to Your Build + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +The ScalarDB library is available on the [Maven Central Repository](https://mvnrepository.com/artifact/com.scalar-labs/scalardb). You can add the library as a build dependency to your application by using Gradle or Maven. + +## Configure your application based on your build tool + +Select your build tool, and follow the instructions to add the build dependency for ScalarDB to your application. + + + + To add the build dependency for ScalarDB by using Gradle, add the following to `build.gradle` in your application, replacing `` with the version of ScalarDB that you want to use: + + ```gradle + dependencies { + implementation 'com.scalar-labs:scalardb:' + } + ``` + + + To add the build dependency for ScalarDB by using Maven, add the following to `pom.xml` in your application, replacing `` with the version of ScalarDB that you want to use: + + ```xml + + com.scalar-labs + scalardb + + + ``` + + diff --git a/versioned_docs/version-3.X/api-guide.mdx b/versioned_docs/version-3.X/api-guide.mdx new file mode 100644 index 00000000..7be336f1 --- /dev/null +++ b/versioned_docs/version-3.X/api-guide.mdx @@ -0,0 +1,1770 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# ScalarDB Java API Guide + +import JavadocLink from '/src/theme/JavadocLink.js'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +The ScalarDB Java API is mainly composed of the Administrative API and Transactional API. This guide briefly explains what kinds of APIs exist, how to use them, and related topics like how to handle exceptions. + +## Administrative API + +This section explains how to execute administrative operations programmatically by using the Administrative API in ScalarDB. + +:::note + +Another method for executing administrative operations is to use [Schema Loader](schema-loader.mdx). + +::: + +### Get a `DistributedTransactionAdmin` instance + +You first need to get a `DistributedTransactionAdmin` instance to execute administrative operations. + +To get a `DistributedTransactionAdmin` instance, you can use `TransactionFactory` as follows: + +```java +TransactionFactory transactionFactory = TransactionFactory.create(""); +DistributedTransactionAdmin admin = transactionFactory.getTransactionAdmin(); +``` + +For details about configurations, see [ScalarDB Configurations](configurations.mdx). + +After you have executed all administrative operations, you should close the `DistributedTransactionAdmin` instance as follows: + +```java +admin.close(); +``` + +### Create a namespace + +Before creating tables, namespaces must be created since a table belongs to one namespace. + +You can create a namespace as follows: + +```java +// Create the namespace "ns". If the namespace already exists, an exception will be thrown. +admin.createNamespace("ns"); + +// Create the namespace only if it does not already exist. +boolean ifNotExists = true; +admin.createNamespace("ns", ifNotExists); + +// Create the namespace with options. +Map options = ...; +admin.createNamespace("ns", options); +``` + +#### Creation options + +In the creation operations, like creating a namespace and creating a table, you can specify options that are maps of option names and values (`Map`). By using the options, you can set storage adapter–specific configurations. + +Select your database to see the options available: + + + + No options are available for JDBC databases. + + + | Name | Description | Default | + |------------|-----------------------------------------|---------| + | no-scaling | Disable auto-scaling for DynamoDB. | false | + | no-backup | Disable continuous backup for DynamoDB. | false | + | ru | Base resource unit. | 10 | + + + | Name | Description | Default | + |------------|-----------------------------------------------------|---------| + | ru | Base resource unit. | 400 | + | no-scaling | Disable auto-scaling for Cosmos DB for NoSQL. | false | + + + | Name | Description | Default | + |----------------------|----------------------------------------------------------------------------------------|------------------| + | replication-strategy | Cassandra replication strategy. Must be `SimpleStrategy` or `NetworkTopologyStrategy`. | `SimpleStrategy` | + | compaction-strategy | Cassandra compaction strategy, Must be `LCS`, `STCS` or `TWCS`. | `STCS` | + | replication-factor | Cassandra replication factor. | 1 | + + + +### Create a table + +When creating a table, you should define the table metadata and then create the table. + +To define the table metadata, you can use `TableMetadata`. The following shows how to define the columns, partition key, clustering key including clustering orders, and secondary indexes of a table: + +```java +// Define the table metadata. +TableMetadata tableMetadata = + TableMetadata.newBuilder() + .addColumn("c1", DataType.INT) + .addColumn("c2", DataType.TEXT) + .addColumn("c3", DataType.BIGINT) + .addColumn("c4", DataType.FLOAT) + .addColumn("c5", DataType.DOUBLE) + .addPartitionKey("c1") + .addClusteringKey("c2", Scan.Ordering.Order.DESC) + .addClusteringKey("c3", Scan.Ordering.Order.ASC) + .addSecondaryIndex("c4") + .build(); +``` + +For details about the data model of ScalarDB, see [Data Model](design.mdx#data-model). + +Then, create a table as follows: + +```java +// Create the table "ns.tbl". If the table already exists, an exception will be thrown. +admin.createTable("ns", "tbl", tableMetadata); + +// Create the table only if it does not already exist. +boolean ifNotExists = true; +admin.createTable("ns", "tbl", tableMetadata, ifNotExists); + +// Create the table with options. +Map options = ...; +admin.createTable("ns", "tbl", tableMetadata, options); +``` + +### Create a secondary index + +You can create a secondary index as follows: + +```java +// Create a secondary index on column "c5" for table "ns.tbl". If a secondary index already exists, an exception will be thrown. +admin.createIndex("ns", "tbl", "c5"); + +// Create the secondary index only if it does not already exist. +boolean ifNotExists = true; +admin.createIndex("ns", "tbl", "c5", ifNotExists); + +// Create the secondary index with options. +Map options = ...; +admin.createIndex("ns", "tbl", "c5", options); +``` + +### Add a new column to a table + +You can add a new, non-partition key column to a table as follows: + +```java +// Add a new column "c6" with the INT data type to the table "ns.tbl". +admin.addNewColumnToTable("ns", "tbl", "c6", DataType.INT) +``` + +:::warning + +You should carefully consider adding a new column to a table because the execution time may vary greatly depending on the underlying storage. Please plan accordingly and consider the following, especially if the database runs in production: + +- **For Cosmos DB for NoSQL and DynamoDB:** Adding a column is almost instantaneous as the table schema is not modified. Only the table metadata stored in a separate table is updated. +- **For Cassandra:** Adding a column will only update the schema metadata and will not modify the existing schema records. The cluster topology is the main factor for the execution time. Changes to the schema metadata are shared to each cluster node via a gossip protocol. Because of this, the larger the cluster, the longer it will take for all nodes to be updated. +- **For relational databases (MySQL, Oracle, etc.):** Adding a column shouldn't take a long time to execute. + +::: + +### Truncate a table + +You can truncate a table as follows: + +```java +// Truncate the table "ns.tbl". +admin.truncateTable("ns", "tbl"); +``` + +### Drop a secondary index + +You can drop a secondary index as follows: + +```java +// Drop the secondary index on column "c5" from table "ns.tbl". If the secondary index does not exist, an exception will be thrown. +admin.dropIndex("ns", "tbl", "c5"); + +// Drop the secondary index only if it exists. +boolean ifExists = true; +admin.dropIndex("ns", "tbl", "c5", ifExists); +``` + +### Drop a table + +You can drop a table as follows: + +```java +// Drop the table "ns.tbl". If the table does not exist, an exception will be thrown. +admin.dropTable("ns", "tbl"); + +// Drop the table only if it exists. +boolean ifExists = true; +admin.dropTable("ns", "tbl", ifExists); +``` + +### Drop a namespace + +You can drop a namespace as follows: + +```java +// Drop the namespace "ns". If the namespace does not exist, an exception will be thrown. +admin.dropNamespace("ns"); + +// Drop the namespace only if it exists. +boolean ifExists = true; +admin.dropNamespace("ns", ifExists); +``` + +### Get existing namespaces + +You can get the existing namespaces as follows: + +```java +Set namespaces = admin.getNamespaceNames(); +``` + +:::note + +This method extracts the namespace names of user tables dynamically. As a result, only namespaces that contain tables are returned. Starting from ScalarDB 4.0, we plan to improve the design to remove this limitation. + +::: + +### Get the tables of a namespace + +You can get the tables of a namespace as follows: + +```java +// Get the tables of the namespace "ns". +Set tables = admin.getNamespaceTableNames("ns"); +``` + +### Get table metadata + +You can get table metadata as follows: + +```java +// Get the table metadata for "ns.tbl". +TableMetadata tableMetadata = admin.getTableMetadata("ns", "tbl"); +``` +### Repair a table + +You can repair the table metadata of an existing table as follows: + +```java +// Repair the table "ns.tbl" with options. +TableMetadata tableMetadata = + TableMetadata.newBuilder() + ... + .build(); +Map options = ...; +admin.repairTable("ns", "tbl", tableMetadata, options); +``` + +### Specify operations for the Coordinator table + +The Coordinator table is used by the [Transactional API](#transactional-api) to track the statuses of transactions. + +When using a transaction manager, you must create the Coordinator table to execute transactions. In addition to creating the table, you can truncate and drop the Coordinator table. + +#### Create the Coordinator table + +You can create the Coordinator table as follows: + +```java +// Create the Coordinator table. +admin.createCoordinatorTables(); + +// Create the Coordinator table only if one does not already exist. +boolean ifNotExist = true; +admin.createCoordinatorTables(ifNotExist); + +// Create the Coordinator table with options. +Map options = ...; +admin.createCoordinatorTables(options); +``` + +#### Truncate the Coordinator table + +You can truncate the Coordinator table as follows: + +```java +// Truncate the Coordinator table. +admin.truncateCoordinatorTables(); +``` + +#### Drop the Coordinator table + +You can drop the Coordinator table as follows: + +```java +// Drop the Coordinator table. +admin.dropCoordinatorTables(); + +// Drop the Coordinator table if one exist. +boolean ifExist = true; +admin.dropCoordinatorTables(ifExist); +``` + +### Import a table + +You can import an existing table to ScalarDB as follows: + +```java +// Import the table "ns.tbl". If the table is already managed by ScalarDB, the target table does not +// exist, or the table does not meet the requirements of the ScalarDB table, an exception will be thrown. +admin.importTable("ns", "tbl", options, overrideColumnsType); +``` + +:::warning + +You should carefully plan to import a table to ScalarDB in production because it will add transaction metadata columns to your database tables and the ScalarDB metadata tables. In this case, there would also be several differences between your database and ScalarDB, as well as some limitations. For details, see [Importing Existing Tables to ScalarDB by Using ScalarDB Schema Loader](./schema-loader-import.mdx). + + +::: + +## Transactional API + +This section explains how to execute transactional operations by using the Transactional API in ScalarDB. + +### Get a `DistributedTransactionManager` instance + +You first need to get a `DistributedTransactionManager` instance to execute transactional operations. + +To get a `DistributedTransactionManager` instance, you can use `TransactionFactory` as follows: + +```java +TransactionFactory transactionFactory = TransactionFactory.create(""); +DistributedTransactionManager transactionManager = transactionFactory.getTransactionManager(); +``` + +After you have executed all transactional operations, you should close the `DistributedTransactionManager` instance as follows: + +```java +transactionManager.close(); +``` + +### Execute transactions + +This subsection explains how to execute transactions with multiple CRUD operations. + +#### Begin or start a transaction + +Before executing transactional CRUD operations, you need to begin or start a transaction. + +You can begin a transaction as follows: + +```java +// Begin a transaction. +DistributedTransaction transaction = transactionManager.begin(); +``` + +Or, you can start a transaction as follows: + +```java +// Start a transaction. +DistributedTransaction transaction = transactionManager.start(); +``` + +Alternatively, you can use the `begin` method for a transaction by specifying a transaction ID as follows: + +```java +// Begin a transaction by specifying a transaction ID. +DistributedTransaction transaction = transactionManager.begin(""); +``` + +Or, you can use the `start` method for a transaction by specifying a transaction ID as follows: + +```java +// Start a transaction by specifying a transaction ID. +DistributedTransaction transaction = transactionManager.start(""); +``` + +:::note + +Specifying a transaction ID is useful when you want to link external systems to ScalarDB. Otherwise, you should use the `begin()` method or the `start()` method. + +When you specify a transaction ID, make sure you specify a unique ID (for example, UUID v4) throughout the system since ScalarDB depends on the uniqueness of transaction IDs for correctness. + +::: + +##### Begin or start a transaction in read-only mode + +You can also begin or start a transaction in read-only mode. In this case, the transaction will not allow any write operations, and it will be optimized for read operations. + +:::note + +Using read-only transactions for read-only operations is strongly recommended to improve performance and reduce resource usage. + +::: + +You can begin or start a transaction in read-only mode as follows: + +```java +// Begin a transaction in read-only mode. +DistributedTransaction transaction = transactionManager.beginReadOnly(); +``` + +```java +// Start a transaction in read-only mode. +DistributedTransaction transaction = transactionManager.startReadOnly(); +``` + +Alternatively, you can use the `beginReadOnly` and `startReadOnly` methods by specifying a transaction ID as follows: + +```java +// Begin a transaction in read-only mode by specifying a transaction ID. +DistributedTransaction transaction = transactionManager.beginReadOnly(""); +``` + +```java +// Start a transaction in read-only mode by specifying a transaction ID. +DistributedTransaction transaction = transactionManager.startReadOnly(""); +``` + +:::note + +Specifying a transaction ID is useful when you want to link external systems to ScalarDB. Otherwise, you should use the `beginReadOnly()` method or the `startReadOnly()` method. + +When you specify a transaction ID, make sure you specify a unique ID (for example, UUID v4) throughout the system since ScalarDB depends on the uniqueness of transaction IDs for correctness. + +::: + +#### Join a transaction + +Joining a transaction is particularly useful in a stateful application where a transaction spans multiple client requests. In such a scenario, the application can start a transaction during the first client request. Then, in subsequent client requests, the application can join the ongoing transaction by using the `join()` method. + +You can join an ongoing transaction that has already begun by specifying the transaction ID as follows: + +```java +// Join a transaction. +DistributedTransaction transaction = transactionManager.join(""); +``` + +:::note + +To get the transaction ID with `getId()`, you can specify the following: + +```java +tx.getId(); +``` + +::: + +#### Resume a transaction + +Resuming a transaction is particularly useful in a stateful application where a transaction spans multiple client requests. In such a scenario, the application can start a transaction during the first client request. Then, in subsequent client requests, the application can resume the ongoing transaction by using the `resume()` method. + +You can resume an ongoing transaction that you have already begun by specifying a transaction ID as follows: + +```java +// Resume a transaction. +DistributedTransaction transaction = transactionManager.resume(""); +``` + +:::note + +To get the transaction ID with `getId()`, you can specify the following: + +```java +tx.getId(); +``` + +::: + +#### Implement CRUD operations + +The following sections describe key construction and CRUD operations. + +:::note + +Although all the builders of the CRUD operations can specify consistency by using the `consistency()` methods, those methods are ignored. Instead, the `LINEARIZABLE` consistency level is always used in transactions. + +::: + +##### Key construction + +Most CRUD operations need to specify `Key` objects (partition-key, clustering-key, etc.). So, before moving on to CRUD operations, the following explains how to construct a `Key` object. + +For a single column key, you can use `Key.of()` methods to construct the key as follows: + +```java +// For a key that consists of a single column of INT. +Key key1 = Key.ofInt("col1", 1); + +// For a key that consists of a single column of BIGINT. +Key key2 = Key.ofBigInt("col1", 100L); + +// For a key that consists of a single column of DOUBLE. +Key key3 = Key.ofDouble("col1", 1.3d); + +// For a key that consists of a single column of TEXT. +Key key4 = Key.ofText("col1", "value"); +``` + +For a key that consists of two to five columns, you can use the `Key.of()` method to construct the key as follows. Similar to `ImmutableMap.of()` in Guava, you need to specify column names and values in turns: + +```java +// For a key that consists of two to five columns. +Key key1 = Key.of("col1", 1, "col2", 100L); +Key key2 = Key.of("col1", 1, "col2", 100L, "col3", 1.3d); +Key key3 = Key.of("col1", 1, "col2", 100L, "col3", 1.3d, "col4", "value"); +Key key4 = Key.of("col1", 1, "col2", 100L, "col3", 1.3d, "col4", "value", "col5", false); +``` + +For a key that consists of more than five columns, we can use the builder to construct the key as follows: + +```java +// For a key that consists of more than five columns. +Key key = Key.newBuilder() + .addInt("col1", 1) + .addBigInt("col2", 100L) + .addDouble("col3", 1.3d) + .addText("col4", "value") + .addBoolean("col5", false) + .addInt("col6", 100) + .build(); +``` + +##### `Get` operation + +`Get` is an operation to retrieve a single record specified by a primary key. + +You need to create a `Get` object first, and then you can execute the object by using the `transaction.get()` method as follows: + +```java +// Create a `Get` operation. +Key partitionKey = Key.ofInt("c1", 10); +Key clusteringKey = Key.of("c2", "aaa", "c3", 100L); + +Get get = + Get.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .clusteringKey(clusteringKey) + .projections("c1", "c2", "c3", "c4") + .where(ConditionBuilder.column("c1").isNotEqualToInt(10)) + .build(); + +// Execute the `Get` operation. +Optional result = transaction.get(get); +``` + +You can specify projections to choose which columns are returned. + +###### Use the `WHERE` clause + +You can also specify arbitrary conditions by using the `where()` method. If the retrieved record does not match the conditions specified by the `where()` method, `Option.empty()` will be returned. As an argument of the `where()` method, you can specify a condition, an AND-wise condition set, or an OR-wise condition set. After calling the `where()` method, you can add more conditions or condition sets by using the `and()` method or `or()` method as follows: + +```java +// Create a `Get` operation with condition sets. +Get get = + Get.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .clusteringKey(clusteringKey) + .where( + ConditionSetBuilder.condition(ConditionBuilder.column("c1").isLessThanInt(10)) + .or(ConditionBuilder.column("c1").isGreaterThanInt(20)) + .build()) + .and( + ConditionSetBuilder.condition(ConditionBuilder.column("c2").isLikeText("a%")) + .or(ConditionBuilder.column("c2").isLikeText("b%")) + .build()) + .build(); +``` + +:::note + +In the `where()` condition method chain, the conditions must be an AND-wise junction of `ConditionalExpression` or `OrConditionSet` (known as conjunctive normal form) like the above example or an OR-wise junction of `ConditionalExpression` or `AndConditionSet` (known as disjunctive normal form). + +::: + +For more details about available conditions and condition sets, see the and pages in the Javadoc. + +###### Handle `Result` objects + +The `Get` operation and `Scan` operation return `Result` objects. The following shows how to handle `Result` objects. + +You can get a column value of a result by using `get("")` methods as follows: + +```java +// Get the BOOLEAN value of a column. +boolean booleanValue = result.getBoolean(""); + +// Get the INT value of a column. +int intValue = result.getInt(""); + +// Get the BIGINT value of a column. +long bigIntValue = result.getBigInt(""); + +// Get the FLOAT value of a column. +float floatValue = result.getFloat(""); + +// Get the DOUBLE value of a column. +double doubleValue = result.getDouble(""); + +// Get the TEXT value of a column. +String textValue = result.getText(""); + +// Get the BLOB value of a column as a `ByteBuffer`. +ByteBuffer blobValue = result.getBlob(""); + +// Get the BLOB value of a column as a `byte` array. +byte[] blobValueAsBytes = result.getBlobAsBytes(""); + +// Get the DATE value of a column as a `LocalDate`. +LocalDate dateValue = result.getDate(""); + +// Get the TIME value of a column as a `LocalTime`. +LocalTime timeValue = result.getTime(""); + +// Get the TIMESTAMP value of a column as a `LocalDateTime`. +LocalDateTime timestampValue = result.getTimestamp(""); + +// Get the TIMESTAMPTZ value of a column as a `Instant`. +Instant timestampTZValue = result.getTimestampTZ(""); +``` + +And if you need to check if a value of a column is null, you can use the `isNull("")` method. + +``` java +// Check if a value of a column is null. +boolean isNull = result.isNull(""); +``` + +For more details, see the page in the Javadoc. + +###### Execute `Get` by using a secondary index + +You can execute a `Get` operation by using a secondary index. + +Instead of specifying a partition key, you can specify an index key (indexed column) to use a secondary index as follows: + +```java +// Create a `Get` operation by using a secondary index. +Key indexKey = Key.ofFloat("c4", 1.23F); + +Get get = + Get.newBuilder() + .namespace("ns") + .table("tbl") + .indexKey(indexKey) + .projections("c1", "c2", "c3", "c4") + .where(ConditionBuilder.column("c1").isNotEqualToInt(10)) + .build(); + +// Execute the `Get` operation. +Optional result = transaction.get(get); +``` + +You can also specify arbitrary conditions by using the `where()` method. For details, see [Use the `WHERE` clause](#use-the-where-clause). + +:::note + +If the result has more than one record, `transaction.get()` will throw an exception. If you want to handle multiple results, see [Execute `Scan` by using a secondary index](#execute-scan-by-using-a-secondary-index). + +::: + +##### `Scan` operation + +`Scan` is an operation to retrieve multiple records within a partition. You can specify clustering-key boundaries and orderings for clustering-key columns in `Scan` operations. To execute a `Scan` operation, you can use the `transaction.scan()` method or the `transaction.getScanner()` method: + +- `transaction.scan()`: + - This method immediately executes the given `Scan` operation and returns a list of all matching records. It is suitable when the result set is expected to be small enough to fit in memory. +- `transaction.getScanner()`: + - This method returns a `Scanner` object that allows you to iterate over the result set lazily. It is useful when the result set may be large, as it avoids loading all records into memory at once. + +You need to create a `Scan` object first, and then you can execute the object by using the `transaction.scan()` method or the `transaction.getScanner()` method as follows: + +```java +// Create a `Scan` operation. +Key partitionKey = Key.ofInt("c1", 10); +Key startClusteringKey = Key.of("c2", "aaa", "c3", 100L); +Key endClusteringKey = Key.of("c2", "aaa", "c3", 300L); + +Scan scan = + Scan.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .start(startClusteringKey, true) // Include startClusteringKey + .end(endClusteringKey, false) // Exclude endClusteringKey + .projections("c1", "c2", "c3", "c4") + .orderings(Scan.Ordering.desc("c2"), Scan.Ordering.asc("c3")) + .where(ConditionBuilder.column("c1").isNotEqualToInt(10)) + .limit(10) + .build(); + +// Execute the `Scan` operation by using the `transaction.scan()` method. +List results = transaction.scan(scan); + +// Or, execute the `Scan` operation by using the `transaction.getScanner()` method. +try (TransactionCrudOperable.Scanner scanner = transaction.getScanner(scan)) { + // Fetch the next result from the scanner + Optional result = scanner.one(); + + // Fetch all remaining results from the scanner + List allResults = scanner.all(); +} +``` + +You can omit the clustering-key boundaries or specify either a `start` boundary or an `end` boundary. If you don't specify `orderings`, you will get results ordered by the clustering order that you defined when creating the table. + +In addition, you can specify `projections` to choose which columns are returned and use `limit` to specify the number of records to return in `Scan` operations. + +###### Use the `WHERE` clause + +You can also specify arbitrary conditions by using the `where()` method to filter scanned records. As an argument of the `where()` method, you can specify a condition, an AND-wise condition set, or an OR-wise condition set. After calling the `where()` method, you can add more conditions or condition sets by using the `and()` method or `or()` method as follows: + +```java +// Create a `Scan` operation with condition sets. +Scan scan = + Scan.newBuilder() + .namespace("ns") + .table("tbl") + .all() + .where( + ConditionSetBuilder.condition(ConditionBuilder.column("c1").isLessThanInt(10)) + .or(ConditionBuilder.column("c1").isGreaterThanInt(20)) + .build()) + .and( + ConditionSetBuilder.condition(ConditionBuilder.column("c2").isLikeText("a%")) + .or(ConditionBuilder.column("c2").isLikeText("b%")) + .build()) + .limit(10) + .build(); +``` + +:::note + +In the `where()` condition method chain, the conditions must be an AND-wise junction of `ConditionalExpression` or `OrConditionSet` (known as conjunctive normal form) like the above example or an OR-wise junction of `ConditionalExpression` or `AndConditionSet` (known as disjunctive normal form). + +::: + +For more details about available conditions and condition sets, see the and pages in the Javadoc. + +###### Execute `Scan` by using a secondary index + +You can execute a `Scan` operation by using a secondary index. + +Instead of specifying a partition key, you can specify an index key (indexed column) to use a secondary index as follows: + +```java +// Create a `Scan` operation by using a secondary index. +Key indexKey = Key.ofFloat("c4", 1.23F); + +Scan scan = + Scan.newBuilder() + .namespace("ns") + .table("tbl") + .indexKey(indexKey) + .projections("c1", "c2", "c3", "c4") + .where(ConditionBuilder.column("c1").isNotEqualToInt(10)) + .limit(10) + .build(); + +// Execute the `Scan` operation. +List results = transaction.scan(scan); +``` + +You can also specify arbitrary conditions using the `where()` method. For details, see [Use the `WHERE` clause](#use-the-where-clause-1). + +:::note + +You can't specify clustering-key boundaries and orderings in `Scan` by using a secondary index. + +::: + +###### Execute cross-partition `Scan` without specifying a partition key to retrieve all the records of a table + +You can execute a `Scan` operation across all partitions, which we call *cross-partition scan*, without specifying a partition key by enabling the following configuration in the ScalarDB properties file. + +```properties +scalar.db.cross_partition_scan.enabled=true +``` + +:::warning + +For non-JDBC databases, transactions could be executed at read-committed snapshot isolation (`SNAPSHOT`), which is a lower isolation level, even if you enable cross-partition scan with the `SERIALIZABLE` isolation level. When using non-JDBC databases, use cross-partition scan only if consistency does not matter for your transactions. + +::: + +Instead of calling the `partitionKey()` method in the builder, you can call the `all()` method to scan a table without specifying a partition key as follows: + +```java +// Create a `Scan` operation without specifying a partition key. +Scan scan = + Scan.newBuilder() + .namespace("ns") + .table("tbl") + .all() + .projections("c1", "c2", "c3", "c4") + .limit(10) + .build(); + +// Execute the `Scan` operation. +List results = transaction.scan(scan); +``` + +:::note + +You can't specify any orderings in cross-partition `Scan` when using non-JDBC databases. For details on how to use cross-partition `Scan` with filtering or ordering, see [Execute cross-partition `Scan` with filtering and ordering](#execute-cross-partition-scan-with-filtering-and-ordering). + +::: + +###### Execute cross-partition `Scan` with filtering and ordering + +By enabling the cross-partition scan option with filtering and ordering as follows, you can execute a cross-partition `Scan` operation with flexible conditions and orderings: + +```properties +scalar.db.cross_partition_scan.enabled=true +scalar.db.cross_partition_scan.filtering.enabled=true +scalar.db.cross_partition_scan.ordering.enabled=true +``` + +:::note + +You can't enable `scalar.db.cross_partition_scan.ordering` in non-JDBC databases. + +::: + +You can call the `where()` and `ordering()` methods after calling the `all()` method to specify arbitrary conditions and orderings as follows: + +```java +// Create a `Scan` operation with arbitrary conditions and orderings. +Scan scan = + Scan.newBuilder() + .namespace("ns") + .table("tbl") + .all() + .where(ConditionBuilder.column("c1").isNotEqualToInt(10)) + .projections("c1", "c2", "c3", "c4") + .orderings(Scan.Ordering.desc("c3"), Scan.Ordering.asc("c4")) + .limit(10) + .build(); + +// Execute the `Scan` operation. +List results = transaction.scan(scan); +``` + +For details about the `WHERE` clause, see [Use the `WHERE` clause](#use-the-where-clause-1). + +##### `Put` operation + +:::note + +The `Put` operation is deprecated as of ScalarDB 3.13 and will be removed in a future release. Instead of using the `Put` operation, use the `Insert` operation, the `Upsert` operation, or the `Update` operation. + +::: + +`Put` is an operation to put a record specified by a primary key. The operation behaves as an upsert operation for a record, in which the operation updates the record if the record exists or inserts the record if the record does not exist. + +:::note + +When you update an existing record, you need to read the record by using `Get` or `Scan` before using a `Put` operation. Otherwise, the operation will fail due to a conflict. This occurs because of the specification of ScalarDB to manage transactions properly. Instead of reading the record explicitly, you can enable implicit pre-read. For details, see [Enable implicit pre-read for `Put` operations](#enable-implicit-pre-read-for-put-operations). + +::: + +You need to create a `Put` object first, and then you can execute the object by using the `transaction.put()` method as follows: + +```java +// Create a `Put` operation. +Key partitionKey = Key.ofInt("c1", 10); +Key clusteringKey = Key.of("c2", "aaa", "c3", 100L); + +Put put = + Put.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .clusteringKey(clusteringKey) + .floatValue("c4", 1.23F) + .doubleValue("c5", 4.56) + .build(); + +// Execute the `Put` operation. +transaction.put(put); +``` + +You can also put a record with `null` values as follows: + +```java +Put put = + Put.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .clusteringKey(clusteringKey) + .floatValue("c4", null) + .doubleValue("c5", null) + .build(); +``` + +###### Enable implicit pre-read for `Put` operations + +In Consensus Commit, an application must read a record before mutating the record with `Put` and `Delete` operations to obtain the latest states of the record if the record exists. Instead of reading the record explicitly, you can enable *implicit pre-read*. By enabling implicit pre-read, if an application does not read the record explicitly in a transaction, ScalarDB will read the record on behalf of the application before committing the transaction. + +You can enable implicit pre-read for a `Put` operation by specifying `enableImplicitPreRead()` in the `Put` operation builder as follows: + +```java +Put put = + Put.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .clusteringKey(clusteringKey) + .floatValue("c4", 1.23F) + .doubleValue("c5", 4.56) + .enableImplicitPreRead() + .build(); +``` + +:::note + +If you are certain that a record you are trying to mutate does not exist, you should not enable implicit pre-read for the `Put` operation for better performance. For example, if you load initial data, you should not enable implicit pre-read. A `Put` operation without implicit pre-read is faster than `Put` operation with implicit pre-read because the operation skips an unnecessary read. + +::: + +##### `Insert` operation + +`Insert` is an operation to insert an entry into the underlying storage through a transaction. If the entry already exists, a conflict error will occur. + +You need to create an `Insert` object first, and then you can execute the object by using the `transaction.insert()` method as follows: + +```java +// Create an `Insert` operation. +Key partitionKey = Key.ofInt("c1", 10); +Key clusteringKey = Key.of("c2", "aaa", "c3", 100L); + +Insert insert = + Insert.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .clusteringKey(clusteringKey) + .floatValue("c4", 1.23F) + .doubleValue("c5", 4.56) + .build(); + +// Execute the `Insert` operation. +transaction.insert(insert); +``` + +##### `Upsert` operation + +`Upsert` is an operation to insert an entry into or update an entry in the underlying storage through a transaction. If the entry already exists, it will be updated; otherwise, the entry will be inserted. + +You need to create an `Upsert` object first, and then you can execute the object by using the `transaction.upsert()` method as follows: + +```java +// Create an `Upsert` operation. +Key partitionKey = Key.ofInt("c1", 10); +Key clusteringKey = Key.of("c2", "aaa", "c3", 100L); + +Upsert upsert = + Upsert.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .clusteringKey(clusteringKey) + .floatValue("c4", 1.23F) + .doubleValue("c5", 4.56) + .build(); + +// Execute the `Upsert` operation. +transaction.upsert(upsert); +``` + +##### `Update` operation + +`Update` is an operation to update an entry in the underlying storage through a transaction. If the entry does not exist, the operation will not make any changes. + +You need to create an `Update` object first, and then you can execute the object by using the `transaction.update()` method as follows: + +```java +// Create an `Update` operation. +Key partitionKey = Key.ofInt("c1", 10); +Key clusteringKey = Key.of("c2", "aaa", "c3", 100L); + +Update update = + Update.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .clusteringKey(clusteringKey) + .floatValue("c4", 1.23F) + .doubleValue("c5", 4.56) + .build(); + +// Execute the `Update` operation. +transaction.update(update); +``` + +##### `Delete` operation + +`Delete` is an operation to delete a record specified by a primary key. + +:::note + +When you delete a record, you don't have to read the record beforehand because implicit pre-read is always enabled for `Delete` operations. + +::: + +You need to create a `Delete` object first, and then you can execute the object by using the `transaction.delete()` method as follows: + +```java +// Create a `Delete` operation. +Key partitionKey = Key.ofInt("c1", 10); +Key clusteringKey = Key.of("c2", "aaa", "c3", 100L); + +Delete delete = + Delete.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .clusteringKey(clusteringKey) + .build(); + +// Execute the `Delete` operation. +transaction.delete(delete); +``` + +##### `Put`, `Delete`, and `Update` with a condition + +You can write arbitrary conditions (for example, a bank account balance must be equal to or more than zero) that you require a transaction to meet before being committed by implementing logic that checks the conditions in the transaction. Alternatively, you can write simple conditions in a mutation operation, such as `Put`, `Delete`, and `Update`. + +When a `Put`, `Delete`, or `Update` operation includes a condition, the operation is executed only if the specified condition is met. If the condition is not met when the operation is executed, an exception called `UnsatisfiedConditionException` will be thrown. + +:::note + +When you specify a condition in a `Put` operation, you need to read the record beforehand or enable implicit pre-read. + +::: + +###### Conditions for `Put` + +You can specify a condition in a `Put` operation as follows: + +```java +// Build a condition. +MutationCondition condition = + ConditionBuilder.putIf(ConditionBuilder.column("c4").isEqualToFloat(0.0F)) + .and(ConditionBuilder.column("c5").isEqualToDouble(0.0)) + .build(); + +Put put = + Put.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .clusteringKey(clusteringKey) + .floatValue("c4", 1.23F) + .doubleValue("c5", 4.56) + .condition(condition) // condition + .build(); + +// Execute the `Put` operation. +transaction.put(put); +``` + +In addition to using the `putIf` condition, you can specify the `putIfExists` and `putIfNotExists` conditions as follows: + +```java +// Build a `putIfExists` condition. +MutationCondition putIfExistsCondition = ConditionBuilder.putIfExists(); + +// Build a `putIfNotExists` condition. +MutationCondition putIfNotExistsCondition = ConditionBuilder.putIfNotExists(); +``` + +###### Conditions for `Delete` + +You can specify a condition in a `Delete` operation as follows: + +```java +// Build a condition. +MutationCondition condition = + ConditionBuilder.deleteIf(ConditionBuilder.column("c4").isEqualToFloat(0.0F)) + .and(ConditionBuilder.column("c5").isEqualToDouble(0.0)) + .build(); + +Delete delete = + Delete.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .clusteringKey(clusteringKey) + .condition(condition) // condition + .build(); + +// Execute the `Delete` operation. +transaction.delete(delete); +``` + +In addition to using the `deleteIf` condition, you can specify the `deleteIfExists` condition as follows: + +```java +// Build a `deleteIfExists` condition. +MutationCondition deleteIfExistsCondition = ConditionBuilder.deleteIfExists(); +``` + +###### Conditions for `Update` + +You can specify a condition in an `Update` operation as follows: + +```java +// Build a condition. +MutationCondition condition = + ConditionBuilder.updateIf(ConditionBuilder.column("c4").isEqualToFloat(0.0F)) + .and(ConditionBuilder.column("c5").isEqualToDouble(0.0)) + .build(); + +Update update = + Update.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .clusteringKey(clusteringKey) + .floatValue("c4", 1.23F) + .doubleValue("c5", 4.56) + .condition(condition) // condition + .build(); + +// Execute the `Update` operation. +transaction.update(update); +``` + +In addition to using the `updateIf` condition, you can specify the `updateIfExists` condition as follows: + +```java +// Build a `updateIfExists` condition. +MutationCondition updateIfExistsCondition = ConditionBuilder.updateIfExists(); +``` + +##### Mutate operation + +Mutate is an operation to execute multiple operations for `Put`, `Insert`, `Upsert`, `Update`, and `Delete`. + +You need to create mutation objects first, and then you can execute the objects by using the `transaction.mutate()` method as follows: + +```java +// Create `Put` and `Delete` operations. +Key partitionKey = Key.ofInt("c1", 10); + +Key clusteringKeyForPut = Key.of("c2", "aaa", "c3", 100L); + +Put put = + Put.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .clusteringKey(clusteringKeyForPut) + .floatValue("c4", 1.23F) + .doubleValue("c5", 4.56) + .build(); + +Key clusteringKeyForDelete = Key.of("c2", "bbb", "c3", 200L); + +Delete delete = + Delete.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .clusteringKey(clusteringKeyForDelete) + .build(); + +// Execute the operations. +transaction.mutate(Arrays.asList(put, delete)); +``` + +##### Default namespace for CRUD operations + +A default namespace for all CRUD operations can be set by using a property in the ScalarDB configuration. + +```properties +scalar.db.default_namespace_name= +``` + +Any operation that does not specify a namespace will use the default namespace set in the configuration. + +```java +// This operation will target the default namespace. +Scan scanUsingDefaultNamespace = + Scan.newBuilder() + .table("tbl") + .all() + .build(); +// This operation will target the "ns" namespace. +Scan scanUsingSpecifiedNamespace = + Scan.newBuilder() + .namespace("ns") + .table("tbl") + .all() + .build(); +``` + +##### Operation attributes + +The operation attribute is a key-value pair that can be used to store additional information about an operation. You can set operation attributes by using the `attribute()` or `attributes()` method in the operation builder, as shown below: + +```java +// Set operation attributes in the `Get` operation. +Get get = Get.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .clusteringKey(clusteringKey) + .attribute("attribute1", "value1") + .attributes(ImmutableMap.of("attribute2", "value2", "attribute3", "value3")) + .build(); + +// Set operation attributes in the `Scan` operation. +Scan scan = Scan.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .projections("c1", "c2", "c3", "c4") + .attribute("attribute1", "value1") + .attributes(ImmutableMap.of("attribute2", "value2", "attribute3", "value3")) + .build(); + +// Set operation attributes in the `Insert` operation. +Insert insert = Insert.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .clusteringKey(clusteringKey) + .floatValue("c4", 1.23F) + .doubleValue("c5", 4.56) + .attribute("attribute1", "value1") + .attributes(ImmutableMap.of("attribute2", "value2", "attribute3", "value3")) + .build(); + +// Set operation attributes in the `Upsert` operation. +Upsert upsert = Upsert.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .clusteringKey(clusteringKey) + .floatValue("c4", 1.23F) + .doubleValue("c5", 4.56) + .attribute("attribute1", "value1") + .attributes(ImmutableMap.of("attribute2", "value2", "attribute3", "value3")) + .build(); + +// Set operation attributes in the `Update` operation. +Update update = Update.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .clusteringKey(clusteringKey) + .floatValue("c4", 1.23F) + .doubleValue("c5", 4.56) + .attribute("attribute1", "value1") + .attributes(ImmutableMap.of("attribute2", "value2", "attribute3", "value3")) + .build(); + +// Set operation attributes in the `Delete` operation. +Delete delete = Delete.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .clusteringKey(clusteringKey) + .attribute("attribute1", "value1") + .attributes(ImmutableMap.of("attribute2", "value2", "attribute3", "value3")) + .build(); +``` + +:::note + +ScalarDB currently has no available operation attributes. + +::: + +#### Commit a transaction + +After executing CRUD operations, you need to commit a transaction to finish it. + +You can commit a transaction as follows: + +```java +// Commit a transaction. +transaction.commit(); +``` + +#### Roll back or abort a transaction + +If an error occurs when executing a transaction, you can roll back or abort the transaction. + +You can roll back a transaction as follows: + +```java +// Roll back a transaction. +transaction.rollback(); +``` + +Or, you can abort a transaction as follows: + +```java +// Abort a transaction. +transaction.abort(); +``` + +For details about how to handle exceptions in ScalarDB, see [How to handle exceptions](#how-to-handle-exceptions). + +### Execute transactions without beginning or starting a transaction + +You can execute transactional operations without beginning or starting a transaction. In this case, ScalarDB will automatically begin a transaction before executing the operations and commit the transaction after executing the operations. This section explains how to execute transactions without beginning or starting a transaction. + +#### Execute `Get` operation + +`Get` is an operation to retrieve a single record specified by a primary key. + +You need to create a `Get` object first, and then you can execute the object by using the `transactionManager.get()` method as follows: + +```java +// Create a `Get` operation. +Key partitionKey = Key.ofInt("c1", 10); +Key clusteringKey = Key.of("c2", "aaa", "c3", 100L); + +Get get = + Get.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .clusteringKey(clusteringKey) + .projections("c1", "c2", "c3", "c4") + .build(); + +// Execute the `Get` operation. +Optional result = transactionManager.get(get); +``` + +For details about the `Get` operation, see [`Get` operation](#get-operation). + +#### Execute `Scan` operation + +`Scan` is an operation to retrieve multiple records within a partition. You can specify clustering-key boundaries and orderings for clustering-key columns in `Scan` operations. To execute a `Scan` operation, you can use the `transactionManager.scan()` method or the `transactionManager.getScanner()` method: + +- `transactionManager.scan()`: + - This method immediately executes the given `Scan` operation and returns a list of all matching records. It is suitable when the result set is expected to be small enough to fit in memory. +- `transactionManager.getScanner()`: + - This method returns a `Scanner` object that allows you to iterate over the result set lazily. It is useful when the result set may be large, as it avoids loading all records into memory at once. + +You need to create a `Scan` object first, and then you can execute the object by using the `transactionManager.scan()` method or the `transactionManager.getScanner()` method as follows: + +```java +// Create a `Scan` operation. +Key partitionKey = Key.ofInt("c1", 10); +Key startClusteringKey = Key.of("c2", "aaa", "c3", 100L); +Key endClusteringKey = Key.of("c2", "aaa", "c3", 300L); + +Scan scan = + Scan.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .start(startClusteringKey, true) // Include startClusteringKey + .end(endClusteringKey, false) // Exclude endClusteringKey + .projections("c1", "c2", "c3", "c4") + .orderings(Scan.Ordering.desc("c2"), Scan.Ordering.asc("c3")) + .limit(10) + .build(); + +// Execute the `Scan` operation by using the `transactionManager.scan()` method. +List results = transactionManager.scan(scan); + +// Or, execute the `Scan` operation by using the `transactionManager.getScanner()` method. +try (TransactionManagerCrudOperable.Scanner scanner = transactionManager.getScanner(scan)) { + // Fetch the next result from the scanner + Optional result = scanner.one(); + + // Fetch all remaining results from the scanner + List allResults = scanner.all(); +} +``` + +For details about the `Scan` operation, see [`Scan` operation](#scan-operation). + +#### Execute `Put` operation + +:::note + +The `Put` operation is deprecated as of ScalarDB 3.13 and will be removed in a future release. Instead of using the `Put` operation, use the `Insert` operation, the `Upsert` operation, or the `Update` operation. + +::: + +You need to create a `Put` object first, and then you can execute the object by using the `transactionManager.put()` method as follows: + +```java +// Create a `Put` operation. +Key partitionKey = Key.ofInt("c1", 10); +Key clusteringKey = Key.of("c2", "aaa", "c3", 100L); + +Put put = + Put.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .clusteringKey(clusteringKey) + .floatValue("c4", 1.23F) + .doubleValue("c5", 4.56) + .build(); + +// Execute the `Put` operation. +transactionManager.put(put); +``` + +For details about the `Put` operation, see [`Put` operation](#put-operation). + +#### Execute `Insert` operation + +`Insert` is an operation to insert an entry into the underlying storage through a transaction. If the entry already exists, a conflict error will occur. + +You need to create an `Insert` object first, and then you can execute the object by using the `transactionManager.insert()` method as follows: + +```java +// Create an `Insert` operation. +Key partitionKey = Key.ofInt("c1", 10); +Key clusteringKey = Key.of("c2", "aaa", "c3", 100L); + +Insert insert = + Insert.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .clusteringKey(clusteringKey) + .floatValue("c4", 1.23F) + .doubleValue("c5", 4.56) + .build(); + +// Execute the `Insert` operation. +transactionManager.insert(insert); +``` + +For details about the `Insert` operation, see [`Insert` operation](#insert-operation). + +#### Execute `Upsert` operation + +`Upsert` is an operation to insert an entry into or update an entry in the underlying storage through a transaction. If the entry already exists, it will be updated; otherwise, the entry will be inserted. + +You need to create an `Upsert` object first, and then you can execute the object by using the `transactionManager.upsert()` method as follows: + +```java +// Create an `Upsert` operation. +Key partitionKey = Key.ofInt("c1", 10); +Key clusteringKey = Key.of("c2", "aaa", "c3", 100L); + +Upsert upsert = + Upsert.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .clusteringKey(clusteringKey) + .floatValue("c4", 1.23F) + .doubleValue("c5", 4.56) + .build(); + +// Execute the `Upsert` operation. +transactionManager.upsert(upsert); +``` + +For details about the `Insert` operation, see [`Upsert` operation](#upsert-operation). + +#### Execute `Update` operation + +`Update` is an operation to update an entry in the underlying storage through a transaction. If the entry does not exist, the operation will not make any changes. + +You need to create an `Update` object first, and then you can execute the object by using the `transactionManager.update()` method as follows: + +```java +// Create an `Update` operation. +Key partitionKey = Key.ofInt("c1", 10); +Key clusteringKey = Key.of("c2", "aaa", "c3", 100L); + +Update update = + Update.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .clusteringKey(clusteringKey) + .floatValue("c4", 1.23F) + .doubleValue("c5", 4.56) + .build(); + +// Execute the `Update` operation. +transactionManager.update(update); +``` + +For details about the `Update` operation, see [`Update` operation](#update-operation). + +#### Execute `Delete` operation + +`Delete` is an operation to delete a record specified by a primary key. + +You need to create a `Delete` object first, and then you can execute the object by using the `transaction.delete()` method as follows: + +```java +// Create a `Delete` operation. +Key partitionKey = Key.ofInt("c1", 10); +Key clusteringKey = Key.of("c2", "aaa", "c3", 100L); + +Delete delete = + Delete.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .clusteringKey(clusteringKey) + .build(); + +// Execute the `Delete` operation. +transactionManager.delete(delete); +``` + +For details about the `Delete` operation, see [`Delete` operation](#delete-operation). + +#### Execute Mutate operation + +Mutate is an operation to execute multiple mutations (`Put`, `Insert`, `Upsert`, `Update`, and `Delete` operations). + +You need to create mutation objects first, and then you can execute the objects by using the `transactionManager.mutate()` method as follows: + +```java +// Create `Put` and `Delete` operations. +Key partitionKey = Key.ofInt("c1", 10); + +Key clusteringKeyForPut = Key.of("c2", "aaa", "c3", 100L); + +Put put = + Put.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .clusteringKey(clusteringKeyForPut) + .floatValue("c4", 1.23F) + .doubleValue("c5", 4.56) + .build(); + +Key clusteringKeyForDelete = Key.of("c2", "bbb", "c3", 200L); + +Delete delete = + Delete.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .clusteringKey(clusteringKeyForDelete) + .build(); + +// Execute the operations. +transactionManager.mutate(Arrays.asList(put, delete)); +``` + +For details about the Mutate operation, see [Mutate operation](#mutate-operation). + +In addition, for details about how to handle exceptions in ScalarDB, see [How to handle exceptions](#how-to-handle-exceptions). + +## How to handle exceptions + +When executing a transaction, you will also need to handle exceptions properly. + +:::warning + +If you don't handle exceptions properly, you may face anomalies or data inconsistency. + +::: + +The following sample code shows how to handle exceptions: + +```java +public class Sample { + public static void main(String[] args) throws Exception { + TransactionFactory factory = TransactionFactory.create(""); + DistributedTransactionManager transactionManager = factory.getTransactionManager(); + + int retryCount = 0; + TransactionException lastException = null; + + while (true) { + if (retryCount++ > 0) { + // Retry the transaction three times maximum. + if (retryCount >= 3) { + // Throw the last exception if the number of retries exceeds the maximum. + throw lastException; + } + + // Sleep 100 milliseconds before retrying the transaction. + TimeUnit.MILLISECONDS.sleep(100); + } + + DistributedTransaction transaction = null; + try { + // Begin a transaction. + transaction = transactionManager.begin(); + + // Execute CRUD operations in the transaction. + Optional result = transaction.get(...); + List results = transaction.scan(...); + transaction.put(...); + transaction.delete(...); + + // Commit the transaction. + transaction.commit(); + } catch (UnsatisfiedConditionException e) { + // You need to handle `UnsatisfiedConditionException` only if a mutation operation specifies a condition. + // This exception indicates the condition for the mutation operation is not met. + + try { + transaction.rollback(); + } catch (RollbackException ex) { + // Rolling back the transaction failed. Since the transaction should eventually recover, + // you don't need to do anything further. You can simply log the occurrence here. + } + + // You can handle the exception here, according to your application requirements. + + return; + } catch (UnknownTransactionStatusException e) { + // If you catch `UnknownTransactionStatusException` when committing the transaction, + // it indicates that the status of the transaction, whether it was successful or not, is unknown. + // In such a case, you need to check if the transaction is committed successfully or not and + // retry the transaction if it failed. How to identify a transaction status is delegated to users. + return; + } catch (TransactionException e) { + // For other exceptions, you can try retrying the transaction. + + // For `CrudConflictException`, `CommitConflictException`, and `TransactionNotFoundException`, + // you can basically retry the transaction. However, for the other exceptions, the transaction + // will still fail if the cause of the exception is non-transient. In such a case, you will + // exhaust the number of retries and throw the last exception. + + if (transaction != null) { + try { + transaction.rollback(); + } catch (RollbackException ex) { + // Rolling back the transaction failed. The transaction should eventually recover, + // so you don't need to do anything further. You can simply log the occurrence here. + } + } + + lastException = e; + } + } + } +} +``` + +### `TransactionException` and `TransactionNotFoundException` + +The `begin()` API could throw `TransactionException` or `TransactionNotFoundException`: + +- If you catch `TransactionException`, this exception indicates that the transaction has failed to begin due to transient or non-transient faults. You can try retrying the transaction, but you may not be able to begin the transaction due to non-transient faults. +- If you catch `TransactionNotFoundException`, this exception indicates that the transaction has failed to begin due to transient faults. In this case, you can retry the transaction. + +The `join()` API could also throw `TransactionNotFoundException`. You can handle this exception in the same way that you handle the exceptions for the `begin()` API. + +### `CrudException` and `CrudConflictException` + +The APIs for CRUD operations (`get()`, `scan()`, `put()`, `delete()`, and `mutate()`) could throw `CrudException` or `CrudConflictException`: + +- If you catch `CrudException`, this exception indicates that the transaction CRUD operation has failed due to transient or non-transient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is non-transient. +- If you catch `CrudConflictException`, this exception indicates that the transaction CRUD operation has failed due to transient faults (for example, a conflict error). In this case, you can retry the transaction from the beginning. + +### `UnsatisfiedConditionException` + +The APIs for mutation operations (`put()`, `delete()`, and `mutate()`) could also throw `UnsatisfiedConditionException`. + +If you catch `UnsatisfiedConditionException`, this exception indicates that the condition for the mutation operation is not met. You can handle this exception according to your application requirements. + +### `CommitException`, `CommitConflictException`, and `UnknownTransactionStatusException` + +The `commit()` API could throw `CommitException`, `CommitConflictException`, or `UnknownTransactionStatusException`: + +- If you catch `CommitException`, this exception indicates that committing the transaction fails due to transient or non-transient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is non-transient. +- If you catch `CommitConflictException`, this exception indicates that committing the transaction has failed due to transient faults (for example, a conflict error). In this case, you can retry the transaction from the beginning. +- If you catch `UnknownTransactionStatusException`, this exception indicates that the status of the transaction, whether it was successful or not, is unknown. In this case, you need to check if the transaction is committed successfully and retry the transaction if it has failed. + +How to identify a transaction status is delegated to users. You may want to create a transaction status table and update it transactionally with other application data so that you can get the status of a transaction from the status table. + +### Notes about some exceptions + +Although not illustrated in the sample code, the `resume()` API could also throw `TransactionNotFoundException`. This exception indicates that the transaction associated with the specified ID was not found and/or the transaction might have expired. In either case, you can retry the transaction from the beginning since the cause of this exception is basically transient. + +In the sample code, for `UnknownTransactionStatusException`, the transaction is not retried because the application must check if the transaction was successful to avoid potential duplicate operations. For other exceptions, the transaction is retried because the cause of the exception is transient or non-transient. If the cause of the exception is transient, the transaction may succeed if you retry it. However, if the cause of the exception is non-transient, the transaction will still fail even if you retry it. In such a case, you will exhaust the number of retries. + +:::note + +In the sample code, the transaction is retried three times maximum and sleeps for 100 milliseconds before it is retried. But you can choose a retry policy, such as exponential backoff, according to your application requirements. + +::: + +## Group commit for the Coordinator table + +The Coordinator table that is used for Consensus Commit transactions is a vital data store, and using robust storage for it is recommended. However, utilizing more robust storage options, such as internally leveraging multi-AZ or multi-region replication, may lead to increased latency when writing records to the storage, resulting in poor throughput performance. + +ScalarDB provides a group commit feature for the Coordinator table that groups multiple record writes into a single write operation, improving write throughput. In this case, latency may increase or decrease, depending on the underlying database and the workload. + +To enable the group commit feature, add the following configuration: + +```properties +# By default, this configuration is set to `false`. +scalar.db.consensus_commit.coordinator.group_commit.enabled=true + +# These properties are for tuning the performance of the group commit feature. +# scalar.db.consensus_commit.coordinator.group_commit.group_size_fix_timeout_millis=40 +# scalar.db.consensus_commit.coordinator.group_commit.delayed_slot_move_timeout_millis=800 +# scalar.db.consensus_commit.coordinator.group_commit.old_group_abort_timeout_millis=30000 +# scalar.db.consensus_commit.coordinator.group_commit.timeout_check_interval_millis=10 +# scalar.db.consensus_commit.coordinator.group_commit.metrics_monitor_log_enabled=true +``` + +### Limitations + +This section describes the limitations of the group commit feature. + +#### Custom transaction ID passed by users + +The group commit feature implicitly generates an internal value and uses it as a part of transaction ID. Therefore, a custom transaction ID manually passed by users via `com.scalar.db.transaction.consensuscommit.ConsensusCommitManager.begin(String txId)` or `com.scalar.db.transaction.consensuscommit.TwoPhaseConsensusCommitManager.begin(String txId)` can't be used as is for later API calls. You need to use a transaction ID returned from`com.scalar.db.transaction.consensuscommit.ConsensusCommit.getId()` or `com.scalar.db.transaction.consensuscommit.TwoPhaseConsensusCommit.getId()` instead. + +```java + // This custom transaction ID needs to be used for ScalarDB transactions. + String myTxId = UUID.randomUUID().toString(); + + ... + + DistributedTransaction transaction = manager.begin(myTxId); + + ... + + // When the group commit feature is enabled, a custom transaction ID passed by users can't be used as is. + // logger.info("The transaction state: {}", manager.getState(myTxId)); + logger.info("The transaction state: {}", manager.getState(transaction.getId())); +``` + +#### Prohibition of use with a two-phase commit interface + +The group commit feature manages all ongoing transactions in memory. If this feature is enabled with a two-phase commit interface, the information must be solely maintained by the coordinator service to prevent conflicts caused by participant services' inconsistent writes to the Coordinator table, which may contain different transaction distributions over groups. + +This limitation introduces some complexities and inflexibilities related to application development. Therefore, combining the use of the group commit feature with a two-phase commit interface is currently prohibited. + +##### Enabling the feature on existing applications is not supported + +The group commit feature uses a new column in the Coordinator table. The current [Schema Loader](schema-loader.mdx), as of ScalarDB 3, doesn't support table schema migration for the Coordinator table. + +Therefore, enabling the group commit feature on existing applications where any transactions have been executed is not supported. To use this feature, you'll need to start your application in a clean state. + +Coordinator table schema migration in [Schema Loader](schema-loader.mdx) is expected to be supported in ScalarDB 4.0. + +## Investigating Consensus Commit transaction manager errors + +To investigate errors when using the Consensus Commit transaction manager, you can enable a configuration that will return table metadata augmented with transaction metadata columns, which can be helpful when investigating transaction-related issues. This configuration, which is only available when troubleshooting the Consensus Commit transaction manager, enables you to see transaction metadata column details for a given table by using the `DistributedTransactionAdmin.getTableMetadata()` method. + +By adding the following configuration, `Get` and `Scan` operations results will contain [transaction metadata](schema-loader.mdx#internal-metadata-for-consensus-commit): + +```properties +# By default, this configuration is set to `false`. +scalar.db.consensus_commit.include_metadata.enabled=true +``` diff --git a/versioned_docs/version-3.X/backup-restore.mdx b/versioned_docs/version-3.X/backup-restore.mdx new file mode 100644 index 00000000..0efff032 --- /dev/null +++ b/versioned_docs/version-3.X/backup-restore.mdx @@ -0,0 +1,184 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# How to Back Up and Restore Databases Used Through ScalarDB + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +Since ScalarDB provides transaction capabilities on top of non-transactional or transactional databases non-invasively, you need to take special care to back up and restore the databases in a transactionally consistent way. + +This guide describes how to back up and restore the databases that ScalarDB supports. + +## Create a backup + +How you create a backup depends on which database you're using and whether or not you're using multiple databases. The following decision tree shows which approach you should take. + +```mermaid +flowchart TD + A[Are you using a single database with ScalarDB?] + A -->|Yes| B[Does the database have transaction support?] + B -->|Yes| C[Perform back up without explicit pausing] + B ---->|No| D[Perform back up with explicit pausing] + A ---->|No| D +``` + +### Back up without explicit pausing + +If you're using ScalarDB with a single database with support for transactions, you can create a backup of the database even while ScalarDB continues to accept transactions. + +:::warning + +Before creating a backup, you should consider the safest way to create a transactionally consistent backup of your databases and understand any risks that are associated with the backup process. + +::: + +One requirement for creating a backup in ScalarDB is that backups for all the ScalarDB-managed tables (including the Coordinator table) need to be transactionally consistent or automatically recoverable to a transactionally consistent state. That means that you need to create a consistent backup by dumping all tables in a single transaction. + +How you create a transactionally consistent backup depends on the type of database that you're using. Select a database to see how to create a transactionally consistent backup for ScalarDB. + +:::note + +The backup methods by database listed below are just examples of some of the databases that ScalarDB supports. + +::: + + + + You can restore to any point within the backup retention period by using the automated backup feature. + + + Use the `mysqldump` command with the `--single-transaction` option. + + + Use the `pg_dump` command. + + + Use the `.backup` command with the `.timeout` command as specified in [Special commands to sqlite3 (dot-commands)](https://www.sqlite.org/cli.html#special_commands_to_sqlite3_dot_commands_) + + For an example, see [BASH: SQLite3 .backup command](https://stackoverflow.com/questions/23164445/bash-sqlite3-backup-command). + + + Clusters are backed up automatically based on the backup policy, and these backups are retained for a specific duration. You can also perform on-demand backups. For details on performing backups, see [YugabyteDB Managed: Back up and restore clusters](https://docs.yugabyte.com/preview/yugabyte-cloud/cloud-clusters/backup-clusters/). + + + Use the `backup` command. For details, on performing backups, see [Db2: Backup overview](https://www.ibm.com/docs/en/db2/12.1.0?topic=recovery-backup). + + + +### Back up with explicit pausing + +Another way to create a transactionally consistent backup is to create a backup while a cluster of ScalarDB instances does not have any outstanding transactions. Creating the backup depends on the following: + +- If the underlying database has a point-in-time snapshot or backup feature, you can create a backup during the period when no outstanding transactions exist. +- If the underlying database has a point-in-time restore or recovery (PITR) feature, you can set a restore point to a time (preferably the mid-time) in the pause duration period when no outstanding transactions exist. + +:::note + +When using a PITR feature, you should minimize the clock drifts between clients and servers by using clock synchronization, such as NTP. Otherwise, the time you get as the paused duration might be too different from the time in which the pause was actually conducted, which could restore the backup to a point where ongoing transactions exist. + +In addition, you should pause for a sufficient amount of time (for example, five seconds) and use the mid-time of the paused duration as a restore point since clock synchronization cannot perfectly synchronize clocks between nodes. + +::: + +To make ScalarDB drain outstanding requests and stop accepting new requests so that a pause duration can be created, you should implement the [Scalar Admin](https://github.com/scalar-labs/scalar-admin) interface properly in your application that uses ScalarDB or use [ScalarDB Cluster](scalardb-cluster/index.mdx), which implements the Scalar Admin interface. + +By using the [Scalar Admin client tool](https://github.com/scalar-labs/scalar-admin/blob/main/README.md#scalar-admin-client-tool), you can pause nodes, servers, or applications that implement the Scalar Admin interface without losing ongoing transactions. + +How you create a transactionally consistent backup depends on the type of database that you're using. Select a database to see how to create a transactionally consistent backup for ScalarDB. + +:::note + +The backup methods by database listed below are just examples of some of the databases that ScalarDB supports. + +::: + + + + You must enable the PITR feature for DynamoDB tables. If you're using [ScalarDB Schema Loader](schema-loader.mdx) to create schemas, the tool enables the PITR feature for tables by default. + + To specify a transactionally consistent restore point, pause your application that is using ScalarDB with DynamoDB as described in [Back up with explicit pausing](#back-up-with-explicit-pausing). + + + You must create a Cosmos DB for NoSQL account with a continuous backup policy that has the PITR feature enabled. After enabling the feature, backups are created continuously. + + To specify a transactionally consistent restore point, pause your application that is using ScalarDB with Cosmos DB for NoSQL as described in [Back up with explicit pausing](#back-up-with-explicit-pausing). + + + Cassandra has a built-in replication feature, so you do not always have to create a transactionally consistent backup. For example, if the replication factor is set to `3` and only the data of one of the nodes in a Cassandra cluster is lost, you won't need a transactionally consistent backup (snapshot) because the node can be recovered by using a normal, transactionally inconsistent backup (snapshot) and the repair feature. + + However, if the quorum of cluster nodes loses their data, you will need a transactionally consistent backup (snapshot) to restore the cluster to a certain transactionally consistent point. + + To create a transactionally consistent cluster-wide backup (snapshot), pause the application that is using ScalarDB or [ScalarDB Cluster](scalardb-cluster/index.mdx) and create backups (snapshots) of the nodes as described in [Back up with explicit pausing](#back-up-with-explicit-pausing) or stop the Cassandra cluster, take copies of all the data in the nodes, and start the cluster. + + + You can perform on-demand backups or scheduled backups during a paused duration. For details on performing backups, see [YugabyteDB Managed: Back up and restore clusters](https://docs.yugabyte.com/preview/yugabyte-cloud/cloud-clusters/backup-clusters/). + + + +## Restore a backup + +How you restore a transactionally consistent backup depends on the type of database that you're using. Select a database to see how to create a transactionally consistent backup for ScalarDB. + +:::note + +The restore methods by database listed below are just examples of some of the databases that ScalarDB supports. + +::: + + + + You can restore to any point within the backup retention period by using the automated backup feature. + + + First, stop all the nodes of the Cassandra cluster. Then, clean the `data`, `commitlog`, and `hints` directories, and place the backups (snapshots) in each node. + + After placing the backups (snapshots) in each node, start all the nodes of the Cassandra Cluster. + + + Follow the official Azure documentation for [restore an account by using Azure portal](https://docs.microsoft.com/en-us/azure/cosmos-db/restore-account-continuous-backup#restore-account-portal). After restoring a backup, [configure the default consistency level](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-manage-consistency#configure-the-default-consistency-level) of the restored databases to `STRONG`. In addition, you should use the mid-time of the paused duration as the restore point as previously explained. + + ScalarDB implements the Cosmos DB adapter by using its stored procedures, which are installed when creating schemas by using ScalarDB Schema Loader. However, the PITR feature of Cosmos DB doesn't restore stored procedures. Because of this, you need to re-install the required stored procedures for all tables after restoration. You can do this by using ScalarDB Schema Loader with the `--repair-all` option. For details, see [Repair tables](schema-loader.mdx#repair-tables). + + + Follow the official AWS documentation for [restoring a DynamoDB table to a point in time](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/PointInTimeRecovery.Tutorial.html), but keep in mind that a table can only be restored with an alias. Because of this, you will need to restore the table with an alias, delete the original table, and rename the alias to the original name to restore the tables with the same name. + + To do this procedure: + + 1. Create a backup. + 1. Select the mid-time of the paused duration as the restore point. + 2. Restore by using the PITR of table A to table B. + 3. Create a backup of the restored table B (assuming that the backup is named backup B). + 4. Remove table B. + 2. Restore the backup. + 1. Remove table A. + 2. Create a table named A by using backup B. + +:::note + +* You must do the steps mentioned above for each table because tables can only be restored one at a time. +* Configurations such as PITR and auto-scaling policies are reset to the default values for restored tables, so you must manually configure the required settings. For details, see the official AWS documentation for [How to restore DynamoDB tables with DynamoDB](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/CreateBackup.html#CreateBackup_HowItWorks-restore). + +::: + + + If you used `mysqldump` to create the backup file, use the `mysql` command to restore the backup as specified in [Reloading SQL-Format Backups](https://dev.mysql.com/doc/mysql-backup-excerpt/8.0/en/reloading-sql-format-dumps.html). + + + If you used `pg_dump` to create the backup file, use the `psql` command to restore the backup as specified in [Restoring the Dump](https://www.postgresql.org/docs/current/backup-dump.html#BACKUP-DUMP-RESTORE). + + + Use the `.restore` command as specified in [Special commands to sqlite3 (dot-commands)](https://www.sqlite.org/cli.html#special_commands_to_sqlite3_dot_commands_). + + + You can restore from the scheduled or on-demand backup within the backup retention period. For details on performing backups, see [YugabyteDB Managed: Back up and restore clusters](https://docs.yugabyte.com/preview/yugabyte-cloud/cloud-clusters/backup-clusters/). + + + Use the `restore` command. For details, on restoring the database, see [Db2: Restore overview](https://www.ibm.com/docs/en/db2/12.1.0?topic=recovery-restore). + + diff --git a/versioned_docs/version-3.X/configurations.mdx b/versioned_docs/version-3.X/configurations.mdx new file mode 100644 index 00000000..3e36d53f --- /dev/null +++ b/versioned_docs/version-3.X/configurations.mdx @@ -0,0 +1,243 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# ScalarDB Core Configurations + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +This page describes the available configurations for ScalarDB Core. + +:::tip + +If you are using ScalarDB Cluster, please refer to [ScalarDB Cluster Configurations](./scalardb-cluster/scalardb-cluster-configurations.mdx) instead. + +::: + +## General configurations + +The following configurations are available for the Consensus Commit transaction manager: + +| Name | Description | Default | +|-------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------| +| `scalar.db.transaction_manager` | Transaction manager of ScalarDB. Specify `consensus-commit` to use [Consensus Commit](./consensus-commit.mdx) or `single-crud-operation` to [run non-transactional storage operations](./run-non-transactional-storage-operations-through-library.mdx). Note that the configurations under the `scalar.db.consensus_commit` prefix are ignored if you use `single-crud-operation`. | `consensus-commit` | +| `scalar.db.consensus_commit.isolation_level` | Isolation level used for Consensus Commit. Either `SNAPSHOT`, `SERIALIZABLE`, or `READ_COMMITTED` can be specified. | `SNAPSHOT` | +| `scalar.db.consensus_commit.coordinator.namespace` | Namespace name of Coordinator tables. | `coordinator` | +| `scalar.db.consensus_commit.include_metadata.enabled` | If set to `true`, `Get` and `Scan` operations results will contain transaction metadata. To see the transaction metadata columns details for a given table, you can use the `DistributedTransactionAdmin.getTableMetadata()` method, which will return the table metadata augmented with the transaction metadata columns. Using this configuration can be useful to investigate transaction-related issues. | `false` | + +## Performance-related configurations + +The following performance-related configurations are available for the Consensus Commit transaction manager: + +| Name | Description | Default | +|----------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------| +| `scalar.db.consensus_commit.parallel_executor_count` | Number of executors (threads) for parallel execution. This number refers to the total number of threads across transactions in a ScalarDB Cluster node or a ScalarDB process. | `128` | +| `scalar.db.consensus_commit.parallel_preparation.enabled` | Whether or not the preparation phase is executed in parallel. | `true` | +| `scalar.db.consensus_commit.parallel_validation.enabled` | Whether or not the validation phase (in `EXTRA_READ`) is executed in parallel. | The value of `scalar.db.consensus_commit.parallel_commit.enabled` | +| `scalar.db.consensus_commit.parallel_commit.enabled` | Whether or not the commit phase is executed in parallel. | `true` | +| `scalar.db.consensus_commit.parallel_rollback.enabled` | Whether or not the rollback phase is executed in parallel. | The value of `scalar.db.consensus_commit.parallel_commit.enabled` | +| `scalar.db.consensus_commit.async_commit.enabled` | Whether or not the commit phase is executed asynchronously. | `false` | +| `scalar.db.consensus_commit.async_rollback.enabled` | Whether or not the rollback phase is executed asynchronously. | The value of `scalar.db.consensus_commit.async_commit.enabled` | +| `scalar.db.consensus_commit.parallel_implicit_pre_read.enabled` | Whether or not implicit pre-read is executed in parallel. | `true` | +| `scalar.db.consensus_commit.one_phase_commit.enabled` | Whether or not the one-phase commit optimization is enabled. | `false` | +| `scalar.db.consensus_commit.coordinator.write_omission_on_read_only.enabled` | Whether or not the write omission optimization is enabled for read-only transactions. This optimization is useful for read-only transactions that do not modify any data, as it avoids unnecessary writes to the Coordinator tables. | `true` | +| `scalar.db.consensus_commit.coordinator.group_commit.enabled` | Whether or not committing the transaction state is executed in batch mode. This feature can't be used with a two-phase commit interface. | `false` | +| `scalar.db.consensus_commit.coordinator.group_commit.slot_capacity` | Maximum number of slots in a group for the group commit feature. A large value improves the efficiency of group commit, but may also increase latency and the likelihood of transaction conflicts.[^1] | `20` | +| `scalar.db.consensus_commit.coordinator.group_commit.group_size_fix_timeout_millis` | Timeout to fix the size of slots in a group. A large value improves the efficiency of group commit, but may also increase latency and the likelihood of transaction conflicts.[^1] | `40` | +| `scalar.db.consensus_commit.coordinator.group_commit.delayed_slot_move_timeout_millis` | Timeout to move delayed slots from a group to another isolated group to prevent the original group from being affected by delayed transactions. A large value improves the efficiency of group commit, but may also increase the latency and the likelihood of transaction conflicts.[^1] | `1200` | +| `scalar.db.consensus_commit.coordinator.group_commit.old_group_abort_timeout_millis` | Timeout to abort an old ongoing group. A small value reduces resource consumption through aggressive aborts, but may also increase the likelihood of unnecessary aborts for long-running transactions. | `60000` | +| `scalar.db.consensus_commit.coordinator.group_commit.timeout_check_interval_millis` | Interval for checking the group commit–related timeouts. | `20` | +| `scalar.db.consensus_commit.coordinator.group_commit.metrics_monitor_log_enabled` | Whether or not the metrics of the group commit are logged periodically. | `false` | + +## Storage-related configurations + +ScalarDB has a storage (database) abstraction layer that supports multiple storage implementations. You can specify the storage implementation by using the `scalar.db.storage` property. + +Select a database to see the configurations available for each storage. + + + + The following configurations are available for JDBC databases: + + | Name | Description | Default | + |------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------| + | `scalar.db.storage` | `jdbc` must be specified. | - | + | `scalar.db.contact_points` | JDBC connection URL. | | + | `scalar.db.username` | Username to access the database. | | + | `scalar.db.password` | Password to access the database. | | + | `scalar.db.jdbc.connection_pool.min_idle` | Minimum number of idle connections in the connection pool. | `20` | + | `scalar.db.jdbc.connection_pool.max_idle` | Maximum number of connections that can remain idle in the connection pool. | `50` | + | `scalar.db.jdbc.connection_pool.max_total` | Maximum total number of idle and borrowed connections that can be active at the same time for the connection pool. Use a negative value for no limit. | `100` | + | `scalar.db.jdbc.prepared_statements_pool.enabled` | Setting this property to `true` enables prepared-statement pooling. | `false` | + | `scalar.db.jdbc.prepared_statements_pool.max_open` | Maximum number of open statements that can be allocated from the statement pool at the same time. Use a negative value for no limit. | `-1` | + | `scalar.db.jdbc.isolation_level` | Isolation level for JDBC. `READ_UNCOMMITTED`, `READ_COMMITTED`, `REPEATABLE_READ`, or `SERIALIZABLE` can be specified. | Underlying-database specific | + | `scalar.db.jdbc.table_metadata.schema` | Schema name for the table metadata used for ScalarDB. | `scalardb` | + | `scalar.db.jdbc.table_metadata.connection_pool.min_idle` | Minimum number of idle connections in the connection pool for the table metadata. | `5` | + | `scalar.db.jdbc.table_metadata.connection_pool.max_idle` | Maximum number of connections that can remain idle in the connection pool for the table metadata. | `10` | + | `scalar.db.jdbc.table_metadata.connection_pool.max_total` | Maximum total number of idle and borrowed connections that can be active at the same time for the connection pool for the table metadata. Use a negative value for no limit. | `25` | + | `scalar.db.jdbc.admin.connection_pool.min_idle` | Minimum number of idle connections in the connection pool for admin. | `5` | + | `scalar.db.jdbc.admin.connection_pool.max_idle` | Maximum number of connections that can remain idle in the connection pool for admin. | `10` | + | `scalar.db.jdbc.admin.connection_pool.max_total` | Maximum total number of idle and borrowed connections that can be active at the same time for the connection pool for admin. Use a negative value for no limit. | `25` | + | `scalar.db.jdbc.mysql.variable_key_column_size` | Column size for TEXT and BLOB columns in MySQL when they are used as a primary key or secondary key. Minimum 64 bytes. | `128` | + | `scalar.db.jdbc.oracle.variable_key_column_size` | Column size for TEXT and BLOB columns in Oracle when they are used as a primary key or secondary key. Minimum 64 bytes. | `128` | + | `scalar.db.jdbc.oracle.time_column.default_date_component` | Value of the date component used for storing `TIME` data in Oracle. Since Oracle has no data type to only store a time without a date component, ScalarDB stores `TIME` data with the same date component value for ease of comparison and sorting. | `1970-01-01` | + | `scalar.db.jdbc.db2.variable_key_column_size` | Column size for TEXT and BLOB columns in IBM Db2 when they are used as a primary key or secondary key. Minimum 64 bytes. | `128` | + | `scalar.db.jdbc.db2.time_column.default_date_component` | Value of the date component used for storing `TIME` data in IBM Db2. Since the IBM Db2 TIMESTAMP type is used to store ScalarDB `TIME` type data because it provides fractional-second precision, ScalarDB stores `TIME` data with the same date component value for ease of comparison and sorting. | `1970-01-01` | + +:::note + +**SQLite3** + +If you're using SQLite3 as a JDBC database, you must set `scalar.db.contact_points` as follows: + +```properties +scalar.db.contact_points=jdbc:sqlite:?busy_timeout=10000 +``` + +Unlike other JDBC databases, [SQLite3 doesn't fully support concurrent access](https://www.sqlite.org/lang_transaction.html). To avoid frequent errors caused internally by [`SQLITE_BUSY`](https://www.sqlite.org/rescode.html#busy), setting a [`busy_timeout`](https://www.sqlite.org/c3ref/busy_timeout.html) parameter is recommended. + +**YugabyteDB** + +If you're using YugabyteDB as a JDBC database, you can specify multiple endpoints in `scalar.db.contact_points` as follows: + +```properties +scalar.db.contact_points=jdbc:yugabytedb://127.0.0.1:5433\\,127.0.0.2:5433\\,127.0.0.3:5433/?load-balance=true +``` + +Multiple endpoints should be separated by escaped commas. + +For information on YugabyteDB's smart driver and load balancing, see [YugabyteDB smart drivers for YSQL](https://docs.yugabyte.com/preview/drivers-orms/smart-drivers/). + +::: + + + + The following configurations are available for DynamoDB: + + | Name | Description | Default | + |---------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------| + | `scalar.db.storage` | `dynamo` must be specified. | - | + | `scalar.db.contact_points` | AWS region with which ScalarDB should communicate (e.g., `us-east-1`). | | + | `scalar.db.username` | AWS access key used to identify the user interacting with AWS. | | + | `scalar.db.password` | AWS secret access key used to authenticate the user interacting with AWS. | | + | `scalar.db.dynamo.endpoint_override` | Amazon DynamoDB endpoint with which ScalarDB should communicate. This is primarily used for testing with a local instance instead of an AWS service. | | + | `scalar.db.dynamo.table_metadata.namespace` | Namespace name for the table metadata used for ScalarDB. | `scalardb` | + | `scalar.db.dynamo.namespace.prefix` | Prefix for the user namespaces and metadata namespace names. Since AWS requires having unique tables names in a single AWS region, this is useful if you want to use multiple ScalarDB environments (development, production, etc.) in a single AWS region. | | + + + The following configurations are available for CosmosDB for NoSQL: + + | Name | Description | Default | + |--------------------------------------------|----------------------------------------------------------------------------------------------------------|------------| + | `scalar.db.storage` | `cosmos` must be specified. | - | + | `scalar.db.contact_points` | Azure Cosmos DB for NoSQL endpoint with which ScalarDB should communicate. | | + | `scalar.db.password` | Either a master or read-only key used to perform authentication for accessing Azure Cosmos DB for NoSQL. | | + | `scalar.db.cosmos.table_metadata.database` | Database name for the table metadata used for ScalarDB. | `scalardb` | + | `scalar.db.cosmos.consistency_level` | Consistency level used for Cosmos DB operations. `STRONG` or `BOUNDED_STALENESS` can be specified. | `STRONG` | + + + The following configurations are available for Cassandra: + + | Name | Description | Default | + |-----------------------------------------|-----------------------------------------------------------------------|------------| + | `scalar.db.storage` | `cassandra` must be specified. | - | + | `scalar.db.contact_points` | Comma-separated contact points. | | + | `scalar.db.contact_port` | Port number for all the contact points. | | + | `scalar.db.username` | Username to access the database. | | + | `scalar.db.password` | Password to access the database. | | + + + +##### Multi-storage support + +ScalarDB supports using multiple storage implementations simultaneously. You can use multiple storages by specifying `multi-storage` as the value for the `scalar.db.storage` property. + +For details about using multiple storages, see [Multi-Storage Transactions](multi-storage-transactions.mdx). + +##### Cross-partition scan configurations + +By enabling the cross-partition scan option as described below, the `Scan` operation can retrieve all records across partitions. In addition, you can specify arbitrary conditions and orderings in the cross-partition `Scan` operation by enabling `cross_partition_scan.filtering` and `cross_partition_scan.ordering`, respectively. Currently, the cross-partition scan with ordering option is available only for JDBC databases. To enable filtering and ordering, `scalar.db.cross_partition_scan.enabled` must be set to `true`. + +For details on how to use cross-partition scan, see [Scan operation](./api-guide.mdx#scan-operation). + +:::warning + +For non-JDBC databases, transactions could be executed at read-committed snapshot isolation (`SNAPSHOT`), which is a lower isolation level, even if you enable cross-partition scan with the `SERIALIZABLE` isolation level. When using non-JDBC databases, use cross-partition scan only if consistency does not matter for your transactions. + +::: + +| Name | Description | Default | +|----------------------------------------------------|-----------------------------------------------|---------| +| `scalar.db.cross_partition_scan.enabled` | Enable cross-partition scan. | `true` | +| `scalar.db.cross_partition_scan.filtering.enabled` | Enable filtering in cross-partition scan. | `false` | +| `scalar.db.cross_partition_scan.ordering.enabled` | Enable ordering in cross-partition scan. | `false` | + +##### Scan fetch size + +You can configure the fetch size for storage scan operations by using the following property: + +| Name | Description | Default | +|-----------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------| +| `scalar.db.scan_fetch_size` | Specifies the number of records to fetch in a single batch during a storage scan operation. A larger value can improve performance for a large result set by reducing round trips to the storage, but it also increases memory usage. A smaller value uses less memory but may increase latency. | `10` | + +## Other ScalarDB configurations + +The following are additional configurations available for ScalarDB: + +| Name | Description | Default | +|------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------| +| `scalar.db.metadata.cache_expiration_time_secs` | ScalarDB has a metadata cache to reduce the number of requests to the database. This setting specifies the expiration time of the cache in seconds. If you specify `-1`, the cache will never expire. | `60` | +| `scalar.db.active_transaction_management.expiration_time_millis` | ScalarDB maintains in-progress transactions, which can be resumed by using a transaction ID. This process expires transactions that have been idle for an extended period to prevent resource leaks. This setting specifies the expiration time of this transaction management feature in milliseconds. | `-1` (no expiration) | +| `scalar.db.default_namespace_name` | The given namespace name will be used by operations that do not already specify a namespace. | | + +## Placeholder usage + +You can use placeholders in the values, and they are replaced with environment variables (`${env:}`) or system properties (`${sys:}`). You can also specify default values in placeholders like `${sys::-}`. + +The following is an example of a configuration that uses placeholders: + +```properties +scalar.db.username=${env:SCALAR_DB_USERNAME:-admin} +scalar.db.password=${env:SCALAR_DB_PASSWORD} +``` + +In this example configuration, ScalarDB reads the username and password from environment variables. If the environment variable `SCALAR_DB_USERNAME` does not exist, ScalarDB uses the default value `admin`. + +## Configuration example - App and database + +```mermaid +flowchart LR + app["App
(ScalarDB library with
Consensus Commit)"] + db[(Underlying storage or database)] + app --> db +``` + +In this example configuration, the app (ScalarDB library with Consensus Commit) connects to an underlying storage or database (in this case, Cassandra) directly. + +:::warning + +This configuration exists only for development purposes and isn't suitable for a production environment. This is because the app needs to implement the [Scalar Admin](https://github.com/scalar-labs/scalar-admin) interface to take transactionally consistent backups for ScalarDB, which requires additional configurations. + +::: + +The following is an example of the configuration for connecting the app to the underlying database through ScalarDB: + +```properties +# Transaction manager implementation. +scalar.db.transaction_manager=consensus-commit + +# Storage implementation. +scalar.db.storage=cassandra + +# Comma-separated contact points. +scalar.db.contact_points= + +# Credential information to access the database. +scalar.db.username= +scalar.db.password= +``` diff --git a/versioned_docs/version-3.X/consensus-commit.mdx b/versioned_docs/version-3.X/consensus-commit.mdx new file mode 100644 index 00000000..16df6dc7 --- /dev/null +++ b/versioned_docs/version-3.X/consensus-commit.mdx @@ -0,0 +1,246 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Consensus Commit Protocol + +import JavadocLink from '/src/theme/JavadocLink.js'; + +Consensus Commit is the transaction protocol used in ScalarDB and is designed for executing transactions spanning multiple diverse databases. Its uniqueness is that the protocol achieves ACID transactions without relying on the transaction capabilities of the underlying databases, unlike X/Open XA-based solutions. This document explains the details of the protocol, including how it works, the guaranteed isolation levels, the interfaces, the performance optimization that it employs, and its limitations. + +## The protocol + +This section explains how the Consensus Commit protocol works. The Consensus Commit protocol uses a concurrency control protocol to guarantee isolation and an atomic commitment protocol to guarantee atomicity and durability. + +### Concurrency control protocol + +The Consensus Commit protocol employs optimistic concurrency control (OCC) as its concurrency control protocol. OCC operates under the assumption that conflicts are rare, allowing transactions to proceed without the need for locks and resolving conflicts only when they actually occur. Therefore, OCC performs great in low-contention environments. It is also particularly beneficial in distributed environments, where managing locks is tricky. + +:::note + +Pessimistic concurrency control (PCC), on the other hand, assumes conflicts are common and takes locks on resources when they are used to avoid interference. Therefore, PCC performs great in high-contention environments. + +::: + +The OCC protocol of ScalarDB has three phases, as the commonly used OCC protocols, each of which does the following: + +* Read phase: + * ScalarDB tracks the read and write sets of transactions. ScalarDB copies every record that a transaction accesses from databases to its local workspace and stores its writes in the local workspace. +* Validation phase: + * ScalarDB checks if the committing transaction conflicts with other transactions. ScalarDB uses backward validation; it goes to the write phase only if other transactions have not written what the transaction reads and writes, which are called read validation and write validation, respectively. +* Write phase: + * ScalarDB propagates the changes in the transaction's write set to the database and makes them visible to other transactions. + +As described next, ScalarDB provides an isolation mode (isolation level) where it skips the read validation in the validation phase to allow for more performance for some workloads that don't require the read validation for correctness. + +:::note + +The OCC of ScalarDB without the read validation works similarly to snapshot isolation. However, it works with a single version and causes read-skew anomalies because it does not create global snapshots. + +::: + +### Atomic commitment protocol + +The Consensus Commit protocol employs a variant of the two-phase commit protocol as an atomic commitment protocol (ACP). The ACP of ScalarDB comprises two phases, each of which has two sub-phases, and briefly works as follows: + +* Prepare phase (prepare-records phase \+ validate-records phase): + * In the prepare-records phase, ScalarDB runs the write validation of the OCC protocol for all the records written by the transaction by updating the statuses of the records to PREPARED and moves on to the next phase if all the records are successfully validated. + * In the validate-records phase, ScalarDB runs the read validation of the OCC protocol for all the records read by the transaction and moves on to the next phase if all the records are successfully validated. +* Commit phase (commit-state phase \+ commit-records phase): + * In the commit-state phase, ScalarDB commits the transaction by writing a COMMITTED state to a special table called a coordinator table. + * In the commit-records phase, ScalarDB runs the write phase of the OCC protocol for all the records written by the transaction by updating the statuses of the records to COMMITTED. + +:::note + +In case of deleting records, the statuses of the records are first changed to DELETED in the prepare phase and later physically deleted in the commit phase. + +::: + +#### How it works in more detail + +Let's see how the protocol works in each phase in more detail. + +##### Before the prepare phase + +First, a transaction begins when a client accesses ScalarDB (or a ScalarDB Cluster node) and issues a `begin` command. When a transaction begins, ScalarDB acts as a transaction coordinator, accessing the underlying databases, and first generates a transaction ID (TxID) with UUID version 4. Then, when the client is ready to commit the transaction after performing operations such as reading and writing records, it calls a `commit` command to request ScalarDB to commit the transaction and enters the prepare phase. As described previously, ScalarDB holds the read set (readSet) and write set (writeSet) of the transaction in its local workspace at the time of committing. + +##### Prepare phase + +ScalarDB first prepares the records of the write set by propagating the records, including transaction logs like TxID as described later, with PREPARED states to the underlying databases as the prepare-records phase. Here, we assume a write set maintains updated records composed of the original records and updated columns. If any preparation fails, it aborts the transaction by writing an ABORTED state record to a Coordinator table, where all the transactions’ final states are determined and managed. We explain the Coordinator table in more detail later in this section. + +:::note + +ScalarDB checks conflicting preparations by using linearizable conditional writes. A transaction updates a record if the record has not been updated by another transaction since the transaction read it by checking if the TxID of the record has not been changed. + +::: + +ScalarDB then moves on to the validate-records phase as necessary. The validate-records phase is only necessary if the isolation level is set to SERIALIZABLE. In this phase, ScalarDB re-reads all the records in the read set to see if other transactions have written the records that the transaction has read before. If the read set has not been changed, the transaction can go to the commit-state phase since there are no anti-dependencies; otherwise, it aborts the transaction. + +##### Commit phase + +If all the validations in the prepare phase are done successfully, ScalarDB commits the transaction by writing a COMMITTED state record to the Coordinator table as the commit-state phase. + +:::note + +* ScalarDB uses linearizable conditional writes to coordinate concurrent writes to the Coordinator table, creating a state record with a TxID if there is no record for the TxID. Once the COMMITTED state is correctly written to the Coordinator table, the transaction is regarded as committed. +* By default, if a transaction contains only read operations, ScalarDB skips the commit-state phase. However, you can configure ScalarDB to write a COMMITTED state record to the Coordinator table even for read-only transactions by setting the following parameter to `false`: + * `scalar.db.consensus_commit.coordinator.write_omission_on_read_only.enabled` + +::: + +Then, ScalarDB commits all the validated (prepared) records by changing the states of the records to COMMITTED as the commit-records phase. + +#### Distributed WAL + +ScalarDB stores transaction logs, which are for write-ahead logging (WAL), in the underlying database records that it manages. Specifically, as shown in the following figure, ScalarDB manages special columns for the log information in a record in addition to the columns that an application manages. The log information comprises, for example, a transaction ID (TxID) that has updated the corresponding record most recently, a record version number (Version), a record state (TxState) (for example, COMMITTED or PREPARED), timestamps (not shown in the diagram), and a before image that comprises the previous version's application data and its metadata. + +ScalarDB also manages transaction states separately from the application records in the Coordinator table. The Coordinator table determines and manages transaction states as a single source of truth. The Coordinator table can be collocated with application-managed tables or located in a separate dedicated database. + +![Distributed WAL](images/scalardb-metadata.png) + +:::note + +The Coordinator table can be replicated for high availability by using the replication and consensus capabilities of the underlying databases. For example, if you manage the Coordinator table by using Cassandra with a replication factor of three, you can make the transaction coordination of ScalarDB tolerate one replica crash. Hence, you can make the atomic commitment protocol of ScalarDB perform like the Paxos Commit protocol; it could mitigate liveness issues (for example, blocking problems) without sacrificing safety. + +::: + +#### Lazy recovery + +Transactions can crash at any time and could leave records in an uncommitted state. ScalarDB recovers uncommitted records lazily when it reads them, depending on the transaction states of the Coordinator table. Specifically, if a record is in the PREPARED state, but the transaction that updated the record has expired or been aborted, the record will be rolled back. If a record is in the PREPARED state and the transaction that updated the record is committed, the record will be rolled forward. + +A transaction expires after a certain amount of time (currently 15 seconds). When ScalarDB observes a record that has been prepared by an expired transaction, ScalarDB writes the ABORTED state for the transaction to the Coordinator table (with retries). If ScalarDB successfully writes the ABORTED state to the Coordinator table, the transaction is aborted. Otherwise, the transaction will be committed by the original process that is slow but still alive for some reason, or it will remain in the UNKNOWN state until it is either aborted or committed. + +## Isolation levels + +The Consensus Commit protocol supports three isolation levels: read-committed snapshot isolation (a weaker variant of snapshot isolation), serializable, and read-committed, each of which has the following characteristics: + +* Read-committed snapshot isolation (SNAPSHOT - default) + * Possible anomalies: read skew, write skew, read only + * Faster than serializable, but guarantees weaker correctness. +* Serializable (SERIALIZABLE) + * Possible anomalies: None + * Slower than read-committed snapshot isolation, but guarantees stronger (strongest) correctness. +* Read-committed (READ_COMMITTED) + * Possible anomalies: read skew, write skew, read only + * Faster than read-committed snapshot isolation because it could return non-latest committed records. + +As described above, serializable is preferable from a correctness perspective, but slower than read-committed snapshot isolation. Choose the appropriate one based on your application requirements and workload. For details on how to configure read-committed snapshot isolation, serializable, and read-committed, see [ScalarDB Configuration](configurations.mdx#basic-configurations). + +:::note + +The Consensus Commit protocol of ScalarDB requires each underlying database to provide linearizable operations, as described in [Configurations for the Underlying Databases of ScalarDB](database-configurations.mdx#transactions); thus, it guarantees strict serializability. + +::: + +:::warning + +Scanning records without specifying a partition key (for example, or `SELECT * FROM table`) for non-JDBC databases does not always guarantee serializability, even if `SERIALIZABLE` is specified. Therefore, you should do so at your own discretion and consider updating the schemas if possible. For more details, refer to [Cross-partition scan configurations](configurations.mdx#cross-partition-scan-configurations). + +::: + +## Interfaces + +The Consensus Commit protocol provides two interfaces: [a one-phase commit interface and a two-phase commit interface](scalardb-cluster/run-transactions-through-scalardb-cluster.mdx#run-transactions). + +The one-phase commit interface is a simple interface that provides only a single `commit` method, where all the phases of the atomic commitment protocol are executed in the method. On the other hand, the two-phase commit interface exposes each phase of the protocol with `prepare`, `validate`, and `commit` methods. + +:::note + +The `prepare` method is for the prepare-records phase, and the `validate` method is for the validate-records phase. + +::: + +In most cases, using the one-phase commit interface is recommended since it is easier to use and handle errors. But the two-phase commit interface is useful when running a transaction across multiple applications or services without directly accessing databases from ScalarDB, such as maintaining the consistency of databases in microservices. + +## Performance optimization + +The Consensus Commit protocol employs several performance optimizations. + +### Parallel execution + +Consensus Commit executes each phase of the atomic commitment protocol in parallel, using intra-transaction parallelism without sacrificing correctness. Specifically, it tries to execute the prepare-records phase by writing records with PREPARED status in parallel. Likewise, it uses a similar parallel execution for the validate-records phase, the commit-records phase, and the rollback process. + +You can enable respective parallel execution by using the following parameters: + +* Prepare-records phase + * `scalar.db.consensus_commit.parallel_preparation.enabled` +* Validate-records phase + * `scalar.db.consensus_commit.parallel_validation.enabled` +* Commit-records phase + * `scalar.db.consensus_commit.parallel_commit.enabled` +* Rollback processing + * `scalar.db.consensus_commit.parallel_rollback.enabled` + +You can also configure the execution parallelism by using the following parameter: + +* `scalar.db.consensus_commit.parallel_executor_count` + +For details about the configuration, refer to [Performance-related configurations](configurations.mdx#performance-related-configurations). + +### Asynchronous execution + +Since a transaction is regarded as committed if the commit-state phase completes successfully, it can also return to the client without waiting for the completion of the commit-records phase, executing the phase asynchronously. Likewise, when a transaction fails for some reason and does a rollback, the rollback process can also be executed asynchronously without waiting for its completion. + +You can enable respective asynchronous execution by using the following parameters: + +* Commit-records phase + * `scalar.db.consensus_commit.async_commit.enabled` +* Rollback processing + * `scalar.db.consensus_commit.async_rollback.enabled` + +### One-phase commit + +With one-phase commit optimization, ScalarDB can omit the prepare-records and commit-state phases without sacrificing correctness, provided that the transaction only updates records that the underlying database can atomically update. + +You can enable one-phase commit optimization by using the following parameter: + +* `scalar.db.consensus_commit.one_phase_commit.enabled` + +### Group commit + +Consensus Commit provides a group-commit feature to execute the commit-state phase of multiple transactions in a batch, reducing the number of writes for the commit-state phase. It is especially useful when writing to a Coordinator table is slow, for example, when the Coordinator table is deployed in a multi-region environment for high availability. + +You can enable group commit by using the following parameter: + +* `scalar.db.consensus_commit.coordinator.group_commit.enabled` + +Group commit has several other parameters. For more details, refer to [Performance-related configurations](configurations.mdx#performance-related-configurations). + +## Limitations + +ScalarDB has several limitations in achieving database-agnostic transactions. + +### Applications must access ScalarDB to access the underlying databases + +Since ScalarDB with the Consensus Commit protocol handles transactions in its layer without depending on the transactional capability of the underlying databases, your applications cannot bypass ScalarDB. Bypassing it will cause unexpected behavior, mostly resulting in facing some database anomalies. Even for read operations, accessing the underlying databases of ScalarDB directly will give you inconsistent data with the transaction metadata, so it is not allowed. + +However, for tables that are not managed or touched by ScalarDB transactions, you can read from and write to the tables. For example, it is OK to check tables' metadata information, such as information schema, by directly accessing the tables without going through ScalarDB. Also, there are several other cases where you can access databases directly without going through ScalarDB. The basic criterion is whether or not you update the data of the underlying databases. If you are sure that you do not write to the databases, you can access the databases directly. For example, it is OK to take a backup of databases by using database-native tools. + +:::note + +If you take backups from multiple databases or from non-transactional databases, you need to pause your applications or ScalarDB Cluster. For more details, refer to [How to Back Up and Restore Databases Used Through ScalarDB](backup-restore.mdx). + +::: + +### Executing particular operations in a certain sequence is prohibited for correctness + +In the current implementation, ScalarDB throws an exception in the following cases: + +* Executing scan operations after write (Put, Insert, Update, Upsert, Delete) operations for the same record in a transaction. +* Executing write (Put, Insert, Update, and Upsert) operations after Delete operations for the same record in a transaction. + +## See also + +You can learn more about the Consesnus Commit protocol by seeing the following presentation and YouTube video, which summarizes, visually, how the protocol works: + +- **Speaker Deck presentation:** [ScalarDB: Universal Transaction Manager](https://speakerdeck.com/scalar/scalar-db-universal-transaction-manager) +- **YouTube (Japanese):** [How ScalarDB runs transactions (a part of DBSJ lecture)](https://www.youtube.com/watch?v=s6Q7QQccDTc) + +In addition, more details about the protocol, including the background, the challenges, and the novelty, are discussed in the following research paper and its presentation: + +- **Research paper:** [ScalarDB: Universal Transaction Manager for Polystores](https://www.vldb.org/pvldb/vol16/p3768-yamada.pdf) +- **Speaker Deck presentation:** [ScalarDB: Universal Transaction Manager for Polystores](https://speakerdeck.com/scalar/scalardb-universal-transaction-manager-for-polystores-vldb23) diff --git a/versioned_docs/version-3.X/data-modeling.mdx b/versioned_docs/version-3.X/data-modeling.mdx new file mode 100644 index 00000000..3995a8bb --- /dev/null +++ b/versioned_docs/version-3.X/data-modeling.mdx @@ -0,0 +1,132 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Model Your Data + +Data modeling (or in other words, designing your database schemas) is the process of conceptualizing and visualizing how data will be stored and used by identifying the patterns used to access data and the types of queries to be performed within business operations. + +This page first explains the ScalarDB data model and then describes how to design your database schemas based on the data model. + +## ScalarDB data model + +ScalarDB's data model is an extended key-value model inspired by the Bigtable data model. It is similar to the relational model but differs in several ways, as described below. The data model is chosen to abstract various databases, such as relational databases, NoSQL databases, and NewSQL databases. + +The following diagram shows an example of ScalarDB tables, each of which is a collection of records. This section first explains what objects, such as tables and records, ScalarDB defines and then describes how to locate the records. + +![ScalarDB data model](images/scalardb_data_model.png) + +### Objects in ScalarDB + +The ScalarDB data model has several objects. + +#### Namespace + +A namespace is a collection of tables analogous to an SQL namespace or database. + +#### Table + +A table is a collection of partitions. A namespace most often contains one or more tables, each identified by a name. + +#### Partition + +A partition is a collection of records and a unit of distribution to nodes, whether logical or physical. Therefore, records within the same partition are placed in the same node. ScalarDB assumes multiple partitions are distributed by hashing. + +#### Record / row + +A record or row is a set of columns that is uniquely identifiable among all other records. + +#### Column + +A column is a fundamental data element and does not need to be broken down any further. Each record is composed of one or more columns. Each column has a data type. For details about the data type, refer to [Data-type mapping between ScalarDB and other databases](schema-loader.mdx#data-type-mapping-between-scalardb-and-other-databases). + +#### Secondary index + +A secondary index is a sorted copy of a column in a single base table. Each index entry is linked to a corresponding table partition. ScalarDB currently doesn't support multi-column indexes, so it can create indexes with only one column. + +### How to locate records + +This section discusses how to locate records from a table. + +#### Primary key + +A primary key uniquely identifies each record; no two records can have the same primary key. Therefore, you can locate a record by specifying a primary key. A primary key comprises a partition key and, optionally, a clustering key. + +#### Partition key + +A partition key uniquely identifies a partition. A partition key comprises a set of columns, which are called partition key columns. When you specify only a partition key, you can get a set of records that belong to the partition. + +#### Clustering key + +A clustering key uniquely identifies a record within a partition. It comprises a set of columns called clustering-key columns. When you want to specify a clustering key, you should specify a partition key for efficient lookups. When you specify a clustering key without a partition key, you end up scanning all the partitions. Scanning all the partitions is time consuming, especially when the amount of data is large, so only do so at your own discretion. + +Records within a partition are assumed to be sorted by clustering-key columns, specified as a clustering order. Therefore, you can specify a part of clustering-key columns in the defined order to narrow down the results to be returned. + +#### Index key + +An index key identifies records by looking up the key in indexes. An index key lookup spans all the partitions, so it is not necessarily efficient, especially if the selectivity of a lookup is not low. + +## How to design your database schemas + +You can design your database schemas similarly to the relational model, but there is a basic principle and are a few best practices to follow. + +### Query-driven data modeling + +In relational databases, data is organized in normalized tables with foreign keys used to reference related data in other tables. The queries that the application will make are structured by the tables, and the related data is queried as table joins. + +Although ScalarDB supports join operations in ScalarDB SQL, data modeling should be more query-driven, like NoSQL databases. The data access patterns and application queries should determine the structure and organization of tables. + +### Best practices + +This section describes best practices for designing your database schemas. + +#### Consider data distribution + +Preferably, you should try to balance loads to partitions by properly selecting partition and clustering keys. + +For example, in a banking application, if you choose an account ID as a partition key, you can perform any account operations for a specific account within the partition to which the account belongs. So, if you operate on different account IDs, you will access different partitions. + +On the other hand, if you choose a branch ID as a partition key and an account ID as a clustering key, all the accesses to a branch's account IDs go to the same partition, causing an imbalance in loads and data sizes. In addition, you should choose a high-cardinality column as a partition key because creating a small number of large partitions also causes an imbalance in loads and data sizes. + +#### Try to read a single partition + +Because of the data model characteristics, single partition lookup is most efficient. If you need to issue a scan or select a request that requires multi-partition lookups or scans, which you can [enable with cross-partition scan](configurations.mdx#cross-partition-scan-configurations), do so at your own discretion and consider updating the schemas if possible. + +For example, in a banking application, if you choose email as a partition key and an account ID as a clustering key, and issue a query that specifies an account ID, the query will span all the partitions because it cannot identify the corresponding partition efficiently. In such a case, you should always look up the table with an account ID. + +:::note + +If you read multiple partitions on a relational database with proper indexes, your query might be efficient because the query is pushed down to the database. + +::: + +#### Try to avoid using secondary indexes + +Similarly to the above, if you need to issue a scan or select a request that uses a secondary index, the request will span all the partitions of a table. Therefore, you should try to avoid using secondary indexes. If you need to use a secondary index, use it through a low-selectivity query, which looks up a small portion. + +As an alternative to secondary indexes, you can create another table that works as a clustered index of a base table. + +For example, assume there is a table with three columns: `table1(A, B, C)`, with the primary key `A`. Then, you can create a table like `index-table1(C, A, B)` with `C` as the primary key so that you can look up a single partition by specifying a value for `C`. This approach could speed up read queries but might create more load to write queries because you need to write to two tables by using ScalarDB transactions. + +:::note + +There are plans to have a table-based secondary-index feature in ScalarDB in the future. + +::: + +#### Consider data is assumed to be distributed by hashing + +In the current ScalarDB data model, data is assumed to be distributed by hashing. Therefore, you can't perform range queries efficiently without a partition key. + +If you want to issue range queries efficiently, you need to do so within a partition. However, if you follow this approach, you must specify a partition key. This can pose scalability issues as the range queries always go to the same partition, potentially overloading it. This limitation is not specific to ScalarDB but to databases where data is distributed by hashing for scalability. + +:::note + +If you run ScalarDB on a relational database with proper indexes, your range query might be efficient because the query is pushed down to the database. + +::: + diff --git a/versioned_docs/version-3.X/database-configurations.mdx b/versioned_docs/version-3.X/database-configurations.mdx new file mode 100644 index 00000000..e2b599f2 --- /dev/null +++ b/versioned_docs/version-3.X/database-configurations.mdx @@ -0,0 +1,120 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Configurations for the Underlying Databases of ScalarDB + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +This document explains how to configure the underlying databases of ScalarDB to make applications that use ScalarDB work correctly and efficiently. + +## General requirements for the underlying databases + +ScalarDB requires each underlying database to provide certain capabilities to run transactions and analytics on the databases. This document explains the general requirements and how to configure each database to achieve the requirements. + +### Transactions + +ScalarDB requires each underlying database to provide at least the following capabilities to run transactions on the databases: + +- Linearizable read and conditional mutations (write and delete) on a single database record. +- Durability of written database records. +- Ability to store arbitrary data beside application data in each database record. + +### Analytics + +ScalarDB requires each underlying database to provide the following capability to run analytics on the databases: + +- Ability to return only committed records. + +:::note + +You need to have database accounts that have enough privileges to access the databases through ScalarDB since ScalarDB runs on the underlying databases not only for CRUD operations but also for performing operations like creating or altering schemas, tables, or indexes. ScalarDB basically requires a fully privileged account to access the underlying databases. + +::: + +## How to configure databases to achieve the general requirements + +Select your database for details on how to configure it to achieve the general requirements. + + + +

Transactions

+ + - Use a single primary server or synchronized multi-primary servers for all operations (no read operations on read replicas that are asynchronously replicated from a primary database). + - Use read-committed or stricter isolation levels. + +

Analytics

+ + - Use read-committed or stricter isolation levels. + + +

Transactions

+ + - Use a single primary region for all operations. (No read and write operations on global tables in non-primary regions.) + - There is no concept for primary regions in DynamoDB, so you must designate a primary region by yourself. + +

Analytics

+ + - Not applicable. DynamoDB always returns committed records, so there are no DynamoDB-specific requirements. + + +

Transactions

+ + - Use a single primary region for all operations with `Strong` or `Bounded Staleness` consistency. + +

Analytics

+ + - Not applicable. Cosmos DB always returns committed records, so there are no Cosmos DB–specific requirements. + + +

Transactions

+ + - Use a single primary cluster for all operations (no read or write operations in non-primary clusters). + - Use `batch` or `group` for `commitlog_sync`. + - If you're using Cassandra-compatible databases, those databases must properly support lightweight transactions (LWT). + +

Analytics

+ + - Not applicable. Cassandra always returns committed records, so there are no Cassandra-specific requirements. + + + +## Recommendations + +Properly configuring each underlying database of ScalarDB for high performance and high availability is recommended. The following recommendations include some knobs and configurations to update. + +:::note + +ScalarDB can be seen as an application of underlying databases, so you may want to try updating other knobs and configurations that are commonly used to improve efficiency. + +::: + + + + - Use read-committed isolation for better performance. + - Follow the performance optimization best practices for each database. For example, increasing the buffer size (for example, `shared_buffers` in PostgreSQL) and increasing the number of connections (for example, `max_connections` in PostgreSQL) are usually recommended for better performance. + + + - Increase the number of read capacity units (RCUs) and write capacity units (WCUs) for high throughput. + - Enable point-in-time recovery (PITR). + +:::note + +Since DynamoDB stores data in multiple availability zones by default, you don’t need to adjust any configurations to improve availability. + +::: + + + - Increase the number of Request Units (RUs) for high throughput. + - Enable point-in-time restore (PITR). + - Enable availability zones. + + + - Increase `concurrent_reads` and `concurrent_writes` for high throughput. For details, see the official Cassandra documentation about [`concurrent_writes`](https://cassandra.apache.org/doc/stable/cassandra/configuration/cass_yaml_file.html#concurrent_writes). + + diff --git a/versioned_docs/version-3.X/deploy-overview.mdx b/versioned_docs/version-3.X/deploy-overview.mdx new file mode 100644 index 00000000..d72a68d1 --- /dev/null +++ b/versioned_docs/version-3.X/deploy-overview.mdx @@ -0,0 +1,23 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Deploy Overview + +In this category, you can follow guides to help you become more familiar with deploying ScalarDB, specifically ScalarDB Cluster and ScalarDB Analytics, in local and cloud-based Kubernetes environments. + +## Deploy ScalarDB Cluster in a local Kubernetes environment + +To learn how to deploy ScalarDB Cluster in a local Kubernetes environment by using a Helm Chart and a PostgreSQL database, see [Deploy ScalarDB Cluster Locally](scalardb-cluster/setup-scalardb-cluster-on-kubernetes-by-using-helm-chart.mdx). + +## Deploy ScalarDB Cluster in a cloud-based Kubernetes environment + +To learn how to deploy ScalarDB Cluster in a cloud-based Kubernetes environment by using a Helm Chart, see [Deploy ScalarDB Cluster on Amazon Elastic Kubernetes Service (EKS)](scalar-kubernetes/ManualDeploymentGuideScalarDBClusterOnEKS.mdx). + +## Deploy ScalarDB Analytics in a public cloud-based environment + +To learn how to deploy ScalarDB Analytics in a public cloud-based environment, see [Deploy ScalarDB Analytics in Public Cloud Environments](scalardb-analytics/deployment.mdx). diff --git a/versioned_docs/version-3.X/design.mdx b/versioned_docs/version-3.X/design.mdx new file mode 100644 index 00000000..34b85e49 --- /dev/null +++ b/versioned_docs/version-3.X/design.mdx @@ -0,0 +1,80 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# ScalarDB Design + +This document briefly explains the design and implementation of ScalarDB. For what ScalarDB is and its use cases, see [ScalarDB Overview](./overview.mdx). + +## Overall architecture + +ScalarDB is hybrid transaction/analytical processing (HTAP) middleware that sits in between applications and databases. As shown in the following figure, ScalarDB consists of three components: Core, Cluster, and Analytics. ScalarDB basically employs a layered architecture, so the Cluster and Analytics components use the Core component to interact with underlying databases but sometimes bypass the Core component for performance optimization without sacrificing correctness. Likewise, each component also consists of several layers. + +![ScalarDB architecture](images/scalardb-architecture.png) + +## Components + +The following subsections explain each component one by one. + +### Core + +ScalarDB Core, which is provided as open-source software under the Apache 2 License, is an integral part of ScalarDB. Core provides a database manager that has an abstraction layer that abstracts underlying databases and adapters (or shims) that implement the abstraction for each database. In addition, it provides a transaction manager on top of the database abstraction that achieves database-agnostic transaction management based on Scalar's novel distributed transaction protocol called [Consensus Commit](./consensus-commit.mdx). Core is provided as a library that offers a simple CRUD interface. + +### Cluster + +ScalarDB Cluster, which is licensed under a commercial license, is a component that provides a clustering solution for the Core component to work as a clustered server. Cluster is mainly designed for OLTP workloads, which have many small, transactional and non-transactional reads and writes. In addition, it provides several enterprise features such as authentication, authorization, encryption at rest, and fine-grained access control (attribute-based access control). Not only does Cluster offer the same CRUD interface as the Core component, but it also offers SQL and GraphQL interfaces. Furthermore, it offers a vector store interface to interact with several vector stores. Since Cluster is provided as a container in a Kubernetes Pod, you can increase performance and availability by having more containers. + +### Analytics + +ScalarDB Analytics, which is licensed under a commercial license, is a component that provides scalable analytical processing for the data managed by the Core component or managed by applications that don’t use ScalarDB. Analytics is mainly designed for OLAP workloads, which have a small number of large, analytical read queries. In addition, it offers a SQL and DataSet API through Spark. Since the Analytics component is provided as a Java package that can be installed on Apache Spark engines, you can increase performance by having more Spark worker nodes. + +## Metadata tables + +ScalarDB manages various types of metadata in the underlying databases to provide its capabilities. The following table summarizes the metadata managed by each component. + +| Component | Metadata tables | Purpose | Location | +| --------- | ------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- | +| Core | `scalardb.metadata` | For database schema information | In all the databases under ScalarDB | +| Core | `coordinator.state` | For transaction statuses | In one designated database specified to store the Coordinator table | +| Core | Application-managed tables | For WAL information | In all the tables accessed by Consensus Commit | +| Cluster | `scalardb.users`, `scalardb.namespace_privileges`, `scalardb.table_privileges`, `scalardb.auth_tokens` | For [authentication and authorization](./scalardb-cluster/scalardb-auth-with-sql.mdx) | In one designated database specified to store the scalardb system namespace | +| Cluster | `scalardb.encrypted_columns` | For [encryption at rest](./scalardb-cluster/encrypt-data-at-rest.mdx) | In one designated database specified to store the scalardb system namespace | +| Cluster | `scalardb.abac_*` | For [attribute-based access control](./scalardb-cluster/authorize-with-abac.mdx) | In one designated database specified to store the scalardb system namespace | +| Analytics | All the tables managed by the catalog server | For [data catalog](./scalardb-analytics/design.mdx#universal-data-catalog) | In the catalog server database | + +:::note + +If you need to take backups of the databases accessed by ScalarDB, you will also need to take backups of the metadata managed by ScalarDB. For more details, see [How to Back Up and Restore Databases Used Through ScalarDB](./backup-restore.mdx). + +::: + +## Limitations + +ScalarDB operates between applications and databases, which leads to certain limitations. This section summarizes the limitations of ScalarDB. + +### Applications cannot bypass ScalarDB to run transactions and analytical queries + +ScalarDB Core offers a database-agnostic transaction capability that operates outside of databases. Therefore, applications must interact with ScalarDB to execute transactions; otherwise, ScalarDB cannot ensure transaction correctness, such as snapshot and serializable isolation. For more details, see [Consensus Commit](./consensus-commit.mdx). + +Likewise, ScalarDB Analytics offers a scalable analytical query processing capability that operates outside of databases. Therefore, applications must interact with ScalarDB Analytics to execute analytical queries; otherwise, ScalarDB cannot ensure correctness, such as read-committed isolation. For more details, see [ScalarDB Analytics Design](./scalardb-analytics/design.mdx). + +### Applications cannot use all the capabilities of the underlying databases + +ScalarDB serves as an abstraction layer over the underlying databases, which means that applications cannot use all the capabilities and data types of these databases. For instance, ScalarDB does not support database-specific features such as Oracle PL/SQL. + +ScalarDB has been enhanced to provide features that are commonly found in most supported databases. For a list of features, see [ScalarDB Features](./features.mdx). To learn about the features planned for future releases, see [Roadmap](./roadmap.mdx). + +## Further reading + +For more details about the design and implementation of ScalarDB, see the following documents: + +- **Speaker Deck presentation:** [ScalarDB: Universal Transaction Manager](https://speakerdeck.com/scalar/scalar-db-universal-transaction-manager) + +In addition, the following materials were presented at the VLDB 2023 conference: + +- **Speaker Deck presentation:** [ScalarDB: Universal Transaction Manager for Polystores](https://speakerdeck.com/scalar/scalardb-universal-transaction-manager-for-polystores-vldb23) +- **Detailed paper:** [ScalarDB: Universal Transaction Manager for Polystores](https://www.vldb.org/pvldb/vol16/p3768-yamada.pdf) diff --git a/versioned_docs/version-3.X/develop-overview.mdx b/versioned_docs/version-3.X/develop-overview.mdx new file mode 100644 index 00000000..652ca411 --- /dev/null +++ b/versioned_docs/version-3.X/develop-overview.mdx @@ -0,0 +1,35 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Develop Overview + +In this category, you can follow guides to help you become more familiar with ScalarDB, specifically with how to run transactions, analytical queries, and non-transactional storage operations. + +To get started with developing applications for ScalarDB, see the following sub-categories. + +## Run transactions + +In this sub-category, you can learn how to model your data based on the ScalarDB data model and create schemas. Then, you can learn how to run transactions through the ScalarDB Core library and ScalarDB Cluster, a gRPC server that wraps the Core library. + +For an overview of this sub-category, see [Run Transactions Overview](develop-run-transactions-overview.mdx). + +## Run non-transactional operations + +In this sub-category, you can learn how to run such non-transactional storage operations. + +For an overview of this sub-category, see [Run Non-Transactional Operations Overview](develop-run-non-transactional-operations-overview.mdx). + +## Run analytical queries + +To learn how to run analytical queries by using ScalarDB Analytics, see [Run Analytical Queries Through ScalarDB Analytics](scalardb-analytics/run-analytical-queries.mdx). + +## Run sample applications + +In this sub-category, you can learn how to run various sample applications that take advantage of ScalarDB. + +For an overview of this sub-category, see [Run Sample Applications Overview](scalardb-samples/README.mdx). diff --git a/versioned_docs/version-3.X/develop-run-non-transactional-operations-overview.mdx b/versioned_docs/version-3.X/develop-run-non-transactional-operations-overview.mdx new file mode 100644 index 00000000..a4c22448 --- /dev/null +++ b/versioned_docs/version-3.X/develop-run-non-transactional-operations-overview.mdx @@ -0,0 +1,21 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Run Non-Transactional Storage Operations Overview + +ScalarDB was initially designed to provide a unified abstraction between diverse databases and transactions across such databases. However, there are cases where you only need the unified abstraction to simplify your applications that use multiple, possibly diverse, databases. + +ScalarDB can be configured to provide only the unified abstraction, without transaction capabilities, so that it only runs non-transactional operations on the underlying database and storage. Since ScalarDB in this configuration doesn't guarantee ACID across multiple operations, you can perform operations with better performance. + +In this sub-category, you can learn how to run such non-transactional storage operations. + +- Run Through the CRUD Interface + - [Use the ScalarDB Core Library](run-non-transactional-storage-operations-through-library.mdx) + - [Use ScalarDB Cluster](scalardb-cluster/run-non-transactional-storage-operations-through-scalardb-cluster.mdx) +- [Run Through the SQL Interface](scalardb-cluster/run-non-transactional-storage-operations-through-sql-interface.mdx) +- [Run Through the Primitive CRUD Interface](run-non-transactional-storage-operations-through-primitive-crud-interface.mdx) diff --git a/versioned_docs/version-3.X/develop-run-transactions-overview.mdx b/versioned_docs/version-3.X/develop-run-transactions-overview.mdx new file mode 100644 index 00000000..17cc7831 --- /dev/null +++ b/versioned_docs/version-3.X/develop-run-transactions-overview.mdx @@ -0,0 +1,17 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Run Transactions Overview + +In this sub-category, you can learn how to model your data based on the ScalarDB data model and create schemas. Then, you can learn how to run transactions through the ScalarDB Core library and ScalarDB Cluster, a gRPC server that wraps the Core library. + +- [Model Your Data](data-modeling.mdx) +- Run Through the CRUD Interface + - [Use the ScalarDB Core Library](run-transactions-through-scalardb-core-library.mdx) + - [Use ScalarDB Cluster](scalardb-cluster/run-transactions-through-scalardb-cluster.mdx) +- [Run Through the SQL Interface](scalardb-cluster/run-transactions-through-scalardb-cluster-sql.mdx) diff --git a/versioned_docs/version-3.X/features.mdx b/versioned_docs/version-3.X/features.mdx new file mode 100644 index 00000000..0ba5766d --- /dev/null +++ b/versioned_docs/version-3.X/features.mdx @@ -0,0 +1,29 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# ScalarDB Features + +This document briefly explains which features are available in which editions of ScalarDB. + +| | ScalarDB Core (Community) | ScalarDB Cluster (Enterprise Standard) | ScalarDB Cluster (Enterprise Premium) | ScalarDB Analytics (Enterprise) | +|-------------------------------------------------------------------------------------------------------------------------------------|---------------------------|----------------------------------------|------------------------------------------------------------|---------------------------------| +| [Transaction processing across databases with primitive interfaces](getting-started-with-scalardb.mdx) | ✅ | ✅ | ✅ | – | +| [Clustering](scalardb-cluster/developer-guide-for-scalardb-cluster-with-java-api.mdx) | - | ✅ | ✅ | – | +| [Non-transactional storage operations](develop-run-non-transactional-operations-overview.mdx) | – | ✅ (3.14+) | ✅ (3.14+) | – | +| [Authentication/authorization](scalardb-cluster/scalardb-auth-with-sql.mdx) | – | ✅ | ✅ | – | +| [Encryption](scalardb-cluster/encrypt-data-at-rest.mdx) | – | – | ✅ (3.14+) | – | +| [Attribute-based access control](scalardb-cluster/authorize-with-abac.mdx) | – | – | ✅ (3.15+) (Enterprise Premium Option*, Private Preview**) | – | +| [SQL interface (SQL API, JDBC, Spring Data JDBC, and LINQ)](scalardb-sql/index.mdx) | – | – | ✅ | – | +| [GraphQL interface](scalardb-graphql/index.mdx) | – | – | ✅ | – | +| [Vector search interface](scalardb-cluster/getting-started-with-vector-search.mdx) | – | – | ✅ (3.15+) (Private Preview**) | – | +| [Analytical query processing across ScalarDB-managed data sources](scalardb-samples/scalardb-analytics-spark-sample/README.mdx) | – | – | – | ✅ (3.14+) | +| [Analytical query processing across non-ScalarDB-managed data sources](scalardb-samples/scalardb-analytics-spark-sample/README.mdx) | – | – | – | ✅ (3.15+) | + +\* This feature is not available in the Enterprise Premium edition. If you want to use this feature, please [contact us](https://www.scalar-labs.com/contact). + +\*\* This feature is currently in Private Preview. For details, please [contact us](https://www.scalar-labs.com/contact) or wait for this feature to become publicly available in a future version. diff --git a/versioned_docs/version-3.X/getting-started-with-scalardb-by-using-kotlin.mdx b/versioned_docs/version-3.X/getting-started-with-scalardb-by-using-kotlin.mdx new file mode 100644 index 00000000..2284966c --- /dev/null +++ b/versioned_docs/version-3.X/getting-started-with-scalardb-by-using-kotlin.mdx @@ -0,0 +1,413 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Getting Started with ScalarDB by Using Kotlin + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +This getting started tutorial explains how to configure your preferred database in ScalarDB and set up a basic electronic money application by using Kotlin. Since Kotlin has Java interoperability, you can use ScalarDB directly from Kotlin. + +:::warning + +The electronic money application is simplified for this tutorial and isn't suitable for a production environment. + +::: + +## Prerequisites for this sample application + +- OpenJDK LTS version (8, 11, 17, or 21) from [Eclipse Temurin](https://adoptium.net/temurin/releases/) +- [Docker](https://www.docker.com/get-started/) 20.10 or later with [Docker Compose](https://docs.docker.com/compose/install/) V2 or later + +:::note + +This sample application has been tested with OpenJDK from Eclipse Temurin. ScalarDB itself, however, has been tested with JDK distributions from various vendors. For details about the requirements for ScalarDB, including compatible JDK distributions, please see [Requirements](./requirements.mdx). + +::: + +## Clone the ScalarDB samples repository + +Open **Terminal**, then clone the ScalarDB samples repository by running the following command: + +```console +git clone https://github.com/scalar-labs/scalardb-samples +``` + +Then, go to the directory that contains the sample application by running the following command: + +```console +cd scalardb-samples/scalardb-kotlin-sample +``` + +## Set up your database for ScalarDB + +Select your database, and follow the instructions to configure it for ScalarDB. + +For a list of databases that ScalarDB supports, see [Databases](requirements.mdx#databases). + + + +

Run MySQL locally

+ + You can run MySQL in Docker Compose by using the `docker-compose.yml` file in the `scalardb-samples/scalardb-kotlin-sample` directory. + + To start MySQL, run the following command: + + ```console + docker compose up -d mysql + ``` + +

Configure ScalarDB

+ + The **database.properties** file in the `scalardb-samples/scalardb-kotlin-sample` directory contains database configurations for ScalarDB. Please uncomment the properties for MySQL in the **database.properties** file so that the configuration looks as follows: + + ```properties + # For MySQL + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:mysql://localhost:3306/ + scalar.db.username=root + scalar.db.password=mysql + ``` + + +

Run PostgreSQL locally

+ + You can run PostgreSQL in Docker Compose by using the `docker-compose.yml` file in the `scalardb-samples/scalardb-kotlin-sample` directory. + + To start PostgreSQL, run the following command: + + ```console + docker compose up -d postgres + ``` + +

Configure ScalarDB

+ + The **database.properties** file in the `scalardb-samples/scalardb-kotlin-sample` directory contains database configurations for ScalarDB. Please uncomment the properties for PostgreSQL in the **database.properties** file so that the configuration looks as follows: + + ```properties + # For PostgreSQL + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:postgresql://localhost:5432/ + scalar.db.username=postgres + scalar.db.password=postgres + ``` + + +

Run Oracle Database locally

+ + You can run Oracle Database in Docker Compose by using the `docker-compose.yml` file in the `scalardb-samples/scalardb-kotlin-sample` directory. + + To start Oracle Database, run the following command: + + ```console + docker compose up -d oracle + ``` + +

Configure ScalarDB

+ + The **database.properties** file in the `scalardb-samples/scalardb-kotlin-sample` directory contains database configurations for ScalarDB. Please uncomment the properties for Oracle Database in the **database.properties** file so that the configuration looks as follows: + + ```properties + # For Oracle + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:oracle:thin:@//localhost:1521/FREEPDB1 + scalar.db.username=SYSTEM + scalar.db.password=Oracle + ``` + + +

Run SQL Server locally

+ + You can run SQL Server in Docker Compose by using the `docker-compose.yml` file in the `scalardb-samples/scalardb-kotlin-sample` directory. + + To start SQL Server, run the following command: + + ```console + docker compose up -d sqlserver + ``` + +

Configure ScalarDB

+ + The **database.properties** file in the `scalardb-samples/scalardb-kotlin-sample` directory contains database configurations for ScalarDB. Please uncomment the properties for SQL Server in the **database.properties** file so that the configuration looks as follows: + + ```properties + # For SQL Server + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:sqlserver://localhost:1433;encrypt=true;trustServerCertificate=true + scalar.db.username=sa + scalar.db.password=SqlServer22 + ``` + + +

Run Amazon DynamoDB Local

+ + You can run Amazon DynamoDB Local in Docker Compose by using the `docker-compose.yml` file in the `scalardb-samples/scalardb-kotlin-sample` directory. + + To start Amazon DynamoDB Local, run the following command: + + ```console + docker compose up -d dynamodb + ``` + +

Configure ScalarDB

+ + The **database.properties** file in the `scalardb-samples/scalardb-kotlin-sample` directory contains database configurations for ScalarDB. Please uncomment the properties for Amazon DynamoDB Local in the **database.properties** file so that the configuration looks as follows: + + ```properties + # For DynamoDB Local + scalar.db.storage=dynamo + scalar.db.contact_points=sample + scalar.db.username=sample + scalar.db.password=sample + scalar.db.dynamo.endpoint_override=http://localhost:8000 + ``` + + + To use Azure Cosmos DB for NoSQL, you must have an Azure account. If you don't have an Azure account, visit [Create an Azure Cosmos DB account](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/quickstart-portal#create-account). + +

Configure Cosmos DB for NoSQL

+ + Set the **default consistency level** to **Strong** according to the official document at [Configure the default consistency level](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-manage-consistency#configure-the-default-consistency-level). + +

Configure ScalarDB

+ + The following instructions assume that you have properly installed and configured the JDK in your local environment and properly configured your Cosmos DB for NoSQL account in Azure. + + The **database.properties** file in the `scalardb-samples/scalardb-kotlin-sample` directory contains database configurations for ScalarDB. Be sure to change the values for `scalar.db.contact_points` and `scalar.db.password` as described. + + ```properties + # For Cosmos DB + scalar.db.storage=cosmos + scalar.db.contact_points= + scalar.db.password= + ``` + +:::note + +You can use the primary key or the secondary key in your Azure Cosmos DB account as the value for `scalar.db.password`. + +::: + + +

Run Cassandra locally

+ + You can run Apache Cassandra in Docker Compose by using the `docker-compose.yml` file in the `scalardb-samples/scalardb-kotlin-sample` directory. + + To start Apache Cassandra, run the following command: + ```console + docker compose up -d cassandra + ``` + +

Configure ScalarDB

+ + The **database.properties** file in the `scalardb-samples/scalardb-kotlin-sample` directory contains database configurations for ScalarDB. Please uncomment the properties for Cassandra in the **database.properties** file so that the configuration looks as follows: + + ```properties + # For Cassandra + scalar.db.storage=cassandra + scalar.db.contact_points=localhost + scalar.db.username=cassandra + scalar.db.password=cassandra + ``` + + + +## Load the database schema + +You need to define the database schema (the method in which the data will be organized) in the application. For details about the supported data types, see [Data type mapping between ScalarDB and other databases](schema-loader.mdx#data-type-mapping-between-scalardb-and-other-databases). + +For this tutorial, a file named **schema.json** already exists in the `scalardb-samples/scalardb-kotlin-sample` directory. To apply the schema, go to the [`scalardb` Releases](https://github.com/scalar-labs/scalardb/releases) page and download the ScalarDB Schema Loader that matches the version of ScalarDB that you are using to the `scalardb-samples/scalardb-kotlin-sample` directory. + +Then, based on your database, run the following command, replacing `` with the version of the ScalarDB Schema Loader that you downloaded: + + + + ```console + java -jar scalardb-schema-loader-.jar --config database.properties --schema-file schema.json --coordinator + ``` + +:::note + +The `--coordinator` option is specified because a table with `transaction` set to `true` exists in the schema. For details about configuring and loading a schema, see [ScalarDB Schema Loader](schema-loader.mdx). + +::: + + + ```console + java -jar scalardb-schema-loader-.jar --config database.properties --schema-file schema.json --coordinator + ``` + +:::note + +The `--coordinator` option is specified because a table with `transaction` set to `true` exists in the schema. For details about configuring and loading a schema, see [ScalarDB Schema Loader](schema-loader.mdx). + +::: + + + ```console + java -jar scalardb-schema-loader-.jar --config database.properties --schema-file schema.json --coordinator + ``` + +:::note + +The `--coordinator` option is specified because a table with `transaction` set to `true` exists in the schema. For details about configuring and loading a schema, see [ScalarDB Schema Loader](schema-loader.mdx). + +::: + + + ```console + java -jar scalardb-schema-loader-.jar --config database.properties --schema-file schema.json --coordinator + ``` + +:::note + +The `--coordinator` option is specified because a table with `transaction` set to `true` exists in the schema. For details about configuring and loading a schema, see [ScalarDB Schema Loader](schema-loader.mdx). + +::: + + + ```console + java -jar scalardb-schema-loader-.jar --config database.properties --schema-file schema.json --coordinator --no-backup --no-scaling + ``` + +:::note + +The `--coordinator` option is specified because a table with `transaction` set to `true` exists in the schema. For details about configuring and loading a schema, see [ScalarDB Schema Loader](schema-loader.mdx). + +Also, `--no-backup` and `--no-scaling` options are specified because Amazon DynamoDB Local does not support continuous backup and auto-scaling. + +::: + + + ```console + java -jar scalardb-schema-loader-.jar --config database.properties --schema-file schema.json --coordinator + ``` + +:::note + +The `--coordinator` option is specified because a table with `transaction` set to `true` exists in the schema. For details about configuring and loading a schema, see [ScalarDB Schema Loader](schema-loader.mdx). + +::: + + + ```console + java -jar scalardb-schema-loader-.jar --config database.properties --schema-file schema.json --coordinator --replication-factor=1 + ``` + +:::note + +The `--coordinator` option is specified because a table with `transaction` set to `true` exists in the schema. For details about configuring and loading a schema, see [ScalarDB Schema Loader](schema-loader.mdx). + +In addition, the `--replication-factor=1` option has an effect only when using Cassandra. The default replication factor is `3`, but to facilitate the setup in this tutorial, `1` is used so that you only need to prepare a cluster with one node instead of three nodes. However, keep in mind that a replication factor of `1` is not suited for production. + +::: + + + +## Execute transactions and retrieve data in the basic electronic money application + +After loading the schema, you can execute transactions and retrieve data in the basic electronic money application that is included in the repository that you cloned. + +The application supports the following types of transactions: + +- Create an account. +- Add funds to an account. +- Send funds between two accounts. +- Get an account balance. + +:::note + +When you first execute a Gradle command, Gradle will automatically install the necessary libraries. + +::: + +### Create an account with a balance + +You need an account with a balance so that you can send funds between accounts. + +To create an account for **customer1** that has a balance of **500**, run the following command: + +```console +./gradlew run --args="-action charge -amount 500 -to customer1" +``` + +### Create an account without a balance + +After setting up an account that has a balance, you need another account for sending funds to. + +To create an account for **merchant1** that has a balance of **0**, run the following command: + +```console +./gradlew run --args="-action charge -amount 0 -to merchant1" +``` + +### Add funds to an account + +You can add funds to an account in the same way that you created and added funds to an account in [Create an account with a balance](#create-an-account-with-a-balance). + +To add **500** to the account for **customer1**, run the following command: + +```console +./gradlew run --args="-action charge -amount 500 -to customer1" +``` + +The account for **customer1** will now have a balance of **1000**. + +### Send electronic money between two accounts + +Now that you have created two accounts, with at least one of those accounts having a balance, you can send funds from one account to the other account. + +To have **customer1** pay **100** to **merchant1**, run the following command: + +```console +./gradlew run --args="-action pay -amount 100 -from customer1 -to merchant1" +``` + +### Get an account balance + +After sending funds from one account to the other, you can check the balance of each account. + +To get the balance of **customer1**, run the following command: + +```console +./gradlew run --args="-action getBalance -id customer1" +``` + +You should see the following output: + +```console +... +The balance for customer1 is 900 +... +``` + +To get the balance of **merchant1**, run the following command: + +```console +./gradlew run --args="-action getBalance -id merchant1" +``` + +You should see the following output: + +```console +... +The balance for merchant1 is 100 +... +``` + +## Stop the database + +To stop the database, stop the Docker container by running the following command: + +```console +docker compose down +``` + +## Reference + +To see the source code for the electronic money application used in this tutorial, see [`ElectronicMoney.kt`](https://github.com/scalar-labs/scalardb-samples/blob/main/scalardb-kotlin-sample/src/main/kotlin/sample/ElectronicMoney.kt). diff --git a/versioned_docs/version-3.X/getting-started-with-scalardb.mdx b/versioned_docs/version-3.X/getting-started-with-scalardb.mdx new file mode 100644 index 00000000..0c7bcd8f --- /dev/null +++ b/versioned_docs/version-3.X/getting-started-with-scalardb.mdx @@ -0,0 +1,556 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Getting Started with ScalarDB + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +This getting started tutorial explains how to configure your preferred database in ScalarDB and illustrates the process of creating a sample e-commerce application, where items can be ordered and paid for with a credit card by using ScalarDB. The sample e-commerce application shows how users can order and pay for items by using a line of credit. + +:::warning + +Since the focus of the sample application is to demonstrate using ScalarDB, application-specific error handling, authentication processing, and similar functions are not included in the sample application. For details about exception handling in ScalarDB, see [How to handle exceptions](api-guide.mdx#how-to-handle-exceptions). + +::: + +## Prerequisites for this sample application + +- OpenJDK LTS version (8, 11, 17, or 21) from [Eclipse Temurin](https://adoptium.net/temurin/releases/) +- [Docker](https://www.docker.com/get-started/) 20.10 or later with [Docker Compose](https://docs.docker.com/compose/install/) V2 or later + +:::note + +This sample application has been tested with OpenJDK from Eclipse Temurin. ScalarDB itself, however, has been tested with JDK distributions from various vendors. For details about the requirements for ScalarDB, including compatible JDK distributions, please see [Requirements](./requirements.mdx). + +::: + +## Clone the ScalarDB samples repository + +Open **Terminal**, then clone the ScalarDB samples repository by running the following command: + +```console +git clone https://github.com/scalar-labs/scalardb-samples +``` + +Then, go to the directory that contains the sample application by running the following command: + +```console +cd scalardb-samples/scalardb-sample +``` + +## Set up your database for ScalarDB + +Select your database, and follow the instructions to configure it for ScalarDB. + +For a list of databases that ScalarDB supports, see [Databases](requirements.mdx#databases). + + + +

Run MySQL locally

+ + You can run MySQL in Docker Compose by using the `docker-compose.yml` file in the `scalardb-samples/scalardb-sample` directory. + + To start MySQL, run the following command: + + ```console + docker compose up -d mysql + ``` + +

Configure ScalarDB

+ + The **database.properties** file in the `scalardb-samples/scalardb-sample` directory contains database configurations for ScalarDB. Please uncomment the properties for MySQL in the **database.properties** file so that the configuration looks as follows: + + ```properties + # For MySQL + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:mysql://localhost:3306/ + scalar.db.username=root + scalar.db.password=mysql + ``` + + +

Run PostgreSQL locally

+ + You can run PostgreSQL in Docker Compose by using the `docker-compose.yml` file in the `scalardb-samples/scalardb-sample` directory. + + To start PostgreSQL, run the following command: + + ```console + docker compose up -d postgres + ``` + +

Configure ScalarDB

+ + The **database.properties** file in the `scalardb-samples/scalardb-sample` directory contains database configurations for ScalarDB. Please uncomment the properties for PostgreSQL in the **database.properties** file so that the configuration looks as follows: + + ```properties + # For PostgreSQL + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:postgresql://localhost:5432/ + scalar.db.username=postgres + scalar.db.password=postgres + ``` + + +

Run Oracle Database locally

+ + You can run Oracle Database in Docker Compose by using the `docker-compose.yml` file in the `scalardb-samples/scalardb-sample` directory. + + To start Oracle Database, run the following command: + + ```console + docker compose up -d oracle + ``` + +

Configure ScalarDB

+ + The **database.properties** file in the `scalardb-samples/scalardb-sample` directory contains database configurations for ScalarDB. Please uncomment the properties for Oracle Database in the **database.properties** file so that the configuration looks as follows: + + ```properties + # For Oracle + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:oracle:thin:@//localhost:1521/FREEPDB1 + scalar.db.username=SYSTEM + scalar.db.password=Oracle + ``` + + +

Run SQL Server locally

+ + You can run SQL Server in Docker Compose by using the `docker-compose.yml` file in the `scalardb-samples/scalardb-sample` directory. + + To start SQL Server, run the following command: + + ```console + docker compose up -d sqlserver + ``` + +

Configure ScalarDB

+ + The **database.properties** file in the `scalardb-samples/scalardb-sample` directory contains database configurations for ScalarDB. Please uncomment the properties for SQL Server in the **database.properties** file so that the configuration looks as follows: + + ```properties + # For SQL Server + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:sqlserver://localhost:1433;encrypt=true;trustServerCertificate=true + scalar.db.username=sa + scalar.db.password=SqlServer22 + ``` + + +

Run Db2 locally

+ + You can run IBM Db2 in Docker Compose by using the `docker-compose.yml` file in the `scalardb-samples/scalardb-sample` directory. + + To start IBM Db2, run the following command: + + ```console + docker compose up -d db2 + ``` + +

Configure ScalarDB

+ + The **database.properties** file in the `scalardb-samples/scalardb-sample` directory contains database configurations for ScalarDB. Please uncomment the properties for Db2 in the **database.properties** file so that the configuration looks as follows: + + ```properties + # For Db2 + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:db2://localhost:50000/sample + scalar.db.username=db2inst1 + scalar.db.password=db2inst1 + ``` + + +

Run Amazon DynamoDB Local

+ + You can run Amazon DynamoDB Local in Docker Compose by using the `docker-compose.yml` file in the `scalardb-samples/scalardb-sample` directory. + + To start Amazon DynamoDB Local, run the following command: + + ```console + docker compose up -d dynamodb + ``` + +

Configure ScalarDB

+ + The **database.properties** file in the `scalardb-samples/scalardb-sample` directory contains database configurations for ScalarDB. Please uncomment the properties for Amazon DynamoDB Local in the **database.properties** file so that the configuration looks as follows: + + ```properties + # For DynamoDB Local + scalar.db.storage=dynamo + scalar.db.contact_points=sample + scalar.db.username=sample + scalar.db.password=sample + scalar.db.dynamo.endpoint_override=http://localhost:8000 + ``` + + + To use Azure Cosmos DB for NoSQL, you must have an Azure account. If you don't have an Azure account, visit [Create an Azure Cosmos DB account](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/quickstart-portal#create-account). + +

Configure Cosmos DB for NoSQL

+ + Set the **default consistency level** to **Strong** according to the official document at [Configure the default consistency level](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-manage-consistency#configure-the-default-consistency-level). + +

Configure ScalarDB

+ + The following instructions assume that you have properly installed and configured the JDK in your local environment and properly configured your Cosmos DB for NoSQL account in Azure. + + The **database.properties** file in the `scalardb-samples/scalardb-sample` directory contains database configurations for ScalarDB. Be sure to change the values for `scalar.db.contact_points` and `scalar.db.password` as described. + + ```properties + # For Cosmos DB + scalar.db.storage=cosmos + scalar.db.contact_points= + scalar.db.password= + ``` + +:::note + +You can use the primary key or the secondary key in your Azure Cosmos DB account as the value for `scalar.db.password`. + +::: + + +

Run Cassandra locally

+ + You can run Apache Cassandra in Docker Compose by using the `docker-compose.yml` file in the `scalardb-samples/scalardb-sample` directory. + + To start Apache Cassandra, run the following command: + ```console + docker compose up -d cassandra + ``` + +

Configure ScalarDB

+ + The **database.properties** file in the `scalardb-samples/scalardb-sample` directory contains database configurations for ScalarDB. Please uncomment the properties for Cassandra in the **database.properties** file so that the configuration looks as follows: + + ```properties + # For Cassandra + scalar.db.storage=cassandra + scalar.db.contact_points=localhost + scalar.db.username=cassandra + scalar.db.password=cassandra + ``` + + + +## Load the database schema + +You need to define the database schema (the method in which the data will be organized) in the application. For details about the supported data types, see [Data type mapping between ScalarDB and other databases](schema-loader.mdx#data-type-mapping-between-scalardb-and-other-databases). + +For this tutorial, a file named **schema.json** already exists in the `scalardb-samples/scalardb-sample` directory. To apply the schema, go to the [`scalardb` Releases](https://github.com/scalar-labs/scalardb/releases) page and download the ScalarDB Schema Loader that matches the version of ScalarDB that you are using to the `scalardb-samples/scalardb-sample` directory. + +Then, run the following command, replacing `` with the version of the ScalarDB Schema Loader that you downloaded: + + + + ```console + java -jar scalardb-schema-loader-.jar --config database.properties --schema-file schema.json --coordinator + ``` + +:::note + +The `--coordinator` option is specified because a table with `transaction` set to `true` exists in the schema. For details about configuring and loading a schema, see [ScalarDB Schema Loader](schema-loader.mdx). + +::: + + + ```console + java -jar scalardb-schema-loader-.jar --config database.properties --schema-file schema.json --coordinator + ``` + +:::note + +The `--coordinator` option is specified because a table with `transaction` set to `true` exists in the schema. For details about configuring and loading a schema, see [ScalarDB Schema Loader](schema-loader.mdx). + +::: + + + ```console + java -jar scalardb-schema-loader-.jar --config database.properties --schema-file schema.json --coordinator + ``` + +:::note + +The `--coordinator` option is specified because a table with `transaction` set to `true` exists in the schema. For details about configuring and loading a schema, see [ScalarDB Schema Loader](schema-loader.mdx). + +::: + + + ```console + java -jar scalardb-schema-loader-.jar --config database.properties --schema-file schema.json --coordinator + ``` + +:::note + +The `--coordinator` option is specified because a table with `transaction` set to `true` exists in the schema. For details about configuring and loading a schema, see [ScalarDB Schema Loader](schema-loader.mdx). + +::: + + + ```console + java -jar scalardb-schema-loader-.jar --config database.properties --schema-file schema.json --coordinator --no-backup --no-scaling + ``` + +:::note + +The `--coordinator` option is specified because a table with `transaction` set to `true` exists in the schema. For details about configuring and loading a schema, see [ScalarDB Schema Loader](schema-loader.mdx). + +Also, `--no-backup` and `--no-scaling` options are specified because Amazon DynamoDB Local does not support continuous backup and auto-scaling. + +::: + + + ```console + java -jar scalardb-schema-loader-.jar --config database.properties --schema-file schema.json --coordinator + ``` + +:::note + +The `--coordinator` option is specified because a table with `transaction` set to `true` exists in the schema. For details about configuring and loading a schema, see [ScalarDB Schema Loader](schema-loader.mdx). + +::: + + + ```console + java -jar scalardb-schema-loader-.jar --config database.properties --schema-file schema.json --coordinator --replication-factor=1 + ``` + +:::note + +The `--coordinator` option is specified because a table with `transaction` set to `true` exists in the schema. For details about configuring and loading a schema, see [ScalarDB Schema Loader](schema-loader.mdx). + +In addition, the `--replication-factor=1` option has an effect only when using Cassandra. The default replication factor is `3`, but to facilitate the setup in this tutorial, `1` is used so that you only need to prepare a cluster with one node instead of three nodes. However, keep in mind that a replication factor of `1` is not suited for production. + +::: + + + +### Schema details + +As shown in [`schema.json`](https://github.com/scalar-labs/scalardb-samples/tree/main/scalardb-sample/schema.json) for the sample application, all the tables are created in the `sample` namespace. + +- `sample.customers`: a table that manages customer information + - `credit_limit`: the maximum amount of money that the lender will allow the customer to spend from their line of credit + - `credit_total`: the amount of money that the customer has spent from their line of credit +- `sample.orders`: a table that manages order information +- `sample.statements`: a table that manages order statement information +- `sample.items`: a table that manages information for items to be ordered + +The Entity Relationship Diagram for the schema is as follows: + +![ERD](images/getting-started-ERD.png) + +### Load the initial data + +Before running the sample application, you need to load the initial data by running the following command: + +```console +./gradlew run --args="LoadInitialData" +``` + +After the initial data has loaded, the following records should be stored in the tables. + +**`sample.customers` table** + +| customer_id | name | credit_limit | credit_total | +|-------------|---------------|--------------|--------------| +| 1 | Yamada Taro | 10000 | 0 | +| 2 | Yamada Hanako | 10000 | 0 | +| 3 | Suzuki Ichiro | 10000 | 0 | + +**`sample.items` table** + +| item_id | name | price | +|---------|--------|-------| +| 1 | Apple | 1000 | +| 2 | Orange | 2000 | +| 3 | Grape | 2500 | +| 4 | Mango | 5000 | +| 5 | Melon | 3000 | + +## Execute transactions and retrieve data in the sample application + +The following sections describe how to execute transactions and retrieve data in the sample e-commerce application. + +### Get customer information + +Start with getting information about the customer whose ID is `1` by running the following command: + +```console +./gradlew run --args="GetCustomerInfo 1" +``` + +You should see the following output: + +```console +... +{"id": 1, "name": "Yamada Taro", "credit_limit": 10000, "credit_total": 0} +... +``` + +### Place an order + +Then, have customer ID `1` place an order for three apples and two oranges by running the following command: + +:::note + +The order format in this command is `./gradlew run --args="PlaceOrder :,:,..."`. + +::: + +```console +./gradlew run --args="PlaceOrder 1 1:3,2:2" +``` + +You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: + +```console +... +{"order_id": "dea4964a-ff50-4ecf-9201-027981a1566e"} +... +``` + +### Check order details + +Check details about the order by running the following command, replacing `` with the UUID for the `order_id` that was shown after running the previous command: + +```console +./gradlew run --args="GetOrder " +``` + +You should see a similar output as below, with different UUIDs for `order_id` and `timestamp`: + +```console +... +{"order": {"order_id": "dea4964a-ff50-4ecf-9201-027981a1566e","timestamp": 1650948340914,"customer_id": 1,"customer_name": "Yamada Taro","statement": [{"item_id": 1,"item_name": "Apple","price": 1000,"count": 3,"total": 3000},{"item_id": 2,"item_name": "Orange","price": 2000,"count": 2,"total": 4000}],"total": 7000}} +... +``` + +### Place another order + +Place an order for one melon that uses the remaining amount in `credit_total` for customer ID `1` by running the following command: + +```console +./gradlew run --args="PlaceOrder 1 5:1" +``` + +You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: + +```console +... +{"order_id": "bcc34150-91fa-4bea-83db-d2dbe6f0f30d"} +... +``` + +### Check order history + +Get the history of all orders for customer ID `1` by running the following command: + +```console +./gradlew run --args="GetOrders 1" +``` + +You should see a similar output as below, with different UUIDs for `order_id` and `timestamp`, which shows the history of all orders for customer ID `1` in descending order by timestamp: + +```console +... +{"order": [{"order_id": "dea4964a-ff50-4ecf-9201-027981a1566e","timestamp": 1650948340914,"customer_id": 1,"customer_name": "Yamada Taro","statement": [{"item_id": 1,"item_name": "Apple","price": 1000,"count": 3,"total": 3000},{"item_id": 2,"item_name": "Orange","price": 2000,"count": 2,"total": 4000}],"total": 7000},{"order_id": "bcc34150-91fa-4bea-83db-d2dbe6f0f30d","timestamp": 1650948412766,"customer_id": 1,"customer_name": "Yamada Taro","statement": [{"item_id": 5,"item_name": "Melon","price": 3000,"count": 1,"total": 3000}],"total": 3000}]} +... +``` + +### Check credit total + +Get the credit total for customer ID `1` by running the following command: + +```console +./gradlew run --args="GetCustomerInfo 1" +``` + +You should see the following output, which shows that customer ID `1` has reached their `credit_limit` in `credit_total` and cannot place anymore orders: + +```console +... +{"id": 1, "name": "Yamada Taro", "credit_limit": 10000, "credit_total": 10000} +... +``` + +Try to place an order for one grape and one mango by running the following command: + +```console +./gradlew run --args="PlaceOrder 1 3:1,4:1" +``` + +You should see the following output, which shows that the order failed because the `credit_total` amount would exceed the `credit_limit` amount. + +```console +... +java.lang.RuntimeException: Credit limit exceeded + at sample.Sample.placeOrder(Sample.java:205) + at sample.command.PlaceOrderCommand.call(PlaceOrderCommand.java:33) + at sample.command.PlaceOrderCommand.call(PlaceOrderCommand.java:8) + at picocli.CommandLine.executeUserObject(CommandLine.java:1783) + at picocli.CommandLine.access$900(CommandLine.java:145) + at picocli.CommandLine$RunLast.handle(CommandLine.java:2141) + at picocli.CommandLine$RunLast.handle(CommandLine.java:2108) + at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:1975) + at picocli.CommandLine.execute(CommandLine.java:1904) + at sample.command.SampleCommand.main(SampleCommand.java:35) +... +``` + +### Make a payment + +To continue making orders, customer ID `1` must make a payment to reduce the `credit_total` amount. + +Make a payment by running the following command: + +```console +./gradlew run --args="Repayment 1 8000" +``` + +Then, check the `credit_total` amount for customer ID `1` by running the following command: + +```console +./gradlew run --args="GetCustomerInfo 1" +``` + +You should see the following output, which shows that a payment was applied to customer ID `1`, reducing the `credit_total` amount: + +```console +... +{"id": 1, "name": "Yamada Taro", "credit_limit": 10000, "credit_total": 2000} +... +``` + +Now that customer ID `1` has made a payment, place an order for one grape and one melon by running the following command: + +```console +./gradlew run --args="PlaceOrder 1 3:1,4:1" +``` + +You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: + +```console +... +{"order_id": "8911cab3-1c2b-4322-9386-adb1c024e078"} +... +``` + +## Stop the database + +To stop the database, stop the Docker container by running the following command: + +```console +docker compose down +``` + +## Reference + +To see the source code for the e-commerce application used in this tutorial, see [`Sample.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/scalardb-sample/src/main/java/sample/Sample.java). diff --git a/versioned_docs/version-3.X/glossary.mdx b/versioned_docs/version-3.X/glossary.mdx new file mode 100644 index 00000000..6cd569d6 --- /dev/null +++ b/versioned_docs/version-3.X/glossary.mdx @@ -0,0 +1,119 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Glossary + +This glossary includes database and distributed-system terms that are often used when using ScalarDB. + +## ACID + +Atomicity, consistency, isolation, and durability (ACID) is a set of properties that ensure database transactions are processed reliably, maintaining integrity even in cases of errors or system failures. + +## concurrency control + +Concurrency control in databases ensures that multiple transactions can occur simultaneously without causing data inconsistency, usually through mechanisms like locking or timestamp ordering. + +## consensus + +Consensus in distributed systems refers to the process of achieving agreement among multiple computers or nodes on a single data value or system state. + +## data federation + +Data federation is the process of integrating data from different sources without moving the data, creating a unified view for querying and analysis. + +## data mesh + +A data mesh is a decentralized data architecture that enables each business domain within a company to autonomously manage data and use it efficiently. + +## data virtualization + +Data virtualization is similar to data federation in many aspects, meaning that it virtualizes multiple data sources into a unified view, simplifying queries without moving the data. + +## database anomalies + +Database anomalies are inconsistencies or errors in data that can occur when operations such as insertions, updates, or deletions are performed without proper transaction management. + +## federation engine + +A federation engine facilitates data integration and querying across multiple disparate data sources, often as part of a data federation architecture. + +## global transaction + +A global transaction spans multiple databases or distributed systems and ensures that all involved systems commit or roll back changes as a single unit. + +## heterogeneous databases + +Heterogeneous databases refer to systems composed of different database technologies that may have distinct data models, query languages, and transaction mechanisms. + +## HTAP + +Hybrid transactional/analytical processing (HTAP) refers to a system that can handle both transactional and analytical workloads concurrently on the same data set, removing the need for separate databases. + +## JDBC + +Java Database Connectivity (JDBC) is an API that allows Java applications to interact with databases, providing methods for querying and updating data in relational databases. + +## linearizability + +Linearizability is a strong consistency model in distributed systems where operations appear to occur atomically in some order, and each operation takes effect between its start and end. + +## NoSQL database + +A NoSQL database is a non-relational databases designed for specific data models, such as document, key-value, wide-column, or graph stores, often used for handling large-scale, distributed data. + +## Paxos + +Paxos is a family of protocols used in distributed systems to achieve consensus, even in the presence of node failures. + +## PITR + +Point-in-time recovery (PITR) allows a database to be restored to a previous state at any specific time, usually after an unintended event like data corruption. + +## polystores + +Polystores are database architectures that allow users to interact with multiple, heterogeneous data stores, each optimized for a specific workload or data type, as if they were a single system. + +## read-committed isolation + +Read-committed isolation is an isolation level where each transaction sees only committed data, preventing dirty reads but allowing non-repeatable reads. + +## relational database + +A relational database stores data in tables with rows and columns, using a structured query language (SQL) to define, query, and manipulate the data. + +## replication + +Replication in databases involves copying and distributing data across multiple machines or locations to ensure reliability, availability, and fault tolerance. + +## Saga + +The Saga pattern is a method for managing long-running transactions in a distributed system, where each operation in the transaction is followed by a compensating action in case of failure. + +## serializable isolation + +Serializable isolation (serializability) is the highest isolation level in transactional systems, ensuring that the outcome of concurrently executed transactions is the same as if they were executed sequentially. + +## snapshot isolation + +Snapshot isolation is an isolation level that allows transactions to read a consistent snapshot of the database, protecting them from seeing changes made by other transactions until they complete. + +## TCC + +Try-Confirm/Cancel (TCC) is a pattern for distributed transactions that splits an operation into three steps, allowing for coordination and recovery across multiple systems. + +## transaction + +A transaction in databases is a sequence of operations treated as a single logical unit of work, ensuring consistency and integrity, typically conforming to ACID properties. + +## transaction manager + +A transaction manager coordinates the execution of transactions across multiple systems or databases, ensuring that all steps of the transaction succeed or fail as a unit. + +## two-phase commit + +Two-phase commit is a protocol for ensuring all participants in a distributed transaction either commit or roll back the transaction, ensuring consistency across systems. diff --git a/versioned_docs/version-3.X/images/data_model.png b/versioned_docs/version-3.X/images/data_model.png new file mode 100644 index 00000000..15a0e4d4 Binary files /dev/null and b/versioned_docs/version-3.X/images/data_model.png differ diff --git a/versioned_docs/version-3.X/images/getting-started-ERD.png b/versioned_docs/version-3.X/images/getting-started-ERD.png new file mode 100644 index 00000000..1a6d13c5 Binary files /dev/null and b/versioned_docs/version-3.X/images/getting-started-ERD.png differ diff --git a/versioned_docs/version-3.X/images/scalardb-architecture.png b/versioned_docs/version-3.X/images/scalardb-architecture.png new file mode 100644 index 00000000..6f22111c Binary files /dev/null and b/versioned_docs/version-3.X/images/scalardb-architecture.png differ diff --git a/versioned_docs/version-3.X/images/scalardb-metadata.png b/versioned_docs/version-3.X/images/scalardb-metadata.png new file mode 100644 index 00000000..49880267 Binary files /dev/null and b/versioned_docs/version-3.X/images/scalardb-metadata.png differ diff --git a/versioned_docs/version-3.X/images/scalardb.png b/versioned_docs/version-3.X/images/scalardb.png new file mode 100644 index 00000000..658486cb Binary files /dev/null and b/versioned_docs/version-3.X/images/scalardb.png differ diff --git a/versioned_docs/version-3.X/images/scalardb_data_model.png b/versioned_docs/version-3.X/images/scalardb_data_model.png new file mode 100644 index 00000000..7a02fa23 Binary files /dev/null and b/versioned_docs/version-3.X/images/scalardb_data_model.png differ diff --git a/versioned_docs/version-3.X/images/software_stack.png b/versioned_docs/version-3.X/images/software_stack.png new file mode 100644 index 00000000..75fba6e6 Binary files /dev/null and b/versioned_docs/version-3.X/images/software_stack.png differ diff --git a/versioned_docs/version-3.X/images/two_phase_commit_load_balancing.png b/versioned_docs/version-3.X/images/two_phase_commit_load_balancing.png new file mode 100644 index 00000000..5cdc26f0 Binary files /dev/null and b/versioned_docs/version-3.X/images/two_phase_commit_load_balancing.png differ diff --git a/versioned_docs/version-3.X/images/two_phase_commit_sequence_diagram.png b/versioned_docs/version-3.X/images/two_phase_commit_sequence_diagram.png new file mode 100644 index 00000000..116ef635 Binary files /dev/null and b/versioned_docs/version-3.X/images/two_phase_commit_sequence_diagram.png differ diff --git a/versioned_docs/version-3.X/manage-backup-and-restore.mdx b/versioned_docs/version-3.X/manage-backup-and-restore.mdx new file mode 100644 index 00000000..a89dea81 --- /dev/null +++ b/versioned_docs/version-3.X/manage-backup-and-restore.mdx @@ -0,0 +1,23 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Back Up and Restore Databases + +This guide explains how to back up and restore databases that are used by ScalarDB. + +## Basic guidelines to back up and restore databases + +Before performing a backup, be sure to read [How to Back Up and Restore Databases Used Through ScalarDB](backup-restore.mdx). + +## Back up databases when using ScalarDB in a Kubernetes environment + +For details on how to back up databases in a Kubernetes environment, see [Back up a NoSQL database in a Kubernetes environment](scalar-kubernetes/BackupNoSQL.mdx). + +## Restore databases when using ScalarDB in a Kubernetes environment + +For details on how to restore databases in a Kubernetes environment, see [Restore databases in a Kubernetes environment](scalar-kubernetes/RestoreDatabase.mdx). diff --git a/versioned_docs/version-3.X/manage-monitor-overview.mdx b/versioned_docs/version-3.X/manage-monitor-overview.mdx new file mode 100644 index 00000000..d120478c --- /dev/null +++ b/versioned_docs/version-3.X/manage-monitor-overview.mdx @@ -0,0 +1,21 @@ +--- +tags: + - Enterprise Option +displayed_sidebar: docsEnglish +--- + +# Monitor Overview + +Scalar Manager is a centralized management and monitoring solution for ScalarDB within Kubernetes cluster environments that allows you to: + +- Check the availability of ScalarDB. +- Schedule or execute pausing jobs that create transactionally consistent periods in the databases used by ScalarDB. +- Check the time-series metrics and logs of ScalarDB through Grafana dashboards. + +For more details about Scalar Manager, see [Scalar Manager Overview](scalar-manager/overview.mdx). + +## Deploy Scalar Manager + +You can deploy Scalar Manager by using a Helm Chart. + +For details on how to deploy Scalar Manager, see [Deploy Scalar Manager](helm-charts/getting-started-scalar-manager.mdx). diff --git a/versioned_docs/version-3.X/manage-overview.mdx b/versioned_docs/version-3.X/manage-overview.mdx new file mode 100644 index 00000000..a58fdea0 --- /dev/null +++ b/versioned_docs/version-3.X/manage-overview.mdx @@ -0,0 +1,26 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Manage Overview + +In this category, you can follow guides to help you manage ScalarDB. + +- For details on how to scale ScalarDB, see [Scale](scalar-kubernetes/HowToScaleScalarDB.mdx). +- For details on how to upgrade ScalarDB, see [Upgrade](scalar-kubernetes/HowToUpgradeScalarDB.mdx). + +## Monitor + +In this sub-category, you can learn how to monitor your ScalarDB deployment. + +For an overview of this sub-category, see [Monitor Overview](manage-monitor-overview.mdx). + +## Back up and restore + +In this sub-category, you can learn how to back up and restore the databases that are connected to your ScalarDB deployment. + +For an overview of this sub-category, see [Back Up and Restore Databases](manage-backup-and-restore.mdx). diff --git a/versioned_docs/version-3.X/migrate-overview.mdx b/versioned_docs/version-3.X/migrate-overview.mdx new file mode 100644 index 00000000..5b67453f --- /dev/null +++ b/versioned_docs/version-3.X/migrate-overview.mdx @@ -0,0 +1,14 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Migrate Overview + +For details on importing your tables or migrating your applications and databases to a ScalarDB-based environment, see the following guides: + +- [Importing Existing Tables to ScalarDB by Using ScalarDB Schema Loader](schema-loader-import.mdx) +- [How to Migrate Your Applications and Databases into a ScalarDB-Based Environment](scalardb-sql/migration-guide.mdx) diff --git a/versioned_docs/version-3.X/multi-storage-transactions.mdx b/versioned_docs/version-3.X/multi-storage-transactions.mdx new file mode 100644 index 00000000..de8da288 --- /dev/null +++ b/versioned_docs/version-3.X/multi-storage-transactions.mdx @@ -0,0 +1,68 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Multi-Storage Transactions + +ScalarDB transactions can span multiple storages or databases while maintaining ACID compliance by using a feature called *multi-storage transactions*. + +This page explains how multi-storage transactions work and how to configure the feature in ScalarDB. + +## How multi-storage transactions work in ScalarDB + +In ScalarDB, the `multi-storage` implementation holds multiple storage instances and has mappings from a namespace name to a proper storage instance. When an operation is executed, the multi-storage transactions feature chooses a proper storage instance from the specified namespace by using the namespace-storage mapping and uses that storage instance. + +## How to configure ScalarDB to support multi-storage transactions + +To enable multi-storage transactions, you need to specify `consensus-commit` as the value for `scalar.db.transaction_manager`, `multi-storage` as the value for `scalar.db.storage`, and configure your databases in the ScalarDB properties file. + +The following is an example of configurations for multi-storage transactions: + +```properties +# Consensus Commit is required to support multi-storage transactions. +scalar.db.transaction_manager=consensus-commit + +# Multi-storage implementation is used for Consensus Commit. +scalar.db.storage=multi-storage + +# Define storage names by using a comma-separated format. +# In this case, "cassandra" and "mysql" are used. +scalar.db.multi_storage.storages=cassandra,mysql + +# Define the "cassandra" storage. +# When setting storage properties, such as `storage`, `contact_points`, `username`, and `password`, for multi-storage transactions, the format is `scalar.db.multi_storage.storages..`. +# For example, to configure the `scalar.db.contact_points` property for Cassandra, specify `scalar.db.multi_storage.storages.cassandra.contact_point`. +scalar.db.multi_storage.storages.cassandra.storage=cassandra +scalar.db.multi_storage.storages.cassandra.contact_points=localhost +scalar.db.multi_storage.storages.cassandra.username=cassandra +scalar.db.multi_storage.storages.cassandra.password=cassandra + +# Define the "mysql" storage. +# When defining JDBC-specific configurations for multi-storage transactions, you can follow a similar format of `scalar.db.multi_storage.storages..`. +# For example, to configure the `scalar.db.jdbc.connection_pool.min_idle` property for MySQL, specify `scalar.db.multi_storage.storages.mysql.jdbc.connection_pool.min_idle`. +scalar.db.multi_storage.storages.mysql.storage=jdbc +scalar.db.multi_storage.storages.mysql.contact_points=jdbc:mysql://localhost:3306/ +scalar.db.multi_storage.storages.mysql.username=root +scalar.db.multi_storage.storages.mysql.password=mysql +# Define the JDBC-specific configurations for the "mysql" storage. +scalar.db.multi_storage.storages.mysql.jdbc.connection_pool.min_idle=5 +scalar.db.multi_storage.storages.mysql.jdbc.connection_pool.max_idle=10 +scalar.db.multi_storage.storages.mysql.jdbc.connection_pool.max_total=25 + +# Define namespace mapping from a namespace name to a storage. +# The format is ":,...". +scalar.db.multi_storage.namespace_mapping=user:cassandra,coordinator:mysql + +# Define the default storage that's used if a specified table doesn't have any mapping. +scalar.db.multi_storage.default_storage=cassandra +``` + +For additional configurations, see [ScalarDB Configurations](configurations.mdx). + +## Hands-on tutorial + +For a hands-on tutorial, see [Create a Sample Application That Supports Multi-Storage Transactions](scalardb-samples/multi-storage-transaction-sample/README.mdx). diff --git a/versioned_docs/version-3.X/overview.mdx b/versioned_docs/version-3.X/overview.mdx new file mode 100644 index 00000000..b51df6be --- /dev/null +++ b/versioned_docs/version-3.X/overview.mdx @@ -0,0 +1,77 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# ScalarDB Overview + +This page describes what ScalarDB is and its primary use cases. + +## What is ScalarDB? + +ScalarDB is a universal hybrid transaction/analytical processing (HTAP) engine for diverse databases. It runs as middleware on databases and virtually unifies diverse databases by achieving ACID transactions and real-time analytics across them to simplify the complexity of managing multiple databases or multiple instances of a single database. + +![How ScalarDB simplifies complex data management architecture.](images/scalardb.png) + +As a versatile solution, ScalarDB supports a range of databases, including: + +- Relational databases that support JDBC, such as IBM Db2, MariaDB, Microsoft SQL Server, MySQL, Oracle Database, PostgreSQL, SQLite, and their compatible databases, like Amazon Aurora and YugabyteDB. +- NoSQL databases like Amazon DynamoDB, Apache Cassandra, and Azure Cosmos DB. + +For details on which databases ScalarDB supports, refer to [Databases](requirements.mdx#databases). + +## Why ScalarDB? + +Several solutions, such as global transaction managers, data federation engines, and HTAP systems, have similar goals, but they are limited in the following perspectives: + +- Global transaction managers (like Oracle MicroTx and Atomikos) are designed to run transactions across a limited set of heterogeneous databases (like only XA-compliant databases). +- Data federation engines (like Denodo and Starburst) are designed to run analytical queries across heterogeneous databases. +- HTAP systems (like TiDB and SingleStore) run both transactions and analytical queries only on homogeneous databases. + +In other words, they virtually unify databases, but with limitations. For example, with data federation engines, users can run read-only analytical queries on a virtualized view across multiple databases. However, they often need to run update queries separately for each database. + +Unlike other solutions, ScalarDB stands out by offering the ability to run both transactional and analytical queries on heterogeneous databases, which can significantly simplify database management. + +The following table summarizes how ScalarDB is different from the other solutions. + +| | Transactions across heterogeneous databases | Analytics across heterogeneous databases | +| :------------------------------------------------------------: | :------------------------------------------------------------------: | :--------------------------------------: | +| Global transaction managers (like Oracle MicroTx and Atomikos) | Yes (but existing solutions support only a limited set of databases) | No | +| Data federation engines (like Denodo and Starburst) | No | Yes | +| HTAP systems (like TiDB and SingleStore) | No (support homogeneous databases only) | No (support homogeneous databases only) | +| **ScalarDB** | **Yes (supports various databases)** | **Yes** | + + +## ScalarDB use cases + +ScalarDB can be used in various ways. Here are the three primary use cases of ScalarDB. + +### Managing siloed databases easily +Many enterprises comprise several organizations, departments, and business units to support agile business operations, which often leads to siloed information systems. In particular, different organizations likely manage different applications with different databases. Managing such siloed databases is challenging because applications must communicate with each database separately and properly deal with the differences between databases. + +ScalarDB simplifies the management of siloed databases with a unified interface, enabling users to treat the databases as if they were a single database. For example, users can run (analytical) join queries over multiple databases without interacting with the databases respectively. + +### Managing consistency between multiple database +Modern architectures, like the microservice architecture, encourage a system to separate a service and its database into smaller subsets to increase system modularity and development efficiency. However, managing diverse databases, especially of different kinds, is challenging because applications must ensure the correct states (or, in other words, consistencies) of those databases, even using transaction management patterns like Saga and TCC. + +ScalarDB simplifies managing such diverse databases with a correctness guarantee (or, in other words, ACID with strict serializability), enabling you to focus on application development without worrying about guaranteeing consistency between databases. + +### Simplifying data management in a data mesh + +Enterprises have been investing their time in building [data meshes](https://martinfowler.com/articles/data-mesh-principles.html) to streamline and scale data utilization. However, constructing a data mesh is not necessarily easy. For example, there are many technical issues in how to manage decentralized data. + +ScalarDB simplifies the management of decentralized databases in a data mesh, for example, by providing a unified API for all the databases in a data mesh to align with the data-as-a-product principle easily. + +### Reducing database migration hurdles + +Applications tend to be locked into using a certain database because of the specific capabilities that the database provides. Such database lock-in discourages upgrading or changing the database because doing so often requires rewriting the application. + +ScalarDB provides a unified interface for diverse databases. Thus, once an application is written by using the ScalarDB interface, it becomes portable, which helps to achieve seamless database migration without rewriting the application. + +## Further reading + +- [ScalarDB Technical Overview](https://speakerdeck.com/scalar/scalar-db-universal-transaction-manager) +- [ScalarDB Research Paper [VLDB'23]](https://dl.acm.org/doi/10.14778/3611540.3611563) \ No newline at end of file diff --git a/versioned_docs/version-3.X/quickstart-overview.mdx b/versioned_docs/version-3.X/quickstart-overview.mdx new file mode 100644 index 00000000..50647c79 --- /dev/null +++ b/versioned_docs/version-3.X/quickstart-overview.mdx @@ -0,0 +1,41 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Quickstart Overview + +In this category, you can follow quickstart tutorials for how to get started with running transactions and queries through ScalarDB. + +## Try running transactions through the ScalarDB Core library + +In this sub-category, you can follow tutorials on how to run ACID transactions through the ScalarDB Core library, which is publicly available under the Apache 2 License. + +For an overview of this sub-category, see [ScalarDB Core Quickstart Overview](quickstart-scalardb-core-overview.mdx). + +## Try running transactions through ScalarDB Cluster + +In this sub-category, you can see tutorials on how to run ACID transactions through ScalarDB Cluster, which is a [gRPC](https://grpc.io/) server that wraps the ScalarDB Core library. + +For an overview of this sub-category, see [ScalarDB Cluster Quickstart Overview](quickstart-scalardb-cluster-overview.mdx). + +:::note + +ScalarDB Cluster is available only in the Enterprise edition. + +::: + +## Try running analytical queries through ScalarDB Analytics + +In this sub-category, you can see tutorials on how to run analytical queries over the databases that you write through ScalarDB by using a component called ScalarDB Analytics. ScalarDB Analytics targets both ScalarDB-managed databases, which are updated through ScalarDB transactions, and non-ScalarDB-managed databases. + +For an overview of this sub-category, see [ScalarDB Analytics Quickstart Overview](quickstart-scalardb-analytics-overview.mdx). + +:::note + +- ScalarDB Analytics with PostgreSQL is available only under the Apache 2 License and doesn't require a commercial license. + +::: diff --git a/versioned_docs/version-3.X/quickstart-scalardb-analytics-overview.mdx b/versioned_docs/version-3.X/quickstart-scalardb-analytics-overview.mdx new file mode 100644 index 00000000..6b3bc3ec --- /dev/null +++ b/versioned_docs/version-3.X/quickstart-scalardb-analytics-overview.mdx @@ -0,0 +1,13 @@ +--- +tags: + - Community + - Enterprise Option +displayed_sidebar: docsEnglish +--- + +# ScalarDB Analytics Quickstart Overview + +In this sub-category, you can see tutorials on how to run analytical queries over the databases that you write through ScalarDB by using a component called ScalarDB Analytics. + +- To try running analytical queries through PostgreSQL, see [Getting Started with ScalarDB Analytics with PostgreSQL](scalardb-analytics-postgresql/getting-started.mdx). +- To try running analytical queries through Spark, see [Getting Started with ScalarDB Analytics](scalardb-samples/scalardb-analytics-spark-sample/README.mdx). diff --git a/versioned_docs/version-3.X/quickstart-scalardb-cluster-overview.mdx b/versioned_docs/version-3.X/quickstart-scalardb-cluster-overview.mdx new file mode 100644 index 00000000..6d5538ca --- /dev/null +++ b/versioned_docs/version-3.X/quickstart-scalardb-cluster-overview.mdx @@ -0,0 +1,15 @@ +--- +tags: + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# ScalarDB Cluster Quickstart Overview + +In this sub-category, you can see tutorials on how to run ACID transactions through ScalarDB Cluster, which is a [gRPC](https://grpc.io/) server that wraps the ScalarDB Core library. + +- To try running transactions, see [Getting Started with ScalarDB Cluster](scalardb-cluster/getting-started-with-scalardb-cluster.mdx). +- To try running transactions through the SQL interface via JDBC, see [Getting Started with ScalarDB Cluster SQL via JDBC](scalardb-cluster/getting-started-with-scalardb-cluster-sql-jdbc.mdx). +- To try running transactions through the SQL interface via Spring Data JDBC, see [Getting Started with ScalarDB Cluster SQL via Spring Data JDBC for ScalarDB](scalardb-cluster/getting-started-with-scalardb-cluster-sql-spring-data-jdbc.mdx). +- To try running transactions through the GraphQL interface, see [Getting Started with ScalarDB Cluster GraphQL](scalardb-cluster/getting-started-with-scalardb-cluster-graphql.mdx). diff --git a/versioned_docs/version-3.X/quickstart-scalardb-core-overview.mdx b/versioned_docs/version-3.X/quickstart-scalardb-core-overview.mdx new file mode 100644 index 00000000..36a59c55 --- /dev/null +++ b/versioned_docs/version-3.X/quickstart-scalardb-core-overview.mdx @@ -0,0 +1,14 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# ScalarDB Core Quickstart Overview + +In this sub-category, you can follow tutorials on how to run ACID transactions through the ScalarDB Core library, which is publicly available under the Apache 2 license. + +- To try running transactions, see [Getting Started with ScalarDB](getting-started-with-scalardb.mdx). +- To try running transactions by using Kotlin, see [Getting Started with ScalarDB by Using Kotlin](getting-started-with-scalardb-by-using-kotlin.mdx). diff --git a/versioned_docs/version-3.X/requirements.mdx b/versioned_docs/version-3.X/requirements.mdx new file mode 100644 index 00000000..a5fdb19b --- /dev/null +++ b/versioned_docs/version-3.X/requirements.mdx @@ -0,0 +1,288 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Requirements + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +This page describes the required tools and their versions to use ScalarDB correctly. + +## Client SDK + +Because ScalarDB is written in Java, the easiest way to interact with ScalarDB is to use the Java client SDKs: + +- [SDK for ScalarDB Core](add-scalardb-to-your-build.mdx) +- [SDK for ScalarDB Cluster](scalardb-cluster/developer-guide-for-scalardb-cluster-with-java-api.mdx) + +### Java + +The following Java Development Kits (JDKs) are verified and supported. + +- **[Oracle JDK](https://www.oracle.com/java/):** 8, 11, 17 or 21 (LTS versions) +- **[OpenJDK](https://openjdk.org/) ([Eclipse Temurin](https://adoptium.net/temurin/), [Amazon Corretto](https://aws.amazon.com/corretto/), or [Microsoft Build of OpenJDK](https://learn.microsoft.com/en-us/java/openjdk/)):** 8, 11, 17, or 21 (LTS versions) + +### .NET + +ScalarDB is provided as a gRPC server called ScalarDB Cluster, which also has a [.NET client SDK](scalardb-cluster-dotnet-client-sdk/index.mdx) that wraps the .NET client generated from the proto file. + +The following .NET versions are verified and supported: + +- [.NET 8.0](https://dotnet.microsoft.com/en-us/download/dotnet/8.0) +- [.NET 6.0](https://dotnet.microsoft.com/en-us/download/dotnet/6.0) + +### Other languages + +ScalarDB Cluster uses gRPC version 1.65.0, so you can create your own client by using the generated clients of your preferred languages. + +## Databases + +ScalarDB is middleware that runs on top of the following databases and their versions. + +### Relational databases + + + + +| Version | Oracle Database 23ai | Oracle Database 21c | Oracle Database 19c | +|:------------------|:--------------------|:------------------|:------------------| +| **ScalarDB 3.16** | ✅ | ✅ | ✅ | +| **ScalarDB 3.15** | ✅ | ✅ | ✅ | +| **ScalarDB 3.14** | ✅ | ✅ | ✅ | +| **ScalarDB 3.13** | ✅ | ✅ | ✅ | +| **ScalarDB 3.12** | ✅ | ✅ | ✅ | +| **ScalarDB 3.11** | ✅ | ✅ | ✅ | +| **ScalarDB 3.10** | ✅ | ✅ | ✅ | +| **ScalarDB 3.9** | ✅ | ✅ | ✅ | +| **ScalarDB 3.8** | ✅ | ✅ | ✅ | +| **ScalarDB 3.7** | ✅ | ✅ | ✅ | + + + + +| Version | Db2 12.1 | Db2 11.5 | +|:------------------|:---------|:---------| +| **ScalarDB 3.16** | ✅ | ✅ | +| **ScalarDB 3.15** | ❌ | ❌ | +| **ScalarDB 3.14** | ❌ | ❌ | +| **ScalarDB 3.13** | ❌ | ❌ | +| **ScalarDB 3.12** | ❌ | ❌ | +| **ScalarDB 3.11** | ❌ | ❌ | +| **ScalarDB 3.10** | ❌ | ❌ | +| **ScalarDB 3.9** | ❌ | ❌ | +| **ScalarDB 3.8** | ❌ | ❌ | +| **ScalarDB 3.7** | ❌ | ❌ | + +:::note + +Only Linux, UNIX, and Windows versions of Db2 are supported. The z/OS version is not currently supported. + +::: + + + + +| Version | MySQL 8.4 | MySQL 8.0 | +|:------------------|:----------|:-----------| +| **ScalarDB 3.16** | ✅ | ✅ | +| **ScalarDB 3.15** | ✅ | ✅ | +| **ScalarDB 3.14** | ✅ | ✅ | +| **ScalarDB 3.13** | ✅ | ✅ | +| **ScalarDB 3.12** | ✅ | ✅ | +| **ScalarDB 3.11** | ✅ | ✅ | +| **ScalarDB 3.10** | ✅ | ✅ | +| **ScalarDB 3.9** | ✅ | ✅ | +| **ScalarDB 3.8** | ✅ | ✅ | +| **ScalarDB 3.7** | ✅ | ✅ | + + + + +| Version | PostgreSQL 17 | PostgreSQL 16 | PostgreSQL 15 | PostgreSQL 14 | PostgreSQL 13 | +|:------------------|:--------------|:--------------|:--------------|:--------------|---------------| +| **ScalarDB 3.16** | ✅ | ✅ | ✅ | ✅ | ✅ | +| **ScalarDB 3.15** | ✅ | ✅ | ✅ | ✅ | ✅ | +| **ScalarDB 3.14** | ✅ | ✅ | ✅ | ✅ | ✅ | +| **ScalarDB 3.13** | ✅ | ✅ | ✅ | ✅ | ✅ | +| **ScalarDB 3.12** | ✅ | ✅ | ✅ | ✅ | ✅ | +| **ScalarDB 3.11** | ✅ | ✅ | ✅ | ✅ | ✅ | +| **ScalarDB 3.10** | ✅ | ✅ | ✅ | ✅ | ✅ | +| **ScalarDB 3.9** | ✅ | ✅ | ✅ | ✅ | ✅ | +| **ScalarDB 3.8** | ✅ | ✅ | ✅ | ✅ | ✅ | +| **ScalarDB 3.7** | ✅ | ✅ | ✅ | ✅ | ✅ | + + + + +| Version | Aurora MySQL 3 | Aurora MySQL 2 | +|:------------------|:----------------|:----------------| +| **ScalarDB 3.16** | ✅ | ✅ | +| **ScalarDB 3.15** | ✅ | ✅ | +| **ScalarDB 3.14** | ✅ | ✅ | +| **ScalarDB 3.13** | ✅ | ✅ | +| **ScalarDB 3.12** | ✅ | ✅ | +| **ScalarDB 3.11** | ✅ | ✅ | +| **ScalarDB 3.10** | ✅ | ✅ | +| **ScalarDB 3.9** | ✅ | ✅ | +| **ScalarDB 3.8** | ✅ | ✅ | +| **ScalarDB 3.7** | ✅ | ✅ | + + + + +| Version | Aurora PostgreSQL 16 | Aurora PostgreSQL 15 | Aurora PostgreSQL 14 | Aurora PostgreSQL 13 | +|:------------------|:---------------------|:---------------------|:---------------------|:---------------------| +| **ScalarDB 3.16** | ✅ | ✅ | ✅ | ✅ | +| **ScalarDB 3.15** | ✅ | ✅ | ✅ | ✅ | +| **ScalarDB 3.14** | ✅ | ✅ | ✅ | ✅ | +| **ScalarDB 3.13** | ✅ | ✅ | ✅ | ✅ | +| **ScalarDB 3.12** | ✅ | ✅ | ✅ | ✅ | +| **ScalarDB 3.11** | ✅ | ✅ | ✅ | ✅ | +| **ScalarDB 3.10** | ✅ | ✅ | ✅ | ✅ | +| **ScalarDB 3.9** | ✅ | ✅ | ✅ | ✅ | +| **ScalarDB 3.8** | ✅ | ✅ | ✅ | ✅ | +| **ScalarDB 3.7** | ✅ | ✅ | ✅ | ✅ | + + + + +| Version | MariaDB 11.4 | MariaDB 10.11 | +|:------------------|:--------------|:--------------| +| **ScalarDB 3.16** | ✅ | ✅ | +| **ScalarDB 3.15** | ✅ | ✅ | +| **ScalarDB 3.14** | ✅ | ✅ | +| **ScalarDB 3.13** | ✅ | ✅ | +| **ScalarDB 3.12** | ✅ | ✅ | +| **ScalarDB 3.11** | ✅ | ✅ | +| **ScalarDB 3.10** | ✅ | ✅ | +| **ScalarDB 3.9** | ✅ | ✅ | +| **ScalarDB 3.8** | ✅ | ✅ | +| **ScalarDB 3.7** | ✅ | ✅ | + + + + +| Version | SQL Server 2022 | SQL Server 2019 | SQL Server 2017 | +|:------------------|:-----------------|:-----------------|:-----------------| +| **ScalarDB 3.16** | ✅ | ✅ | ✅ | +| **ScalarDB 3.15** | ✅ | ✅ | ✅ | +| **ScalarDB 3.14** | ✅ | ✅ | ✅ | +| **ScalarDB 3.13** | ✅ | ✅ | ✅ | +| **ScalarDB 3.12** | ✅ | ✅ | ✅ | +| **ScalarDB 3.11** | ✅ | ✅ | ✅ | +| **ScalarDB 3.10** | ✅ | ✅ | ✅ | +| **ScalarDB 3.9** | ✅ | ✅ | ✅ | +| **ScalarDB 3.8** | ✅ | ✅ | ✅ | +| **ScalarDB 3.7** | ✅ | ✅ | ✅ | + + + + +| Version | SQLite 3 | +|:------------------|:----------| +| **ScalarDB 3.16** | ✅ | +| **ScalarDB 3.15** | ✅ | +| **ScalarDB 3.14** | ✅ | +| **ScalarDB 3.13** | ✅ | +| **ScalarDB 3.12** | ✅ | +| **ScalarDB 3.11** | ✅ | +| **ScalarDB 3.10** | ✅ | +| **ScalarDB 3.9** | ✅ | +| **ScalarDB 3.8** | ❌ | +| **ScalarDB 3.7** | ❌ | + + + + +| Version | YugabyteDB 2 | +|:------------------|:-------------| +| **ScalarDB 3.16** | ✅ | +| **ScalarDB 3.15** | ✅ | +| **ScalarDB 3.14** | ✅ | +| **ScalarDB 3.13** | ✅ | +| **ScalarDB 3.12** | ❌ | +| **ScalarDB 3.11** | ❌ | +| **ScalarDB 3.10** | ❌ | +| **ScalarDB 3.9** | ❌ | +| **ScalarDB 3.8** | ❌ | +| **ScalarDB 3.7** | ❌ | + + + + +### NoSQL databases + + + + +| Version | DynamoDB | +|:------------------|:----------| +| **ScalarDB 3.16** | ✅ | +| **ScalarDB 3.15** | ✅ | +| **ScalarDB 3.14** | ✅ | +| **ScalarDB 3.13** | ✅ | +| **ScalarDB 3.12** | ✅ | +| **ScalarDB 3.11** | ✅ | +| **ScalarDB 3.10** | ✅ | +| **ScalarDB 3.9** | ✅ | +| **ScalarDB 3.8** | ✅ | +| **ScalarDB 3.7** | ✅ | + + + + +| Version | Cassandra 4.1 | Cassandra 4.0 | Cassandra 3.11 | Cassandra 3.0 | +|:------------------|:---------------|:---------------|:----------------|:---------------| +| **ScalarDB 3.16** | ❌ | ❌ | ✅ | ✅ | +| **ScalarDB 3.15** | ❌ | ❌ | ✅ | ✅ | +| **ScalarDB 3.14** | ❌ | ❌ | ✅ | ✅ | +| **ScalarDB 3.13** | ❌ | ❌ | ✅ | ✅ | +| **ScalarDB 3.12** | ❌ | ❌ | ✅ | ✅ | +| **ScalarDB 3.11** | ❌ | ❌ | ✅ | ✅ | +| **ScalarDB 3.10** | ❌ | ❌ | ✅ | ✅ | +| **ScalarDB 3.9** | ❌ | ❌ | ✅ | ✅ | +| **ScalarDB 3.8** | ❌ | ❌ | ✅ | ✅ | +| **ScalarDB 3.7** | ❌ | ❌ | ✅ | ✅ | + + + + +| Version | Cosmos DB for NoSQL | +|:------------------|:---------------------| +| **ScalarDB 3.16** | ✅ | +| **ScalarDB 3.15** | ✅ | +| **ScalarDB 3.14** | ✅ | +| **ScalarDB 3.13** | ✅ | +| **ScalarDB 3.12** | ✅ | +| **ScalarDB 3.11** | ✅ | +| **ScalarDB 3.10** | ✅ | +| **ScalarDB 3.9** | ✅ | +| **ScalarDB 3.8** | ✅ | +| **ScalarDB 3.7** | ✅ | + + + + +:::note + +For details on how to configure each database, see [Configurations for the Underlying Databases of ScalarDB](./database-configurations.mdx). + +::: + +## Kubernetes + +ScalarDB is provided as a Pod on the Kubernetes platform in production environments. ScalarDB supports the following platforms and tools. + +### Platform +- **[Kubernetes](https://kubernetes.io/):** 1.28 - 1.32 + - **[Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/)** + - **[Azure Kubernetes Service (AKS)](https://azure.microsoft.com/en-us/products/kubernetes-service)** +- **[Red Hat OpenShift](https://www.redhat.com/en/technologies/cloud-computing/openshift):** TBD + +### Package manager +- **[Helm](https://helm.sh/):** 3.5+ diff --git a/versioned_docs/version-3.X/roadmap.mdx b/versioned_docs/version-3.X/roadmap.mdx new file mode 100644 index 00000000..b195d884 --- /dev/null +++ b/versioned_docs/version-3.X/roadmap.mdx @@ -0,0 +1,135 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# ScalarDB Roadmap + +This roadmap provides a look into the proposed future of ScalarDB. The purpose of this roadmap is to provide visibility into what changes may be coming so that you can more closely follow progress, learn about key milestones, and give feedback during development. This roadmap will be updated as new versions of ScalarDB are released. + +:::warning + +During the course of development, this roadmap is subject to change based on user needs and feedback. **Do not schedule your release plans according to the contents of this roadmap.** + +If you have a feature request or want to prioritize feature development, please create an issue in [GitHub](https://github.com/scalar-labs/scalardb/issues). + +::: + + +### CY2025 Q2 + + +#### Support for additional databases + +- **IBM Db2** + - Users will be able to use IBM Db2 as an underlying database through ScalarDB Cluster. +- **TiDB** + - Users will be able to use TiDB as an underlying database through ScalarDB Cluster. +- **Databricks** + - Users will be able to use Databricks as an underlying database through ScalarDB Cluster and ScalarDB Analytics. +- **Snowflake** + - Users will be able to use Snowflake as an underlying database through ScalarDB Cluster and ScalarDB Analytics. + +#### Usability + +- **Addition of decimal data types** + - Users will be able to use decimal data types so that users can handle decimal numbers with high precision. +- **Removal of extra-write strategy** + - Users will no longer be able to use the extra-write strategy to make transactions serializable. Although ScalarDB currently provides two strategies (extra-read and extra-write strategies) to make transactions serializable, the extra-write strategy has several limitations. For example, users can't issue write and scan operations in the same transaction. Therefore, the strategy will be removed so that users don't need to worry about such limitations when creating applications. +- **Better governance in ScalarDB Analytics** + - Users will be able to be authenticated and authorized by using the ScalarDB Core features. + +#### Performance + +- **Addition of read-committed isolation** + - Users will be able to run transactions with a read-committed isolation to achieve better performance for applications that do not require strong correctness. +- **One-phase commit optimization** + - Users will be able to run a transaction more efficiently by using one-phase commit if the operations of the transaction are all applied to a single database or a single partition. +- **Optimization for multiple write operations per database** + - Users will be able to run transactions more efficiently with a batch preparation and commitment if there are multiple write operations for a database. +- **Optimization for read-only transactions** + - Users will be able to run transactions more efficiently by avoiding coordinator writes when committing transactions. +- **Removal of WAL-interpreted views in ScalarDB Analytics** + - Users will be able to read committed data by using ScalarDB Core instead of WAL-interpreted views, which will increase query performance. + +#### Cloud support + +- **Container offering in Azure Marketplace for ScalarDB Cluster** + - Users will be able to deploy ScalarDB Cluster by using the Azure container offering, which enables users to use a pay-as-you-go subscription model. +- **Google Cloud Platform (GCP) support for ScalarDB Cluster** + - Users will be able to deploy ScalarDB Cluster in Google Kubernetes Engine (GKE) in GCP. +- **Container offering in Amazon Marketplace for ScalarDB Analytics** + - Users will be able to deploy ScalarDB Analytics by using the container offering, which enables users to use a pay-as-you-go subscription model. + +### CY2025 Q3 + +#### New capabilities + +- **Decoupled metadata management** + - Users will be able to start using ScalarDB Cluster without migrating or changing the schemas of existing applications by managing the transaction metadata of ScalarDB in a separate location. + +#### Usability + +- **Views** + - Users will be able to define views so that they can manage multiple different databases in an easier and simplified way. +- **Addition of SQL operations for aggregation** + - Users will be able to issue aggregation operations in ScalarDB SQL. +- **Elimination of out-of-memory errors due to large scans** + - Users will be able to issue large scans without experiencing out-of-memory errors. +- **Enabling of read operations during a paused duration** + - Users will be able to issue read operations even during a paused duration so that users can still read data while taking backups. + +#### Scalability and availability + +- **Semi-synchronous replication** + - Users will be able to replicate the data of ScalarDB-based applications in a disaster-recoverable manner. For example, assume you provide a primary service in Tokyo and a standby service in Osaka. In case of catastrophic failure in Tokyo, you can switch the primary service to Osaka so that you can continue to provide the service without data loss and extended downtime. + +### CY2025 Q4 + +#### New capabilities + +- **Native secondary index** + - Users will be able to define flexible secondary indexes. The existing secondary index is limited because it is implemented based on the common capabilities of the supported databases' secondary indexes. Therefore, for example, you cannot define a multi-column index. The new secondary index will be created at the ScalarDB layer so that you can create more flexible indexes, like a multi-column index. +- **Universal catalog** + - Users will be able to manage metadata, including schemas and semantic information, for operational and analytical databases across separate business domains in a unified manner. +- **Universal authentication and authorization** + - Users will be able to be given access to ScalarDB Cluster and ScalarDB Analytics by using a unified authentication and authorization method. + +#### Support for additional databases (object storage) + +- **Azure Blob Storage** + - Users will be able to use Azure Blob Storage as an underlying database through ScalarDB Cluster. +- **Amazon S3** + - Users will be able to use Amazon S3 as an underlying database through ScalarDB Cluster. +- **Google Cloud Storage** + - Users will be able to use Google Cloud Storage as an underlying database through ScalarDB Cluster and ScalarDB Analytics. + +#### Performance + +- **Reduction of storage space needed for managing ScalarDB metadata** + - Users will likely use less storage space to run ScalarDB. ScalarDB will remove the before image of committed transactions after they are committed. However, whether or not those committed transactions will impact actual storage space depends on the underlying databases. + +#### Cloud support + +- **Red Hat OpenShift support** + - Users will be able to use Red Hat–certified Helm Charts for ScalarDB Cluster in OpenShift environments. +- **Container offering in Google Cloud Marketplace** + - Users will be able to deploy ScalarDB Cluster by using the Google Cloud container offering, which enables users to use a pay-as-you-go subscription model. + +### CY2026 + +- **Audit logging** + - Users will be able to view and manage the access logs of ScalarDB Cluster and Analytics, mainly for auditing purposes. +- **Stored procedures** + - Users will be able to define stored procedures so that they can execute a set of operations with a complex logic inside ScalarDB Cluster. +- **Triggers** + - Users will be able to define triggers so that they can automatically execute a set of operations when a specific event occurs in ScalarDB Cluster. +- **User-defined functions (UDFs)** + - Users will be able to define functions so that they can use functions in SQLs to express complex logic in a simpler way. +- **Addition of SQL operations for sorting** + - Users will be able to issue arbitrary sorting (ORDER BY) operations in ScalarDB SQL for multiple or non-JDBC databases. (Currently, ScalarDB can issue sorting operations using clustering keys or arbitrary sorting operations for single JDBC databases.) +- **Addition of more data types** + - Users will be able to use complex data types, such as JSON. \ No newline at end of file diff --git a/versioned_docs/version-3.X/run-non-transactional-storage-operations-through-library.mdx b/versioned_docs/version-3.X/run-non-transactional-storage-operations-through-library.mdx new file mode 100644 index 00000000..ea97ee58 --- /dev/null +++ b/versioned_docs/version-3.X/run-non-transactional-storage-operations-through-library.mdx @@ -0,0 +1,295 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Run Non-Transactional Storage Operations Through the Core Library + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +This guide explains how to run non-transactional storage operations through the ScalarDB Core library. + +## Preparation + +For the purpose of this guide, you will set up a database and ScalarDB by using a sample in the ScalarDB samples repository. + +### Clone the ScalarDB samples repository + +Open **Terminal**, then clone the ScalarDB samples repository by running the following command: + +```console +git clone https://github.com/scalar-labs/scalardb-samples +``` + +Then, go to the directory that contains the necessary files by running the following command: + +```console +cd scalardb-samples/scalardb-sample +``` + +## Set up a database + +Select your database, and follow the instructions to configure it for ScalarDB. + +For a list of databases that ScalarDB supports, see [Databases](requirements.mdx#databases). + + + +

Run MySQL locally

+ + You can run MySQL in Docker Compose by using the `docker-compose.yml` file in the `scalardb-samples/scalardb-sample` directory. + + To start MySQL, run the following command: + + ```console + docker compose up -d mysql + ``` + +

Configure ScalarDB

+ + The **database.properties** file in the `scalardb-samples/scalardb-sample` directory contains database configurations for ScalarDB. Please uncomment the properties for MySQL in the **database.properties** file so that the configuration looks as follows: + + ```properties + # For MySQL + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:mysql://localhost:3306/ + scalar.db.username=root + scalar.db.password=mysql + ``` + + +

Run PostgreSQL locally

+ + You can run PostgreSQL in Docker Compose by using the `docker-compose.yml` file in the `scalardb-samples/scalardb-sample` directory. + + To start PostgreSQL, run the following command: + + ```console + docker compose up -d postgres + ``` + +

Configure ScalarDB

+ + The **database.properties** file in the `scalardb-samples/scalardb-sample` directory contains database configurations for ScalarDB. Please uncomment the properties for PostgreSQL in the **database.properties** file so that the configuration looks as follows: + + ```properties + # For PostgreSQL + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:postgresql://localhost:5432/ + scalar.db.username=postgres + scalar.db.password=postgres + ``` + + +

Run Oracle Database locally

+ + You can run Oracle Database in Docker Compose by using the `docker-compose.yml` file in the `scalardb-samples/scalardb-sample` directory. + + To start Oracle Database, run the following command: + + ```console + docker compose up -d oracle + ``` + +

Configure ScalarDB

+ + The **database.properties** file in the `scalardb-samples/scalardb-sample` directory contains database configurations for ScalarDB. Please uncomment the properties for Oracle Database in the **database.properties** file so that the configuration looks as follows: + + ```properties + # For Oracle + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:oracle:thin:@//localhost:1521/FREEPDB1 + scalar.db.username=SYSTEM + scalar.db.password=Oracle + ``` + + +

Run SQL Server locally

+ + You can run SQL Server in Docker Compose by using the `docker-compose.yml` file in the `scalardb-samples/scalardb-sample` directory. + + To start SQL Server, run the following command: + + ```console + docker compose up -d sqlserver + ``` + +

Configure ScalarDB

+ + The **database.properties** file in the `scalardb-samples/scalardb-sample` directory contains database configurations for ScalarDB. Please uncomment the properties for SQL Server in the **database.properties** file so that the configuration looks as follows: + + ```properties + # For SQL Server + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:sqlserver://localhost:1433;encrypt=true;trustServerCertificate=true + scalar.db.username=sa + scalar.db.password=SqlServer22 + ``` + + +

Run Db2 locally

+ + You can run IBM Db2 in Docker Compose by using the `docker-compose.yml` file in the `scalardb-samples/scalardb-sample` directory. + + To start IBM Db2, run the following command: + + ```console + docker compose up -d db2 + ``` + +

Configure ScalarDB

+ + The **database.properties** file in the `scalardb-samples/scalardb-sample` directory contains database configurations for ScalarDB. Please uncomment the properties for Db2 in the **database.properties** file so that the configuration looks as follows: + + ```properties + # For Db2 + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:db2://localhost:50000/sample + scalar.db.username=db2inst1 + scalar.db.password=db2inst1 + ``` + + +

Run Amazon DynamoDB Local

+ + You can run Amazon DynamoDB Local in Docker Compose by using the `docker-compose.yml` file in the `scalardb-samples/scalardb-sample` directory. + + To start Amazon DynamoDB Local, run the following command: + + ```console + docker compose up -d dynamodb + ``` + +

Configure ScalarDB

+ + The **database.properties** file in the `scalardb-samples/scalardb-sample` directory contains database configurations for ScalarDB. Please uncomment the properties for Amazon DynamoDB Local in the **database.properties** file so that the configuration looks as follows: + + ```properties + # For DynamoDB Local + scalar.db.storage=dynamo + scalar.db.contact_points=sample + scalar.db.username=sample + scalar.db.password=sample + scalar.db.dynamo.endpoint_override=http://localhost:8000 + ``` + + + To use Azure Cosmos DB for NoSQL, you must have an Azure account. If you don't have an Azure account, visit [Create an Azure Cosmos DB account](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/quickstart-portal#create-account). + +

Configure Cosmos DB for NoSQL

+ + Set the **default consistency level** to **Strong** according to the official document at [Configure the default consistency level](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-manage-consistency#configure-the-default-consistency-level). + +

Configure ScalarDB

+ + The following instructions assume that you have properly installed and configured the JDK in your local environment and properly configured your Cosmos DB for NoSQL account in Azure. + + The **database.properties** file in the `scalardb-samples/scalardb-sample` directory contains database configurations for ScalarDB. Be sure to change the values for `scalar.db.contact_points` and `scalar.db.password` as described. + + ```properties + # For Cosmos DB + scalar.db.storage=cosmos + scalar.db.contact_points= + scalar.db.password= + ``` + +:::note + +You can use the primary key or the secondary key in your Azure Cosmos DB account as the value for `scalar.db.password`. + +::: + + +

Run Cassandra locally

+ + You can run Apache Cassandra in Docker Compose by using the `docker-compose.yml` file in the `scalardb-samples/scalardb-sample` directory. + + To start Apache Cassandra, run the following command: + ```console + docker compose up -d cassandra + ``` + +

Configure ScalarDB

+ + The **database.properties** file in the `scalardb-samples/scalardb-sample` directory contains database configurations for ScalarDB. Please uncomment the properties for Cassandra in the **database.properties** file so that the configuration looks as follows: + + ```properties + # For Cassandra + scalar.db.storage=cassandra + scalar.db.contact_points=localhost + scalar.db.username=cassandra + scalar.db.password=cassandra + ``` + + + +For a comprehensive list of configurations for ScalarDB, see [ScalarDB Configurations](configurations.mdx). + +## Configure ScalarDB to run non-transactional storage operations + +To run non-transactional storage operations, you need to configure the `scalar.db.transaction_manager` property to `single-crud-operation` in the configuration file **database.properties**: + +```properties +scalar.db.transaction_manager=single-crud-operation +``` + +## Create or import a schema + +ScalarDB has its own data model and schema that maps to the implementation-specific data model and schema. + +- **Need to create a database schema?** See [ScalarDB Schema Loader](schema-loader.mdx). +- **Need to import an existing database?** See [Importing Existing Tables to ScalarDB by Using ScalarDB Schema Loader](schema-loader-import.mdx). + +## Create your Java application + +This section describes how to add the ScalarDB Core library to your project and how to configure it to run non-transactional storage operations by using Java. + +### Add ScalarDB to your project + +The ScalarDB library is available on the [Maven Central Repository](https://mvnrepository.com/artifact/com.scalar-labs/scalardb). You can add the library as a build dependency to your application by using Gradle or Maven. + +Select your build tool, and follow the instructions to add the build dependency for ScalarDB to your application. + + + + To add the build dependency for ScalarDB by using Gradle, add the following to `build.gradle` in your application: + + ```gradle + dependencies { + implementation 'com.scalar-labs:scalardb:3.16.0' + } + ``` + + + To add the build dependency for ScalarDB by using Maven, add the following to `pom.xml` in your application: + + ```xml + + com.scalar-labs + scalardb + 3.16.0 + + ``` + + + +### Use the Java API + +For details about the Java API, see [ScalarDB Java API Guide](api-guide.mdx). + +:::note + +The following limitations apply to non-transactional storage operations: + +- Beginning a transaction is not supported. For more details, see [Execute transactions without beginning or starting a transaction](api-guide.mdx#execute-transactions-without-beginning-or-starting-a-transaction). +- Executing multiple mutations in a single transaction is not supported. + +::: + +### Learn more + +- [Javadoc](https://javadoc.io/doc/com.scalar-labs/scalardb/3.16.0/index.html) diff --git a/versioned_docs/version-3.X/run-non-transactional-storage-operations-through-primitive-crud-interface.mdx b/versioned_docs/version-3.X/run-non-transactional-storage-operations-through-primitive-crud-interface.mdx new file mode 100644 index 00000000..21d732d9 --- /dev/null +++ b/versioned_docs/version-3.X/run-non-transactional-storage-operations-through-primitive-crud-interface.mdx @@ -0,0 +1,860 @@ +--- +tags: + - Community +displayed_sidebar: docsEnglish +--- + +# Run Non-Transactional Storage Operations Through the Primitive CRUD Interface + +This page explains how to run non-transactional storage operations through the primitive CRUD interface, also known as the Storage API. This guide assumes that you have an advanced understanding of ScalarDB. + +One of the keys to achieving storage-agnostic or database-agnostic ACID transactions on top of existing storage and database systems is the storage abstraction capabilities that ScalarDB provides. Storage abstraction defines a [data model](design.mdx#data-model) and the APIs (Storage API) that issue operations on the basis of the data model. + +Although you will likely use the [Transactional API](api-guide.mdx#transactional-api) in most cases, another option is to use the Storage API. + +The benefits of using the Storage API include the following: + +- As with the Transactional API, you can write your application code without worrying too much about the underlying storage implementation. +- If you don't need transactions for some of the data in your application, you can use the Storage API to partially avoid transactions, which results in faster execution. + +:::warning + +Directly using the Storage API or mixing the Transactional API and the Storage API could cause unexpected behavior. For example, since the Storage API cannot provide transaction capability, the API could cause anomalies or data inconsistency if failures occur when executing operations. + +Therefore, you should be *very* careful about using the Storage API and use it only if you know exactly what you are doing. + +::: + +## Storage API Example + +This section explains how the Storage API can be used in a basic electronic money application. + +:::warning + +The electronic money application is simplified for this example and isn’t suitable for a production environment. + +::: + +### ScalarDB configuration + +Before you begin, you should configure ScalarDB in the same way mentioned in [Getting Started with ScalarDB](getting-started-with-scalardb.mdx). + +With that in mind, this Storage API example assumes that the configuration file `scalardb.properties` exists. + +### Set up the database schema + +You need to define the database schema (the method in which the data will be organized) in the application. For details about the supported data types, see [Data type mapping between ScalarDB and other databases](https://scalardb.scalar-labs.com/docs/latest/schema-loader/#data-type-mapping-between-scalardb-and-the-other-databases). + +For this example, create a file named `emoney-storage.json` in the `scalardb/docs/getting-started` directory. Then, add the following JSON code to define the schema. + +:::note + +In the following JSON, the `transaction` field is set to `false`, which indicates that you should use this table with the Storage API. + +::: + +```json +{ + "emoney.account": { + "transaction": false, + "partition-key": [ + "id" + ], + "clustering-key": [], + "columns": { + "id": "TEXT", + "balance": "INT" + } + } +} +``` + +To apply the schema, go to the [ScalarDB Releases](https://github.com/scalar-labs/scalardb/releases) page and download the ScalarDB Schema Loader that matches the version of ScalarDB that you are using to the `getting-started` folder. + +Then, run the following command, replacing `` with the version of the ScalarDB Schema Loader that you downloaded: + +```console +java -jar scalardb-schema-loader-.jar --config scalardb.properties -f emoney-storage.json +``` + +### Example code + +The following is example source code for the electronic money application that uses the Storage API. + +:::warning + +As previously mentioned, since the Storage API cannot provide transaction capability, the API could cause anomalies or data inconsistency if failures occur when executing operations. Therefore, you should be *very* careful about using the Storage API and use it only if you know exactly what you are doing. + +::: + +```java +public class ElectronicMoney { + + private static final String SCALARDB_PROPERTIES = + System.getProperty("user.dir") + File.separator + "scalardb.properties"; + private static final String NAMESPACE = "emoney"; + private static final String TABLENAME = "account"; + private static final String ID = "id"; + private static final String BALANCE = "balance"; + + private final DistributedStorage storage; + + public ElectronicMoney() throws IOException { + StorageFactory factory = StorageFactory.create(SCALARDB_PROPERTIES); + storage = factory.getStorage(); + } + + public void charge(String id, int amount) throws ExecutionException { + // Retrieve the current balance for id + Get get = + Get.newBuilder() + .namespace(NAMESPACE) + .table(TABLENAME) + .partitionKey(Key.ofText(ID, id)) + .build(); + Optional result = storage.get(get); + + // Calculate the balance + int balance = amount; + if (result.isPresent()) { + int current = result.get().getInt(BALANCE); + balance += current; + } + + // Update the balance + Put put = + Put.newBuilder() + .namespace(NAMESPACE) + .table(TABLENAME) + .partitionKey(Key.ofText(ID, id)) + .intValue(BALANCE, balance) + .build(); + storage.put(put); + } + + public void pay(String fromId, String toId, int amount) throws ExecutionException { + // Retrieve the current balances for ids + Get fromGet = + Get.newBuilder() + .namespace(NAMESPACE) + .table(TABLENAME) + .partitionKey(Key.ofText(ID, fromId)) + .build(); + Get toGet = + Get.newBuilder() + .namespace(NAMESPACE) + .table(TABLENAME) + .partitionKey(Key.ofText(ID, toId)) + .build(); + Optional fromResult = storage.get(fromGet); + Optional toResult = storage.get(toGet); + + // Calculate the balances (it assumes that both accounts exist) + int newFromBalance = fromResult.get().getInt(BALANCE) - amount; + int newToBalance = toResult.get().getInt(BALANCE) + amount; + if (newFromBalance < 0) { + throw new RuntimeException(fromId + " doesn't have enough balance."); + } + + // Update the balances + Put fromPut = + Put.newBuilder() + .namespace(NAMESPACE) + .table(TABLENAME) + .partitionKey(Key.ofText(ID, fromId)) + .intValue(BALANCE, newFromBalance) + .build(); + Put toPut = + Put.newBuilder() + .namespace(NAMESPACE) + .table(TABLENAME) + .partitionKey(Key.ofText(ID, toId)) + .intValue(BALANCE, newToBalance) + .build(); + storage.put(fromPut); + storage.put(toPut); + } + + public int getBalance(String id) throws ExecutionException { + // Retrieve the current balances for id + Get get = + Get.newBuilder() + .namespace(NAMESPACE) + .table(TABLENAME) + .partitionKey(Key.ofText(ID, id)) + .build(); + Optional result = storage.get(get); + + int balance = -1; + if (result.isPresent()) { + balance = result.get().getInt(BALANCE); + } + return balance; + } + + public void close() { + storage.close(); + } +} +``` + +## Storage API guide + +The Storage API is composed of the Administrative API and CRUD API. + +### Administrative API + +You can execute administrative operations programmatically as described in this section. + +:::note + +Another method that you could use to execute administrative operations is by using [Schema Loader](schema-loader.mdx). + +::: + +#### Get a `DistributedStorageAdmin` instance + +To execute administrative operations, you first need to get a `DistributedStorageAdmin` instance. You can obtain the `DistributedStorageAdmin` instance from `StorageFactory` as follows: + +```java +StorageFactory storageFactory = StorageFactory.create(""); +DistributedStorageAdmin admin = storageFactory.getStorageAdmin(); +``` + +For details about configurations, see [ScalarDB Configurations](configurations.mdx). + +After you have executed all administrative operations, you should close the `DistributedStorageAdmin` instance as follows: + +```java +admin.close(); +``` + +#### Create a namespace + +Before creating tables, namespaces must be created since a table belongs to one namespace. + +You can create a namespace as follows: + +```java +// Create the namespace "ns". If the namespace already exists, an exception will be thrown. +admin.createNamespace("ns"); + +// Create the namespace only if it does not already exist. +boolean ifNotExists = true; +admin.createNamespace("ns", ifNotExists); + +// Create the namespace with options. +Map options = ...; +admin.createNamespace("ns", options); +``` + +For details about creation options, see [Creation options](api-guide.mdx#creation-options). + +#### Create a table + +When creating a table, you should define the table metadata and then create the table. + +To define the table metadata, you can use `TableMetadata`. The following shows how to define the columns, partition key, clustering key including clustering orders, and secondary indexes of a table: + +```java +// Define the table metadata. +TableMetadata tableMetadata = + TableMetadata.newBuilder() + .addColumn("c1", DataType.INT) + .addColumn("c2", DataType.TEXT) + .addColumn("c3", DataType.BIGINT) + .addColumn("c4", DataType.FLOAT) + .addColumn("c5", DataType.DOUBLE) + .addPartitionKey("c1") + .addClusteringKey("c2", Scan.Ordering.Order.DESC) + .addClusteringKey("c3", Scan.Ordering.Order.ASC) + .addSecondaryIndex("c4") + .build(); +``` + +For details about the data model of ScalarDB, see [Data Model](design.mdx#data-model). + +Then, create a table as follows: + +```java +// Create the table "ns.tbl". If the table already exists, an exception will be thrown. +admin.createTable("ns", "tbl", tableMetadata); + +// Create the table only if it does not already exist. +boolean ifNotExists = true; +admin.createTable("ns", "tbl", tableMetadata, ifNotExists); + +// Create the table with options. +Map options = ...; +admin.createTable("ns", "tbl", tableMetadata, options); +``` + +#### Create a secondary index + +You can create a secondary index as follows: + +```java +// Create a secondary index on column "c5" for table "ns.tbl". If a secondary index already exists, an exception will be thrown. +admin.createIndex("ns", "tbl", "c5"); + +// Create the secondary index only if it does not already exist. +boolean ifNotExists = true; +admin.createIndex("ns", "tbl", "c5", ifNotExists); + +// Create the secondary index with options. +Map options = ...; +admin.createIndex("ns", "tbl", "c5", options); +``` + +#### Add a new column to a table + +You can add a new, non-partition key column to a table as follows: + +```java +// Add a new column "c6" with the INT data type to the table "ns.tbl". +admin.addNewColumnToTable("ns", "tbl", "c6", DataType.INT) +``` + +:::warning + +You should carefully consider adding a new column to a table because the execution time may vary greatly depending on the underlying storage. Please plan accordingly and consider the following, especially if the database runs in production: + +- **For Cosmos DB for NoSQL and DynamoDB:** Adding a column is almost instantaneous as the table schema is not modified. Only the table metadata stored in a separate table is updated. +- **For Cassandra:** Adding a column will only update the schema metadata and will not modify the existing schema records. The cluster topology is the main factor for the execution time. Changes to the schema metadata are shared to each cluster node via a gossip protocol. Because of this, the larger the cluster, the longer it will take for all nodes to be updated. +- **For relational databases (MySQL, Oracle, etc.):** Adding a column shouldn't take a long time to execute. + +::: + +#### Truncate a table + +You can truncate a table as follows: + +```java +// Truncate the table "ns.tbl". +admin.truncateTable("ns", "tbl"); +``` + +#### Drop a secondary index + +You can drop a secondary index as follows: + +```java +// Drop the secondary index on column "c5" from table "ns.tbl". If the secondary index does not exist, an exception will be thrown. +admin.dropIndex("ns", "tbl", "c5"); + +// Drop the secondary index only if it exists. +boolean ifExists = true; +admin.dropIndex("ns", "tbl", "c5", ifExists); +``` + +#### Drop a table + +You can drop a table as follows: + +```java +// Drop the table "ns.tbl". If the table does not exist, an exception will be thrown. +admin.dropTable("ns", "tbl"); + +// Drop the table only if it exists. +boolean ifExists = true; +admin.dropTable("ns", "tbl", ifExists); +``` + +#### Drop a namespace + +You can drop a namespace as follows: + +```java +// Drop the namespace "ns". If the namespace does not exist, an exception will be thrown. +admin.dropNamespace("ns"); + +// Drop the namespace only if it exists. +boolean ifExists = true; +admin.dropNamespace("ns", ifExists); +``` + +#### Get existing namespaces + +You can get the existing namespaces as follows: + +```java +Set namespaces = admin.getNamespaceNames(); +``` + +:::note + +This method extracts the namespace names of user tables dynamically. As a result, only namespaces that contain tables are returned. Starting from ScalarDB 4.0, we plan to improve the design to remove this limitation. + +::: + +#### Get the tables of a namespace + +You can get the tables of a namespace as follows: + +```java +// Get the tables of the namespace "ns". +Set tables = admin.getNamespaceTableNames("ns"); +``` + +#### Get table metadata + +You can get table metadata as follows: + +```java +// Get the table metadata for "ns.tbl". +TableMetadata tableMetadata = admin.getTableMetadata("ns", "tbl"); +``` + +#### Repair a table + +You can repair the table metadata of an existing table as follows: + +```java +// Repair the table "ns.tbl" with options. +TableMetadata tableMetadata = + TableMetadata.newBuilder() + ... + .build(); +Map options = ...; +admin.repairTable("ns", "tbl", tableMetadata, options); +``` + +### Implement CRUD operations + +The following sections describe CRUD operations. + +#### Get a `DistributedStorage` instance + +To execute CRUD operations in the Storage API, you need to get a `DistributedStorage` instance. + +You can get an instance as follows: + +```java +StorageFactory storageFactory = StorageFactory.create(""); +DistributedStorage storage = storageFactory.getStorage(); +``` + +After you have executed all CRUD operations, you should close the `DistributedStorage` instance as follows: + +```java +storage.close(); +``` + +#### `Get` operation + +`Get` is an operation to retrieve a single record specified by a primary key. + +You need to create a `Get` object first, and then you can execute the object by using the `storage.get()` method as follows: + +```java +// Create a `Get` operation. +Key partitionKey = Key.ofInt("c1", 10); +Key clusteringKey = Key.of("c2", "aaa", "c3", 100L); + +Get get = + Get.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .clusteringKey(clusteringKey) + .projections("c1", "c2", "c3", "c4") + .build(); + +// Execute the `Get` operation. +Optional result = storage.get(get); +``` + +You can also specify projections to choose which columns are returned. + +For details about how to construct `Key` objects, see [Key construction](api-guide.mdx#key-construction). And, for details about how to handle `Result` objects, see [Handle Result objects](api-guide.mdx#handle-result-objects). + +##### Specify a consistency level + +You can specify a consistency level in each operation (`Get`, `Scan`, `Put`, and `Delete`) in the Storage API as follows: + +```java +Get get = + Get.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .clusteringKey(clusteringKey) + .consistency(Consistency.LINEARIZABLE) // Consistency level + .build(); +``` + +The following table describes the three consistency levels: + +| Consistency level | Description | +| ------------------- | ----------- | +| `SEQUENTIAL` | Sequential consistency assumes that the underlying storage implementation makes all operations appear to take effect in some sequential order and the operations of each individual process appear in this sequence. | +| `EVENTUAL` | Eventual consistency assumes that the underlying storage implementation makes all operations take effect eventually. | +| `LINEARIZABLE` | Linearizable consistency assumes that the underlying storage implementation makes each operation appear to take effect atomically at some point between its invocation and completion. | + +##### Execute `Get` by using a secondary index + +You can execute a `Get` operation by using a secondary index. + +Instead of specifying a partition key, you can specify an index key (indexed column) to use a secondary index as follows: + +```java +// Create a `Get` operation by using a secondary index. +Key indexKey = Key.ofFloat("c4", 1.23F); + +Get get = + Get.newBuilder() + .namespace("ns") + .table("tbl") + .indexKey(indexKey) + .projections("c1", "c2", "c3", "c4") + .build(); + +// Execute the `Get` operation. +Optional result = storage.get(get); +``` + +:::note + +If the result has more than one record, `storage.get()` will throw an exception. + +::: + +#### `Scan` operation + +`Scan` is an operation to retrieve multiple records within a partition. You can specify clustering-key boundaries and orderings for clustering-key columns in `Scan` operations. + +You need to create a `Scan` object first, and then you can execute the object by using the `storage.scan()` method as follows: + +```java +// Create a `Scan` operation. +Key partitionKey = Key.ofInt("c1", 10); +Key startClusteringKey = Key.of("c2", "aaa", "c3", 100L); +Key endClusteringKey = Key.of("c2", "aaa", "c3", 300L); + +Scan scan = + Scan.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .start(startClusteringKey, true) // Include startClusteringKey + .end(endClusteringKey, false) // Exclude endClusteringKey + .projections("c1", "c2", "c3", "c4") + .orderings(Scan.Ordering.desc("c2"), Scan.Ordering.asc("c3")) + .limit(10) + .build(); + +// Execute the `Scan` operation. +Scanner scanner = storage.scan(scan); +``` + +You can omit the clustering-key boundaries or specify either a `start` boundary or an `end` boundary. If you don't specify `orderings`, you will get results ordered by the clustering order that you defined when creating the table. + +In addition, you can specify `projections` to choose which columns are returned and use `limit` to specify the number of records to return in `Scan` operations. + +##### Handle `Scanner` objects + +A `Scan` operation in the Storage API returns a `Scanner` object. + +If you want to get results one by one from the `Scanner` object, you can use the `one()` method as follows: + +```java +Optional result = scanner.one(); +``` + +Or, if you want to get a list of all results, you can use the `all()` method as follows: + +```java +List results = scanner.all(); +``` + +In addition, since `Scanner` implements `Iterable`, you can use `Scanner` in a for-each loop as follows: + +```java +for (Result result : scanner) { + ... +} +``` + +Remember to close the `Scanner` object after getting the results: + +```java +scanner.close(); +``` + +Or you can use `try`-with-resources as follows: + +```java +try (Scanner scanner = storage.scan(scan)) { + ... +} +``` + +##### Execute `Scan` by using a secondary index + +You can execute a `Scan` operation by using a secondary index. + +Instead of specifying a partition key, you can specify an index key (indexed column) to use a secondary index as follows: + +```java +// Create a `Scan` operation by using a secondary index. +Key indexKey = Key.ofFloat("c4", 1.23F); + +Scan scan = + Scan.newBuilder() + .namespace("ns") + .table("tbl") + .indexKey(indexKey) + .projections("c1", "c2", "c3", "c4") + .limit(10) + .build(); + +// Execute the `Scan` operation. +Scanner scanner = storage.scan(scan); +``` + +:::note + +You can't specify clustering-key boundaries and orderings in `Scan` by using a secondary index. + +::: + +##### Execute `Scan` without specifying a partition key to retrieve all the records of a table + +You can execute a `Scan` operation without specifying a partition key. + +Instead of calling the `partitionKey()` method in the builder, you can call the `all()` method to scan a table without specifying a partition key as follows: + +```java +// Create a `Scan` operation without specifying a partition key. +Key partitionKey = Key.ofInt("c1", 10); +Key startClusteringKey = Key.of("c2", "aaa", "c3", 100L); +Key endClusteringKey = Key.of("c2", "aaa", "c3", 300L); + +Scan scan = + Scan.newBuilder() + .namespace("ns") + .table("tbl") + .all() + .projections("c1", "c2", "c3", "c4") + .limit(10) + .build(); + +// Execute the `Scan` operation. +Scanner scanner = storage.scan(scan); +``` + +:::note + +You can't specify clustering-key boundaries and orderings in `Scan` without specifying a partition key. + +::: + +#### `Put` operation + +`Put` is an operation to put a record specified by a primary key. The operation behaves as an upsert operation for a record, in which the operation updates the record if the record exists or inserts the record if the record does not exist. + +You need to create a `Put` object first, and then you can execute the object by using the `storage.put()` method as follows: + +```java +// Create a `Put` operation. +Key partitionKey = Key.ofInt("c1", 10); +Key clusteringKey = Key.of("c2", "aaa", "c3", 100L); + +Put put = + Put.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .clusteringKey(clusteringKey) + .floatValue("c4", 1.23F) + .doubleValue("c5", 4.56) + .build(); + +// Execute the `Put` operation. +storage.put(put); +``` + +You can also put a record with `null` values as follows: + +```java +Put put = + Put.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .clusteringKey(clusteringKey) + .floatValue("c4", null) + .doubleValue("c5", null) + .build(); +``` + +:::note + +If you specify `enableImplicitPreRead()`, `disableImplicitPreRead()`, or `implicitPreReadEnabled()` in the `Put` operation builder, they will be ignored. + + +::: + +#### `Delete` operation + +`Delete` is an operation to delete a record specified by a primary key. + +You need to create a `Delete` object first, and then you can execute the object by using the `storage.delete()` method as follows: + +```java +// Create a `Delete` operation. +Key partitionKey = Key.ofInt("c1", 10); +Key clusteringKey = Key.of("c2", "aaa", "c3", 100L); + +Delete delete = + Delete.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .clusteringKey(clusteringKey) + .build(); + +// Execute the `Delete` operation. +storage.delete(delete); +``` + +#### `Put` and `Delete` with a condition + +You can write arbitrary conditions (for example, a bank account balance must be equal to or more than zero) that you require an operation to meet before being executed by implementing logic that checks the conditions. Alternatively, you can write simple conditions in a mutation operation, such as `Put` and `Delete`. + +When a `Put` or `Delete` operation includes a condition, the operation is executed only if the specified condition is met. If the condition is not met when the operation is executed, an exception called `NoMutationException` will be thrown. + +##### Conditions for `Put` + +In a `Put` operation in the Storage API, you can specify a condition that causes the `Put` operation to be executed only when the specified condition matches. This operation is like a compare-and-swap operation where the condition is compared and the update is performed atomically. + +You can specify a condition in a `Put` operation as follows: + +```java +// Build a condition. +MutationCondition condition = + ConditionBuilder.putIf(ConditionBuilder.column("c4").isEqualToFloat(0.0F)) + .and(ConditionBuilder.column("c5").isEqualToDouble(0.0)) + .build(); + +Put put = + Put.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .clusteringKey(clusteringKey) + .floatValue("c4", 1.23F) + .doubleValue("c5", 4.56) + .condition(condition) // condition + .build(); +``` + +Other than the `putIf` condition, you can specify the `putIfExists` and `putIfNotExists` conditions as follows: + +```java +// Build a `putIfExists` condition. +MutationCondition putIfExistsCondition = ConditionBuilder.putIfExists(); + +// Build a `putIfNotExists` condition. +MutationCondition putIfNotExistsCondition = ConditionBuilder.putIfNotExists(); +``` + +##### Conditions for `Delete` + +Similar to a `Put` operation, you can specify a condition in a `Delete` operation in the Storage API. + +You can specify a condition in a `Delete` operation as follows: + +```java +// Build a condition. +MutationCondition condition = + ConditionBuilder.deleteIf(ConditionBuilder.column("c4").isEqualToFloat(0.0F)) + .and(ConditionBuilder.column("c5").isEqualToDouble(0.0)) + .build(); + +Delete delete = + Delete.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .clusteringKey(clusteringKey) + .condition(condition) // condition + .build(); +``` + +In addition to using the `deleteIf` condition, you can specify the `deleteIfExists` condition as follows: + +```java +// Build a `deleteIfExists` condition. +MutationCondition deleteIfExistsCondition = ConditionBuilder.deleteIfExists(); +``` + +#### Mutate operation + +Mutate is an operation to execute multiple mutations (`Put` and `Delete` operations) in a single partition. + +You need to create mutation objects first, and then you can execute the objects by using the `storage.mutate()` method as follows: + +```java +// Create `Put` and `Delete` operations. +Key partitionKey = Key.ofInt("c1", 10); + +Key clusteringKeyForPut = Key.of("c2", "aaa", "c3", 100L); + +Put put = + Put.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .clusteringKey(clusteringKeyForPut) + .floatValue("c4", 1.23F) + .doubleValue("c5", 4.56) + .build(); + +Key clusteringKeyForDelete = Key.of("c2", "bbb", "c3", 200L); + +Delete delete = + Delete.newBuilder() + .namespace("ns") + .table("tbl") + .partitionKey(partitionKey) + .clusteringKey(clusteringKeyForDelete) + .build(); + +// Execute the operations. +storage.mutate(Arrays.asList(put, delete)); +``` + +:::note + +A Mutate operation only accepts mutations for a single partition; otherwise, an exception will be thrown. + +In addition, if you specify multiple conditions in a Mutate operation, the operation will be executed only when all the conditions match. + +::: + +#### Default namespace for CRUD operations + +A default namespace for all CRUD operations can be set by using a property in the ScalarDB configuration. + +```properties +scalar.db.default_namespace_name= +``` + +Any operation that does not specify a namespace will use the default namespace set in the configuration. + +```java +// This operation will target the default namespace. +Scan scanUsingDefaultNamespace = + Scan.newBuilder() + .table("tbl") + .all() + .build(); +// This operation will target the "ns" namespace. +Scan scanUsingSpecifiedNamespace = + Scan.newBuilder() + .namespace("ns") + .table("tbl") + .all() + .build(); +``` diff --git a/versioned_docs/version-3.X/run-transactions-through-scalardb-core-library.mdx b/versioned_docs/version-3.X/run-transactions-through-scalardb-core-library.mdx new file mode 100644 index 00000000..b448d61f --- /dev/null +++ b/versioned_docs/version-3.X/run-transactions-through-scalardb-core-library.mdx @@ -0,0 +1,242 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Run Transactions Through the ScalarDB Core Library + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +This guide explains how to configure your ScalarDB properties file and create schemas to run transactions through a one-phase or a two-phase commit interface by using the ScalarDB Core library. + +## Preparation + +For the purpose of this guide, you will set up a database and ScalarDB by using a sample in the ScalarDB samples repository. + +### Clone the ScalarDB samples repository + +Open **Terminal**, then clone the ScalarDB samples repository by running the following command: + +```console +git clone https://github.com/scalar-labs/scalardb-samples +``` + +Then, go to the directory that contains the necessary files by running the following command: + +```console +cd scalardb-samples/scalardb-sample +``` + +## Set up a database + +Select your database, and follow the instructions to configure it for ScalarDB. + +For a list of databases that ScalarDB supports, see [Databases](requirements.mdx#databases). + + + +

Run MySQL locally

+ + You can run MySQL in Docker Compose by using the `docker-compose.yml` file in the `scalardb-samples/scalardb-sample` directory. + + To start MySQL, run the following command: + + ```console + docker compose up -d mysql + ``` + +

Configure ScalarDB

+ + The **database.properties** file in the `scalardb-samples/scalardb-sample` directory contains database configurations for ScalarDB. Please uncomment the properties for MySQL in the **database.properties** file so that the configuration looks as follows: + + ```properties + # For MySQL + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:mysql://localhost:3306/ + scalar.db.username=root + scalar.db.password=mysql + ``` + + +

Run PostgreSQL locally

+ + You can run PostgreSQL in Docker Compose by using the `docker-compose.yml` file in the `scalardb-samples/scalardb-sample` directory. + + To start PostgreSQL, run the following command: + + ```console + docker compose up -d postgres + ``` + +

Configure ScalarDB

+ + The **database.properties** file in the `scalardb-samples/scalardb-sample` directory contains database configurations for ScalarDB. Please uncomment the properties for PostgreSQL in the **database.properties** file so that the configuration looks as follows: + + ```properties + # For PostgreSQL + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:postgresql://localhost:5432/ + scalar.db.username=postgres + scalar.db.password=postgres + ``` + + +

Run Oracle Database locally

+ + You can run Oracle Database in Docker Compose by using the `docker-compose.yml` file in the `scalardb-samples/scalardb-sample` directory. + + To start Oracle Database, run the following command: + + ```console + docker compose up -d oracle + ``` + +

Configure ScalarDB

+ + The **database.properties** file in the `scalardb-samples/scalardb-sample` directory contains database configurations for ScalarDB. Please uncomment the properties for Oracle Database in the **database.properties** file so that the configuration looks as follows: + + ```properties + # For Oracle + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:oracle:thin:@//localhost:1521/FREEPDB1 + scalar.db.username=SYSTEM + scalar.db.password=Oracle + ``` + + +

Run SQL Server locally

+ + You can run SQL Server in Docker Compose by using the `docker-compose.yml` file in the `scalardb-samples/scalardb-sample` directory. + + To start SQL Server, run the following command: + + ```console + docker compose up -d sqlserver + ``` + +

Configure ScalarDB

+ + The **database.properties** file in the `scalardb-samples/scalardb-sample` directory contains database configurations for ScalarDB. Please uncomment the properties for SQL Server in the **database.properties** file so that the configuration looks as follows: + + ```properties + # For SQL Server + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:sqlserver://localhost:1433;encrypt=true;trustServerCertificate=true + scalar.db.username=sa + scalar.db.password=SqlServer22 + ``` + + +

Run Db2 locally

+ + You can run IBM Db2 in Docker Compose by using the `docker-compose.yml` file in the `scalardb-samples/scalardb-sample` directory. + + To start IBM Db2, run the following command: + + ```console + docker compose up -d db2 + ``` + +

Configure ScalarDB

+ + The **database.properties** file in the `scalardb-samples/scalardb-sample` directory contains database configurations for ScalarDB. Please uncomment the properties for Db2 in the **database.properties** file so that the configuration looks as follows: + + ```properties + # For Db2 + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:db2://localhost:50000/sample + scalar.db.username=db2inst1 + scalar.db.password=db2inst1 + ``` + + +

Run Amazon DynamoDB Local

+ + You can run Amazon DynamoDB Local in Docker Compose by using the `docker-compose.yml` file in the `scalardb-samples/scalardb-sample` directory. + + To start Amazon DynamoDB Local, run the following command: + + ```console + docker compose up -d dynamodb + ``` + +

Configure ScalarDB

+ + The **database.properties** file in the `scalardb-samples/scalardb-sample` directory contains database configurations for ScalarDB. Please uncomment the properties for Amazon DynamoDB Local in the **database.properties** file so that the configuration looks as follows: + + ```properties + # For DynamoDB Local + scalar.db.storage=dynamo + scalar.db.contact_points=sample + scalar.db.username=sample + scalar.db.password=sample + scalar.db.dynamo.endpoint_override=http://localhost:8000 + ``` + + + To use Azure Cosmos DB for NoSQL, you must have an Azure account. If you don't have an Azure account, visit [Create an Azure Cosmos DB account](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/quickstart-portal#create-account). + +

Configure Cosmos DB for NoSQL

+ + Set the **default consistency level** to **Strong** according to the official document at [Configure the default consistency level](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-manage-consistency#configure-the-default-consistency-level). + +

Configure ScalarDB

+ + The following instructions assume that you have properly installed and configured the JDK in your local environment and properly configured your Cosmos DB for NoSQL account in Azure. + + The **database.properties** file in the `scalardb-samples/scalardb-sample` directory contains database configurations for ScalarDB. Be sure to change the values for `scalar.db.contact_points` and `scalar.db.password` as described. + + ```properties + # For Cosmos DB + scalar.db.storage=cosmos + scalar.db.contact_points= + scalar.db.password= + ``` + +:::note + +You can use the primary key or the secondary key in your Azure Cosmos DB account as the value for `scalar.db.password`. + +::: + + +

Run Cassandra locally

+ + You can run Apache Cassandra in Docker Compose by using the `docker-compose.yml` file in the `scalardb-samples/scalardb-sample` directory. + + To start Apache Cassandra, run the following command: + ```console + docker compose up -d cassandra + ``` + +

Configure ScalarDB

+ + The **database.properties** file in the `scalardb-samples/scalardb-sample` directory contains database configurations for ScalarDB. Please uncomment the properties for Cassandra in the **database.properties** file so that the configuration looks as follows: + + ```properties + # For Cassandra + scalar.db.storage=cassandra + scalar.db.contact_points=localhost + scalar.db.username=cassandra + scalar.db.password=cassandra + ``` + + + +For a comprehensive list of configurations for ScalarDB, see [ScalarDB Configurations](configurations.mdx). + +## Create or import a schema + +ScalarDB has its own data model and schema that maps to the implementation-specific data model and schema. + +- **Need to create a database schema?** See [ScalarDB Schema Loader](schema-loader.mdx). +- **Need to import an existing database?** See [Importing Existing Tables to ScalarDB by Using ScalarDB Schema Loader](schema-loader-import.mdx). + +## Run transactions by using Java + +- **Want to run transactions by using a one-phase commit interface?** See the [ScalarDB Java API Guide](api-guide.mdx#transactional-api). +- **Want to run transactions by using a two-phase commit interface?** See [Transactions with a Two-Phase Commit Interface](two-phase-commit-transactions.mdx). diff --git a/versioned_docs/version-3.X/scalar-licensing/index.mdx b/versioned_docs/version-3.X/scalar-licensing/index.mdx new file mode 100644 index 00000000..d6a813a6 --- /dev/null +++ b/versioned_docs/version-3.X/scalar-licensing/index.mdx @@ -0,0 +1,65 @@ +--- +tags: + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# How to Configure a Product License Key + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +To run Scalar products, you must create a `.properties` file and add your product license key and a certificate to the file. In your `.properties` file, copy one of the following configurations, based on the product you're using, and paste the contents in the `.properties` file, replacing `` with your license key. + +:::note + +If you don't have a license key, please [contact us](https://www.scalar-labs.com/contact). + +::: + +:::warning + +If you're using a trial license, ScalarDB must be connected to the Internet. An Internet connection is required to check if the trial license is valid and hasn't expired. + +::: + +## ScalarDB Enterprise Edition + + + + ```properties + scalar.db.cluster.node.licensing.license_key= + scalar.db.cluster.node.licensing.license_check_cert_pem=-----BEGIN CERTIFICATE-----\nMIICKzCCAdKgAwIBAgIIBXxj3s8NU+owCgYIKoZIzj0EAwIwbDELMAkGA1UEBhMC\nSlAxDjAMBgNVBAgTBVRva3lvMREwDwYDVQQHEwhTaGluanVrdTEVMBMGA1UEChMM\nU2NhbGFyLCBJbmMuMSMwIQYDVQQDExplbnRlcnByaXNlLnNjYWxhci1sYWJzLmNv\nbTAeFw0yMzExMTYwNzExNTdaFw0yNDAyMTUxMzE2NTdaMGwxCzAJBgNVBAYTAkpQ\nMQ4wDAYDVQQIEwVUb2t5bzERMA8GA1UEBxMIU2hpbmp1a3UxFTATBgNVBAoTDFNj\nYWxhciwgSW5jLjEjMCEGA1UEAxMaZW50ZXJwcmlzZS5zY2FsYXItbGFicy5jb20w\nWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAATJx5gvAr+GZAHcBpUvDFDsUlFo4GNw\npRfsntzwStIP8ac3dew7HT4KbGBWei0BvIthleaqpv0AEP7JT6eYAkNvo14wXDAO\nBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwG\nA1UdEwEB/wQCMAAwHQYDVR0OBBYEFMIe+XuuZcnDX1c3TmUPlu3kNv/wMAoGCCqG\nSM49BAMCA0cAMEQCIGGlqKpgv+KW+Z1ZkjfMHjSGeUZKBLwfMtErVyc9aTdIAiAy\nvsZyZP6Or9o40x3l3pw/BT7wvy93Jm0T4vtVQH6Zuw==\n-----END CERTIFICATE----- + ``` + + + ```properties + scalar.db.cluster.node.licensing.license_key= + scalar.db.cluster.node.licensing.license_check_cert_pem=-----BEGIN CERTIFICATE-----\nMIICKzCCAdKgAwIBAgIIBXxj3s8NU+owCgYIKoZIzj0EAwIwbDELMAkGA1UEBhMC\nSlAxDjAMBgNVBAgTBVRva3lvMREwDwYDVQQHEwhTaGluanVrdTEVMBMGA1UEChMM\nU2NhbGFyLCBJbmMuMSMwIQYDVQQDExplbnRlcnByaXNlLnNjYWxhci1sYWJzLmNv\nbTAeFw0yMzExMTYwNzExNTdaFw0yNDAyMTUxMzE2NTdaMGwxCzAJBgNVBAYTAkpQ\nMQ4wDAYDVQQIEwVUb2t5bzERMA8GA1UEBxMIU2hpbmp1a3UxFTATBgNVBAoTDFNj\nYWxhciwgSW5jLjEjMCEGA1UEAxMaZW50ZXJwcmlzZS5zY2FsYXItbGFicy5jb20w\nWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAATJx5gvAr+GZAHcBpUvDFDsUlFo4GNw\npRfsntzwStIP8ac3dew7HT4KbGBWei0BvIthleaqpv0AEP7JT6eYAkNvo14wXDAO\nBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwG\nA1UdEwEB/wQCMAAwHQYDVR0OBBYEFMIe+XuuZcnDX1c3TmUPlu3kNv/wMAoGCCqG\nSM49BAMCA0cAMEQCIGGlqKpgv+KW+Z1ZkjfMHjSGeUZKBLwfMtErVyc9aTdIAiAy\nvsZyZP6Or9o40x3l3pw/BT7wvy93Jm0T4vtVQH6Zuw==\n-----END CERTIFICATE----- + ``` + + + ```properties + scalar.db.cluster.node.licensing.license_key= + scalar.db.cluster.node.licensing.license_check_cert_pem=-----BEGIN CERTIFICATE-----\nMIICIzCCAcigAwIBAgIIKT9LIGX1TJQwCgYIKoZIzj0EAwIwZzELMAkGA1UEBhMC\nSlAxDjAMBgNVBAgTBVRva3lvMREwDwYDVQQHEwhTaGluanVrdTEVMBMGA1UEChMM\nU2NhbGFyLCBJbmMuMR4wHAYDVQQDExV0cmlhbC5zY2FsYXItbGFicy5jb20wHhcN\nMjMxMTE2MDcxMDM5WhcNMjQwMjE1MTMxNTM5WjBnMQswCQYDVQQGEwJKUDEOMAwG\nA1UECBMFVG9reW8xETAPBgNVBAcTCFNoaW5qdWt1MRUwEwYDVQQKEwxTY2FsYXIs\nIEluYy4xHjAcBgNVBAMTFXRyaWFsLnNjYWxhci1sYWJzLmNvbTBZMBMGByqGSM49\nAgEGCCqGSM49AwEHA0IABBSkIYAk7r5FRDf5qRQ7dbD3ib5g3fb643h4hqCtK+lC\nwM4AUr+PPRoquAy+Ey2sWEvYrWtl2ZjiYyyiZw8slGCjXjBcMA4GA1UdDwEB/wQE\nAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIw\nADAdBgNVHQ4EFgQUbFyOWFrsjkkOvjw6vK3gGUADGOcwCgYIKoZIzj0EAwIDSQAw\nRgIhAKwigOb74z9BdX1+dUpeVG8WrzLTIqdIU0w+9jhAueXoAiEA6cniJ3qsP4j7\nsck62kHnFpH1fCUOc/b/B8ZtfeXI2Iw=\n-----END CERTIFICATE----- + ``` + + + +## ScalarDB Analytics with Spark + + + + ```apacheconf + spark.sql.catalog.scalardb_catalog.license.key + spark.sql.catalog.scalardb_catalog.license.cert_pem -----BEGIN CERTIFICATE-----\nMIICKzCCAdKgAwIBAgIIBXxj3s8NU+owCgYIKoZIzj0EAwIwbDELMAkGA1UEBhMC\nSlAxDjAMBgNVBAgTBVRva3lvMREwDwYDVQQHEwhTaGluanVrdTEVMBMGA1UEChMM\nU2NhbGFyLCBJbmMuMSMwIQYDVQQDExplbnRlcnByaXNlLnNjYWxhci1sYWJzLmNv\nbTAeFw0yMzExMTYwNzExNTdaFw0yNDAyMTUxMzE2NTdaMGwxCzAJBgNVBAYTAkpQ\nMQ4wDAYDVQQIEwVUb2t5bzERMA8GA1UEBxMIU2hpbmp1a3UxFTATBgNVBAoTDFNj\nYWxhciwgSW5jLjEjMCEGA1UEAxMaZW50ZXJwcmlzZS5zY2FsYXItbGFicy5jb20w\nWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAATJx5gvAr+GZAHcBpUvDFDsUlFo4GNw\npRfsntzwStIP8ac3dew7HT4KbGBWei0BvIthleaqpv0AEP7JT6eYAkNvo14wXDAO\nBgNVHQ8BAf8EBAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMAwG\nA1UdEwEB/wQCMAAwHQYDVR0OBBYEFMIe+XuuZcnDX1c3TmUPlu3kNv/wMAoGCCqG\nSM49BAMCA0cAMEQCIGGlqKpgv+KW+Z1ZkjfMHjSGeUZKBLwfMtErVyc9aTdIAiAy\nvsZyZP6Or9o40x3l3pw/BT7wvy93Jm0T4vtVQH6Zuw==\n-----END CERTIFICATE----- + ``` + + + ```apacheconf + spark.sql.catalog.scalardb_catalog.license.key + spark.sql.catalog.scalardb_catalog.license.cert_pem -----BEGIN CERTIFICATE-----\nMIICIzCCAcigAwIBAgIIKT9LIGX1TJQwCgYIKoZIzj0EAwIwZzELMAkGA1UEBhMC\nSlAxDjAMBgNVBAgTBVRva3lvMREwDwYDVQQHEwhTaGluanVrdTEVMBMGA1UEChMM\nU2NhbGFyLCBJbmMuMR4wHAYDVQQDExV0cmlhbC5zY2FsYXItbGFicy5jb20wHhcN\nMjMxMTE2MDcxMDM5WhcNMjQwMjE1MTMxNTM5WjBnMQswCQYDVQQGEwJKUDEOMAwG\nA1UECBMFVG9reW8xETAPBgNVBAcTCFNoaW5qdWt1MRUwEwYDVQQKEwxTY2FsYXIs\nIEluYy4xHjAcBgNVBAMTFXRyaWFsLnNjYWxhci1sYWJzLmNvbTBZMBMGByqGSM49\nAgEGCCqGSM49AwEHA0IABBSkIYAk7r5FRDf5qRQ7dbD3ib5g3fb643h4hqCtK+lC\nwM4AUr+PPRoquAy+Ey2sWEvYrWtl2ZjiYyyiZw8slGCjXjBcMA4GA1UdDwEB/wQE\nAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIw\nADAdBgNVHQ4EFgQUbFyOWFrsjkkOvjw6vK3gGUADGOcwCgYIKoZIzj0EAwIDSQAw\nRgIhAKwigOb74z9BdX1+dUpeVG8WrzLTIqdIU0w+9jhAueXoAiEA6cniJ3qsP4j7\nsck62kHnFpH1fCUOc/b/B8ZtfeXI2Iw=\n-----END CERTIFICATE----- + ``` + + diff --git a/versioned_docs/version-3.X/scalar-manager/images/backup-and-restore-check-pauses.png b/versioned_docs/version-3.X/scalar-manager/images/backup-and-restore-check-pauses.png new file mode 100644 index 00000000..d4f63157 Binary files /dev/null and b/versioned_docs/version-3.X/scalar-manager/images/backup-and-restore-check-pauses.png differ diff --git a/versioned_docs/version-3.X/scalar-manager/images/backup-and-restore-create-pauses.png b/versioned_docs/version-3.X/scalar-manager/images/backup-and-restore-create-pauses.png new file mode 100644 index 00000000..927f1910 Binary files /dev/null and b/versioned_docs/version-3.X/scalar-manager/images/backup-and-restore-create-pauses.png differ diff --git a/versioned_docs/version-3.X/scalar-manager/images/dashboard-cluster.png b/versioned_docs/version-3.X/scalar-manager/images/dashboard-cluster.png new file mode 100644 index 00000000..cdc5c5ab Binary files /dev/null and b/versioned_docs/version-3.X/scalar-manager/images/dashboard-cluster.png differ diff --git a/versioned_docs/version-3.X/scalar-manager/images/dashboard-pod-list.png b/versioned_docs/version-3.X/scalar-manager/images/dashboard-pod-list.png new file mode 100644 index 00000000..ed247f0c Binary files /dev/null and b/versioned_docs/version-3.X/scalar-manager/images/dashboard-pod-list.png differ diff --git a/versioned_docs/version-3.X/scalar-manager/images/logs.png b/versioned_docs/version-3.X/scalar-manager/images/logs.png new file mode 100644 index 00000000..1127bd71 Binary files /dev/null and b/versioned_docs/version-3.X/scalar-manager/images/logs.png differ diff --git a/versioned_docs/version-3.X/scalar-manager/images/metrics.png b/versioned_docs/version-3.X/scalar-manager/images/metrics.png new file mode 100644 index 00000000..e4f4d116 Binary files /dev/null and b/versioned_docs/version-3.X/scalar-manager/images/metrics.png differ diff --git a/versioned_docs/version-3.X/scalar-manager/images/metrics2.png b/versioned_docs/version-3.X/scalar-manager/images/metrics2.png new file mode 100644 index 00000000..6f76551b Binary files /dev/null and b/versioned_docs/version-3.X/scalar-manager/images/metrics2.png differ diff --git a/versioned_docs/version-3.X/scalar-manager/overview.mdx b/versioned_docs/version-3.X/scalar-manager/overview.mdx new file mode 100644 index 00000000..525b9306 --- /dev/null +++ b/versioned_docs/version-3.X/scalar-manager/overview.mdx @@ -0,0 +1,54 @@ +--- +tags: + - Enterprise Option +displayed_sidebar: docsEnglish +--- + +# Scalar Manager Overview + +Scalar Manager is a centralized management and monitoring solution for ScalarDB within Kubernetes cluster environments. +It simplifies the operational tasks associated with these products by aggregating essential functionalities into a graphical user interface (GUI). + +## Why Scalar Manager? + +Before Scalar Manager was released, you would need to use various command-line tools and third-party solutions individually to manage and monitor ScalarDB deployments. +For example, `kubectl` is often used to check deployment status, the Prometheus stack for monitoring metrics, the Loki stack for log analysis, and Scalar's proprietary CLI tool for pausing ScalarDB to ensure transactional consistency between multiple databases. +This constellation of tools presented a steep learning curve and lacked a unified interface, resulting in inefficient workflows for performing routine management tasks or troubleshooting issues. + +Scalar Manager mitigates these pain points by aggregating essential functionalities into a single, user-friendly GUI. +With Scalar Manager, you can reduce the time and effort needed for management and monitoring, allowing you to focus on business development and operations. + +## Key features + +At its core, Scalar Manager provides the following features. + +### Centralized cluster visualization + +You can quickly gain real-time metrics about cluster health, pod logs, hardware usage, performance metrics like requests per second, and deep visibility into time-series data via the Grafana dashboards. + +![dashboard-cluster](images/dashboard-cluster.png) +![dashboard-pod-list](images/dashboard-pod-list.png) + +With the Grafana dashboards, you can also view pod logs and metrics in real-time or in time series. + +![logs](images/logs.png) +![metrics](images/metrics2.png) + +### Streamlined pausing job management + +You can execute or schedule pausing jobs to ensure transactional consistency, review and manage scheduled jobs, and monitor paused states within an intuitive GUI. + +![create-pauses](images/backup-and-restore-create-pauses.png) +![check-pauses](images/backup-and-restore-check-pauses.png) + +### User management + +Scalar Manager includes authentication capabilities, allowing for secure access control to your deployment. The system provides user management functionalities that enable administrators to create, modify, and remove user accounts through an intuitive interface. + +### Authentication and authorization + +By using the authorization feature, administrators can define and assign specific roles to users, controlling their access permissions within the Scalar Manager environment. This control ensures that users only have access to the functionalities relevant to their responsibilities. + +### Integrated authentication with Grafana + +Scalar Manager now offers seamless authentication integration between your Grafana instance and other components of the system. This single-sign-on capability eliminates the need for multiple authentication processes, streamlining the user experience and enhancing security by reducing credential management overhead. diff --git a/versioned_docs/version-3.X/scalardb-analytics-postgresql/getting-started.mdx b/versioned_docs/version-3.X/scalardb-analytics-postgresql/getting-started.mdx new file mode 100644 index 00000000..29e34ce5 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-analytics-postgresql/getting-started.mdx @@ -0,0 +1,98 @@ +--- +tags: + - Community +displayed_sidebar: docsEnglish +--- + +# Getting Started with ScalarDB Analytics with PostgreSQL + +This document explains how to get started with ScalarDB Analytics with PostgreSQL. We assume that you have already installed ScalarDB Analytics with PostgreSQL and that all required services are running. If you don't have such an environment, please follow the instructions in [How to Install ScalarDB Analytics with PostgreSQL in Your Local Environment by Using Docker](./installation.mdx). Because ScalarDB Analytics with PostgreSQL executes queries via PostgreSQL, we also assume that you already have a `psql` client or another PostgreSQL client to send queries to PostgreSQL. + +## What is ScalarDB Analytics with PostgreSQL? + +ScalarDB, as a universal transaction manager, targets mainly transactional workloads and therefore supports limited subsets of relational queries. + +ScalarDB Analytics with PostgreSQL extends the functionality of ScalarDB to process analytical queries on ScalarDB-managed data by using PostgreSQL and its foreign data wrapper (FDW) extension. + +ScalarDB Analytics with PostgreSQL mainly consists of two components: PostgreSQL and Schema Importer. + +PostgreSQL runs as a service, accepting queries from users to process. FDW extensions are used to read data from the back-end storages that ScalarDB manages. Schema Importer is a tool to import the schema of the ScalarDB database into PostgreSQL so that users can see tables on the PostgreSQL side, which are identical to the tables on the ScalarDB side. + +## Set up a ScalarDB database + +First, you need one or more ScalarDB databases to run analytical queries with ScalarDB Analytics with PostgreSQL. If you have your own ScalarDB database, you can skip this section and use your database instead. If you use the [scalardb-samples/scalardb-analytics-postgresql-sample](https://github.com/scalar-labs/scalardb-samples/tree/main/scalardb-analytics-postgresql-sample) project, you can set up a sample database by running the following command in the project directory. + +```console +docker compose run --rm schema-loader \ + -c /etc/scalardb.properties \ + --schema-file /etc/schema.json \ + --coordinator \ + --no-backup \ + --no-scaling +``` + +This command sets up [multiple storage instances](../multi-storage-transactions.mdx) that consist of DynamoDB, PostgreSQL, and Cassandra. Then, the command creates namespaces for `dynamons`, `postgresns`, and `cassandrans` that are mapped to those storages, creates tables for `dynamons.customer`, `postgresns.orders`, and `cassandrans.lineitem` by using [ScalarDB Schema Loader](https://scalardb.scalar-labs.com/docs/latest/schema-loader/). + +![Multi-storage overview](./images/multi-storage-overview.png) + +You can load sample data into the created tables by running the following command. + +```console +docker compose run --rm sample-data-loader +``` + +## Import the schemas from ScalarDB into PostgreSQL + +Next, let's import the schemas of the ScalarDB databases into PostgreSQL that processes analytical queries. ScalarDB Analytics with PostgreSQL provides a tool, Schema Importer, for this purpose. It'll get everything in place to run analytical queries for you. + +```console +docker compose run --rm schema-importer \ + import \ + --config /etc/scalardb.properties \ + --host analytics \ + --port 5432 \ + --database test \ + --user postgres \ + --password postgres \ + --namespace cassandrans \ + --namespace postgresns \ + --namespace dynamons \ + --config-on-postgres-host /etc/scalardb.properties +``` + +If you use your own ScalarDB database, you must replace the `--config` and `--config-on-postgres-host` options with your ScalarDB configuration file and the `--namespace` options with your ScalarDB namespaces to import. + +This creates tables (in precise, views) with the same names as the tables in the ScalarDB databases. In this example, the tables of `dynamons.customer`, `postgresns.orders`, and `cassandrans.lineitem` are created. The column definitions are also identical to the ScalarDB databases. These tables are [foreign tables](https://www.postgresql.org/docs/current/sql-createforeigntable.html) connected to the underlying storage of the ScalarDB databases using FDW. Therefore, you can equate those tables in PostgreSQL with the tables in the ScalarDB databases. + +![Imported schema](./images/imported-schema.png) + +## Run analytical queries + +Now, you have all tables to read the same data in the ScalarDB databases and can run any arbitrary analytical queries supported by PostgreSQL. To run queries, please connect to PostgreSQL with `psql` or other client. + +```console +psql -U postgres -h localhost test +Password for user postgres: + +> select c_mktsegment, count(*) from dynamons.customer group by c_mktsegment; + c_mktsegment | count +--------------+------- + AUTOMOBILE | 4 + BUILDING | 2 + FURNITURE | 1 + HOUSEHOLD | 2 + MACHINERY | 1 +(5 rows) +``` + +For details about the sample data and additional practical work, see the sample application page. + +## Caveats + +### Isolation level + +ScalarDB Analytics with PostgreSQL reads data with the **Read Committed** isolation level set. This isolation level ensures that the data you read has been committed in the past but does not guarantee that you can read consistent data at a particular point in time. + +### Write operations are not supported + +ScalarDB Analytics with PostgreSQL only supports read-only queries. `INSERT`, `UPDATE`, and other write operations are not supported. diff --git a/versioned_docs/version-3.X/scalardb-analytics-postgresql/images/imported-schema.png b/versioned_docs/version-3.X/scalardb-analytics-postgresql/images/imported-schema.png new file mode 100644 index 00000000..1cf8fea3 Binary files /dev/null and b/versioned_docs/version-3.X/scalardb-analytics-postgresql/images/imported-schema.png differ diff --git a/versioned_docs/version-3.X/scalardb-analytics-postgresql/images/multi-storage-overview.png b/versioned_docs/version-3.X/scalardb-analytics-postgresql/images/multi-storage-overview.png new file mode 100644 index 00000000..fc8df1cb Binary files /dev/null and b/versioned_docs/version-3.X/scalardb-analytics-postgresql/images/multi-storage-overview.png differ diff --git a/versioned_docs/version-3.X/scalardb-analytics-postgresql/installation.mdx b/versioned_docs/version-3.X/scalardb-analytics-postgresql/installation.mdx new file mode 100644 index 00000000..ca3e82bf --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-analytics-postgresql/installation.mdx @@ -0,0 +1,61 @@ +--- +tags: + - Community +displayed_sidebar: docsEnglish +--- + +# How to Install ScalarDB Analytics with PostgreSQL in Your Local Environment by Using Docker + +This document explains how to set up a local environment that runs ScalarDB Analytics with PostgreSQL using the multi-storage back-end of Cassandra, PostgreSQL, and DynamoDB local server using [Docker Compose](https://docs.docker.com/compose/). + +## Prerequisites + +- [Docker Engine](https://docs.docker.com/engine/) and [Docker Compose](https://docs.docker.com/compose/). + +Follow the instructions on the Docker website according to your platform. + +## Step 1. Clone the `scalardb-samples` repository + +[scalardb-samples/scalardb-analytics-postgresql-sample](https://github.com/scalar-labs/scalardb-samples/tree/main/scalardb-analytics-postgresql-sample) repository is a project containing a sample configuration to set up ScalarDB Analytics with PostgreSQL. + +Determine the location on your local machine where you want to run the scalardb-analytics-postgresql-sample app. Then, open Terminal, go to the location by using the `cd` command, and run the following commands: + +```console +git clone https://github.com/scalar-labs/scalardb-samples.git +cd scalardb-samples/scalardb-analytics-postgresql-sample +``` + +## Step 2. Start up the ScalarDB Analytics with PostgreSQL services + +The following command starts up the PostgreSQL instance that serves ScalarDB Analytics with PostgreSQL along with the back-end servers of Cassandra, PostgreSQL, and DynamoDB local in the Docker containers. When you first run the command, the required Docker images will be downloaded from the GitHub Container Registry. + +```console +docker-compose up +``` + +If you want to run the containers in the background, add the `-d` (--detach) option: + +```console +docker-compose up -d +``` + +If you already have your own ScalarDB database and want to use it as a back-end service, you can launch only the PostgreSQL instance without starting additional back-end servers in the container. + +```console +docker-compose up analytics +``` + +## Step 3. Run your analytical queries + +Now, you should have all the required services running. To run analytical queries, see [Getting Started with ScalarDB Analytics with PostgreSQL](./getting-started.mdx). + +## Step 4. Shut down the ScalarDB Analytics with PostgreSQL services + +To shut down the containers, do one of the following in Terminal, depending on how you: + +- If you started the containers in the foreground, press Ctrl+C where `docker-compose` is running. +- If you started the containers in the background, run the following command. + +```console +docker-compose down +``` diff --git a/versioned_docs/version-3.X/scalardb-analytics-postgresql/scalardb-fdw.mdx b/versioned_docs/version-3.X/scalardb-analytics-postgresql/scalardb-fdw.mdx new file mode 100644 index 00000000..d8583026 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-analytics-postgresql/scalardb-fdw.mdx @@ -0,0 +1,180 @@ +--- +tags: + - Community +displayed_sidebar: docsEnglish +--- + +# ScalarDB FDW + +ScalarDB FDW is a PostgreSQL extension that implements a foreign data wrapper (FDW) for [ScalarDB](https://www.scalar-labs.com/scalardb/). + +ScalarDB FDW uses the Java Native Interface to directly utilize ScalarDB as a library inside the FDW and read data from external databases via scan operations for ScalarDB. + +## Prerequisites + +You must have the following prerequisites set up in your environment. + +### JDK + +You must install a version of the Java Development Kit (JDK) that is compatible with ScalarDB. In addition, you must set the `JAVA_HOME` environment variable, which points to your JDK installation directory. + +Note that since these extensions use the Java Native Interface (JNI) internally, you must include the dynamic library of the Java virtual machine (JVM), such as `libjvm.so`, in the library search path. + +### PostgreSQL + +This extension supports PostgreSQL 13 or later. For details on how to install PostgreSQL, see the official documentation at [Server Administration](https://www.postgresql.org/docs/current/admin.html). + +## Build and installation + +You can build and install this extension by running the following command. + +```console +make install +``` + +### Common build errors + +This section describes some common build errors that you might encounter. + +#### ld: library not found for -ljvm + +Normally, the build script finds the path for `libjvm.so` and properly sets it as a library search path. However, if you encounter the error `ld: library not found for -ljvm`, please copy the `libjvm.so` file to the default library search path. For example: + +```console +ln -s //libjvm.so /usr/lib64/libjvm.so +``` + +## Usage + +This section provides a usage example and available options for FDW for ScalarDB. + +### Example + +The following example shows you how to install and create the necessary components, and then run a query by using the FDW extension. + +#### 1. Install the extension + +For details on how to install the extension, see the [Build and installation](#build-and-installation) section. + +#### 2. Create an extension + +To create an extension, run the following command: + +```sql +CREATE EXTENSION scalardb_fdw; +``` + +#### 3. Create a foreign server + +To create a foreign server, run the following command: + +```sql +CREATE SERVER scalardb FOREIGN DATA WRAPPER scalardb_fdw OPTIONS ( + config_file_path '/path/to/scalardb.properties' +); +``` + +#### 4. Create user mapping + +To create user mapping, run the following command: + +```sql +CREATE USER MAPPING FOR PUBLIC SERVER scalardb; +``` + +#### 5. Create a foreign table + +To create a foreign table, run the following command: + +```sql +CREATE FOREIGN TABLE sample_table ( + pk int, + ck1 int, + ck2 int, + boolean_col boolean, + bigint_col bigint, + float_col double precision, + double_col double precision, + text_col text, + blob_col bytea +) SERVER scalardb OPTIONS ( + namespace 'ns', + table_name 'sample_table' +); +``` + +#### 6. Run a query + +To run a query, run the following command: + +```sql +select * from sample_table; +``` + +### Available options + +You can set the following options for ScalarDB FDW objects. + +#### `CREATE SERVER` + +You can set the following options on a ScalarDB foreign server object: + +| Name | Required | Type | Description | +| ------------------ | -------- | -------- | --------------------------------------------------------------- | +| `config_file_path` | **Yes** | `string` | The path to the ScalarDB config file. | +| `max_heap_size` | No | `string` | The maximum heap size of JVM. The format is the same as `-Xmx`. | + +#### `CREATE USER MAPPING` + +Currently, no options exist for `CREATE USER MAPPING`. + +#### `CREATE FOREIGN SERVER` + +The following options can be set on a ScalarDB foreign table object: + +| Name | Required | Type | Description | +| ------------ | -------- | -------- | ---------------------------------------------------------------- | +| `namespace` | **Yes** | `string` | The name of the namespace of the table in the ScalarDB instance. | +| `table_name` | **Yes** | `string` | The name of the table in the ScalarDB instance. | + +### Data-type mapping + +| ScalarDB | PostgreSQL | +| -------- | ---------------- | +| BOOLEAN | boolean | +| INT | int | +| BIGINT | bigint | +| FLOAT | float | +| DOUBLE | double precision | +| TEXT | text | +| BLOB | bytea | + +## Testing + +This section describes how to test FDW for ScalarDB. + +### Set up a ScalarDB instance for testing + +Before testing FDW for ScalarDB, you must have a running ScalarDB instance that contains test data. You can set up the instance and load the test data by running the following commands: + +```console +./test/setup.sh +``` + +If you want to reset the instances, you can run the following command, then the above setup command again. + +```console +./test/cleanup.sh +``` + +### Run regression tests + +You can run regression tests by running the following command **after** you have installed the FDW extension. + +```console +make installcheck +``` + +## Limitations + +- This extension aims to enable analytical query processing on ScalarDB-managed databases. Therefore, this extension only supports reading data from ScalarDB. diff --git a/versioned_docs/version-3.X/scalardb-analytics-postgresql/schema-importer.mdx b/versioned_docs/version-3.X/scalardb-analytics-postgresql/schema-importer.mdx new file mode 100644 index 00000000..51457edc --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-analytics-postgresql/schema-importer.mdx @@ -0,0 +1,66 @@ +--- +tags: + - Community +displayed_sidebar: docsEnglish +--- + +# Schema Importer + +Schema Importer is a CLI tool for automatically configuring PostgreSQL. By using this tool, your PostgreSQL database can have identical database objects, such as namespaces and tables, as your ScalarDB instance. + +Schema Importer reads the ScalarDB configuration file, retrieves the schemas of the tables defined in ScalarDB, and creates the corresponding foreign data wrapper external tables and views in that order. For more information, refer to [Getting Started with ScalarDB Analytics with PostgreSQL](getting-started.mdx). + +## Build Schema Importer + +You can build Schema Importer by using [Gradle](https://gradle.org/). To build Schema Importer, run the following command: + +```console +./gradlew build +``` + +You may want to build a fat JAR file so that you can launch Schema Importer by using `java -jar`. To build the fat JAR, run the following command: + + ```console + ./gradlew shadowJar + ``` + +After you build the fat JAR, you can find the fat JAR file in the `app/build/libs/` directory. + +## Run Schema Importer + +To run Schema Importer by using the fat JAR file, run the following command: + +```console +java -jar +``` +Available options are as follows: + +| Name | Required | Description | Default | +| --------------------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------ | +| `--config` | **Yes** | Path to the ScalarDB configuration file | | +| `--config-on-postgres-host` | No | Path to the ScalarDB configuration file on the PostgreSQL-running host | The same value as `--config` will be used. | +| `--namespace`, `-n` | **Yes** | Namespaces to import into the analytics instance. You can specify the `--namespace` option multiple times if you have two or more namespaces. | | +| `--host` | No | PostgreSQL host | localhost | +| `--port` | No | PostgreSQL port | 5432 | +| `--database` | No | PostgreSQL port | postgres | +| `--user` | No | PostgreSQL user | postgres | +| `--password` | No | PostgreSQL password | | +| `--debug` | No | Enable debug mode | | + + +## Test Schema Importer + +To test Schema Importer, run the following command: + +```console +./gradlew test +``` + +## Build a Docker image of Schema Importer + + +To build a Docker image of Schema Importer, run the following command, replacing `` with the tag version of Schema Importer that you want to use: + +```console +docker build -t ghcr.io/scalar-labs/scalardb-analytics-postgresql-schema-importer: -f ./app/Dockerfile . +``` diff --git a/versioned_docs/version-3.X/scalardb-analytics/README.mdx b/versioned_docs/version-3.X/scalardb-analytics/README.mdx new file mode 100644 index 00000000..fa416e71 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-analytics/README.mdx @@ -0,0 +1,20 @@ +--- +tags: + - Enterprise Option +displayed_sidebar: docsEnglish +--- + +# ScalarDB Analytics + +import WarningLicenseKeyContact from '/src/components/en-us/_warning-license-key-contact.mdx'; + +**ScalarDB Analytics** is the analytical component of ScalarDB. Similar to ScalarDB, it unifies diverse data sources - ranging from RDBMSs like PostgreSQL and MySQL to NoSQL databases such as Cassandra and DynamoDB - into a single logical database. While ScalarDB focuses on operational workloads with strong transactional consistency across multiple databases, ScalarDB Analytics is optimized for analytical workloads. It supports a wide range of queries, including complex joins, aggregations, and window functions. ScalarDB Analytics operates seamlessly on both ScalarDB-managed data sources and non-ScalarDB-managed ones, enabling advanced analytical queries across various datasets. + +The current version of ScalarDB Analytics leverages **Apache Spark** as its execution engine. It provides a unified view of ScalarDB-managed and non-ScalarDB-managed data sources by utilizing a Spark custom catalog. Using ScalarDB Analytics, you can treat tables from these data sources as native Spark tables. This allows you to execute arbitrary Spark SQL queries seamlessly. For example, you can join a table stored in Cassandra with a table in PostgreSQL to perform a cross-database analysis with ease. + + + +## Further reading + +* For tutorials on how to use ScalarDB Analytics by using a sample dataset and application, see [Getting Started with ScalarDB Analytics](../scalardb-samples/scalardb-analytics-spark-sample/README.mdx). +* For supported Spark and Scala versions, see [Version Compatibility of ScalarDB Analytics with Spark](./run-analytical-queries.mdx#version-compatibility) diff --git a/versioned_docs/version-3.X/scalardb-analytics/deployment.mdx b/versioned_docs/version-3.X/scalardb-analytics/deployment.mdx new file mode 100644 index 00000000..b1f5a54f --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-analytics/deployment.mdx @@ -0,0 +1,219 @@ +--- +tags: + - Enterprise Option +displayed_sidebar: docsEnglish +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Deploy ScalarDB Analytics in Public Cloud Environments + +This guide explains how to deploy ScalarDB Analytics in a public cloud environment. ScalarDB Analytics currently uses Apache Spark as an execution engine and supports managed Spark services provided by public cloud providers, such as Amazon EMR and Databricks. + +## Supported managed Spark services and their application types + +ScalarDB Analytics supports the following managed Spark services and application types. + +| Public Cloud Service | Spark Driver | Spark Connect | JDBC | +| -------------------------- | ------------ | ------------- | ---- | +| Amazon EMR (EMR on EC2) | ✅ | ✅ | ❌ | +| Databricks | ✅ | ❌ | ✅ | + +## Configure and deploy + +Select your public cloud environment, and follow the instructions to set up and deploy ScalarDB Analytics. + + + + +

Use Amazon EMR

+ +You can use Amazon EMR (EMR on EC2) to run analytical queries through ScalarDB Analytics. For the basics to launch an EMR cluster, please refer to the [AWS EMR on EC2 documentation](https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan.html). + +

ScalarDB Analytics configuration

+ +To enable ScalarDB Analytics, you need to add the following configuration to the Software setting when you launch an EMR cluster. Be sure to replace the content in the angle brackets: + +```json +[ + { + "Classification": "spark-defaults", + "Properties": { + "spark.jars.packages": "com.scalar-labs:scalardb-analytics-spark-all-_:", + "spark.sql.catalog.": "com.scalar.db.analytics.spark.ScalarDbAnalyticsCatalog", + "spark.sql.extensions": "com.scalar.db.analytics.spark.extension.ScalarDbAnalyticsExtensions", + "spark.sql.catalog..license.cert_pem": "", + "spark.sql.catalog..license.key": "", + + // Add your data source configuration below + } + } +] +``` + +The following describes what you should change the content in the angle brackets to: + +- ``: The version of Spark. +- ``: The version of Scala used to build Spark. +- ``: The version of ScalarDB Analytics. +- ``: The name of the catalog. +- ``: The PEM encoded license certificate. +- ``: The license key. + +For more details, refer to [Set up ScalarDB Analytics in the Spark configuration](./run-analytical-queries.mdx#set-up-scalardb-analytics-in-the-spark-configuration). + +

Run analytical queries via the Spark driver

+ +After the EMR Spark cluster has launched, you can use ssh to connect to the primary node of the EMR cluster and run your Spark application. For details on how to create a Spark Driver application, refer to [Spark Driver application](./run-analytical-queries.mdx?spark-application-type=spark-driver-application#develop-a-spark-application). + +

Run analytical queries via Spark Connect

+ +You can use Spark Connect to run your Spark application remotely by using the EMR cluster that you launched. + +You first need to configure the Software setting in the same way as the [Spark Driver application](./run-analytical-queries.mdx?spark-application-type=spark-driver-application#develop-a-spark-application). You also need to set the following configuration to enable Spark Connect. + +
Allow inbound traffic for a Spark Connect server
+ +1. Create a security group to allow inbound traffic for a Spark Connect server. (Port 15001 is the default). +2. Allow the role of "Amazon EMR service role" to attach the security group to the primary node of the EMR cluster. +3. Add the security group to the primary node of the EMR cluster as "Additional security groups" when you launch the EMR cluster. + +
Launch the Spark Connect server via a bootstrap action
+ +1. Create a script file to launch the Spark Connect server as follows: + +```bash +#!/usr/bin/env bash + +set -eu -o pipefail + +cd /var/lib/spark + +sudo -u spark /usr/lib/spark/sbin/start-connect-server.sh --packages org.apache.spark:spark-connect_:,com.scalar-labs:scalardb-analytics-spark-all-_: +``` + +The following describes what you should change the content in the angle brackets to: + +- ``: The major and minor version of Scala that matches your Spark installation (such as 2.12 or 2.13) +- ``: The full version of Spark you are using (such as 3.5.3) +- ``: The major and minor version of Spark you are using (such as 3.5) +- ``: The version of ScalarDB Analytics + +2. Upload the script file to S3. +3. Allow the role of "EC2 instance profile for Amazon EMR" to access the uploaded script file in S3. +4. Add the uploaded script file to "Bootstrap actions" when you launch the EMR cluster. + +
Run analytical queries
+ +You can run your Spark application via Spark Connect from anywhere by using the remote URL of the Spark Connect server, which is `sc://:15001`. + +For details on how to create a Spark application by using Spark Connect, refer to [Spark Connect application](./run-analytical-queries.mdx?spark-application-type=spark-connect#develop-a-spark-application). + + + +

Use Databricks

+ +You can use Databricks to run analytical queries through ScalarDB Analytics. + +:::note + +Note that Databricks provides a modified version of Apache Spark, which works differently from the original Apache Spark. + +::: + +

Launch Databricks cluster

+ +ScalarDB Analytics works with all-purpose and jobs-compute clusters on Databricks. When you launch the cluster, you need to configure the cluster to enable ScalarDB Analytics as follows: + +1. Store the license certificate and license key in the cluster by using the Databricks CLI. + +```console +databricks secrets create-scope scalardb-analytics-secret # you can use any secret scope name +cat license_key.json | databricks secrets put-secret scalardb-analytics-secret license-key +cat license_cert.pem | databricks secrets put-secret scalardb-analytics-secret license-cert +``` + +:::note + +For details on how to install and use the Databricks CLI, refer to the [Databricks CLI documentation](https://docs.databricks.com/en/dev-tools/cli/index.html). + +::: + +2. Select "No isolation shared" for the cluster mode. (This is required. ScalarDB Analytics works only with this cluster mode.) +3. Select an appropriate Databricks runtime version that supports Spark 3.4 or later. +4. Configure "Advanced Options" > "Spark config" as follows, replacing `` with the name of the catalog that you want to use: + +``` +spark.sql.catalog. com.scalar.db.analytics.spark.ScalarDbAnalyticsCatalog +spark.sql.extensions com.scalar.db.analytics.spark.extension.ScalarDbAnalyticsExtensions +spark.sql.catalog..license.key {{secrets/scalardb-analytics-secret/license-key}} +spark.sql.catalog..license.cert_pem {{secrets/scalardb-analytics-secret/license-pem}} +``` + +:::note + +You also need to configure the data source. For details, refer to [Set up ScalarDB Analytics in the Spark configuration](./run-analytical-queries.mdx#set-up-scalardb-analytics-in-the-spark-configuration). + +::: + +:::note + +If you specified different secret names in the previous step, be sure to replace the secret names in the configuration above. + +::: + +5. Add the library of ScalarDB Analytics to the launched cluster as a Maven dependency. For details on how to add the library, refer to the [Databricks cluster libraries documentation](https://docs.databricks.com/en/libraries/cluster-libraries.html). + +

Run analytical queries via the Spark Driver

+ +You can run your Spark application on the properly configured Databricks cluster with Databricks Notebook or Databricks Jobs to access the tables in ScalarDB Analytics. To run the Spark application, you can migrate your Pyspark, Scala, or Spark SQL application to Databricks Notebook, or use Databricks Jobs to run your Spark application. ScalarDB Analytics works with task types for Notebook, Python, JAR, and SQL. + +For more details on how to use Databricks Jobs, refer to the [Databricks Jobs documentation](https://docs.databricks.com/en/jobs/index.html) + +

Run analytical queries via the JDBC driver

+ +Databricks supports JDBC to run SQL jobs on the cluster. You can use this feature to run your Spark application in SQL with ScalarDB Analytics by configuring extra settings as follows: + +1. Download the ScalarDB Analytics library JAR file from the Maven repository. +2. Upload the JAR file to the Databricks workspace. +3. Add the JAR file to the cluster as a library, instead of the Maven dependency. +4. Create an init script as follows, replacing `` with the path to your JAR file in the Databricks workspace: + +```bash +#!/bin/bash + +# Target directories +TARGET_DIRECTORIES=("/databricks/jars" "/databricks/hive_metastore_jars") +JAR_PATH=" + +# Copy the JAR file to the target directories +for TARGET_DIR in "${TARGET_DIRECTORIES[@]}"; do + mkdir -p "$TARGET_DIR" + cp "$JAR_PATH" "$TARGET_DIR/" +done +``` + +5. Upload the init script to the Databricks workspace. +6. Add the init script to the cluster to "Advanced Options" > "Init scripts" when you launch the cluster. + +After the cluster is launched, you can get the JDBC URL of the cluster in the "Advanced Options" > "JDBC/ODBC" tab on the cluster details page. + +To connect to the Databricks cluster by using JDBC, you need to add the Databricks JDBC driver to your application dependencies. For example, if you are using Gradle, you can add the following dependency to your `build.gradle` file: + +```groovy +implementation("com.databricks:databricks-jdbc:0.9.6-oss") +``` + +Then, you can connect to the Databricks cluster by using JDBC with the JDBC URL (``), as is common with JDBC applications. + +```java +Class.forName("com.databricks.client.jdbc.Driver"); +String url = ""; +Connection conn = DriverManager.getConnection(url) +``` + +For more details on how to use JDBC with Databricks, refer to the [Databricks JDBC Driver documentation](https://docs.databricks.com/en/integrations/jdbc/index.html). + + + diff --git a/versioned_docs/version-3.X/scalardb-analytics/design.mdx b/versioned_docs/version-3.X/scalardb-analytics/design.mdx new file mode 100644 index 00000000..e1f99d07 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-analytics/design.mdx @@ -0,0 +1,391 @@ +--- +tags: + - Enterprise Option +displayed_sidebar: docsEnglish +--- + +# ScalarDB Analytics Design + +import Tabs from "@theme/Tabs"; +import TabItem from "@theme/TabItem"; + +ScalarDB Analytics is the analytical component of ScalarDB. Similar to ScalarDB, it unifies diverse data sources—ranging from RDBMSs like PostgreSQL and MySQL to NoSQL databases like Cassandra and DynamoDB—into a single logical database. This enables you to perform analytical queries across multiple databases seamlessly. + +ScalarDB Analytics consists of two main components: a universal data catalog and a query engine: + +- **Universal data catalog.** The universal data catalog is a flexible metadata management system that handles multiple catalog spaces. Each catalog space provides an independent logical grouping of data sources and views, enabling organized management of diverse data environments. +- **Query engine.** The query engine executes queries against the universal data catalog. ScalarDB Analytics provides appropriate data connectors to interface with the underlying data sources. + +ScalarDB Analytics employs a decoupled architecture where the data catalog and query engine are separate components. This design allows for integration with various existing query engines through an extensible architecture. As a result, you can select different query engines to execute queries against the same data catalog based on your specific requirements. + +## Universal data catalog + +The universal data catalog is composed of several levels and is structured as follows: + +```mermaid +graph TD + C[Catalog] --> D[Data Source] + C[Catalog] --> D2[Data Source] + subgraph " " + D --> N[Namespace] + D --> N2[Namespace] + N --> T[Table] + N --> T2[Table] + T --> TC[Column] + T --> TC2[Column] + D2 + end + + C --> VN[View Namespace] + C --> VN2[View Namespace] + subgraph " " + VN --> V[View] + VN --> V2[View] + V --> VC[Column] + V --> VC2[Column] + VN2 + end +``` + +The following are definitions for those levels: + +- **Catalog** is a folder that contains all your data source information. For example, you might have one catalog called `analytics_catalog` for your analytics data and another called `operational_catalog` for your day-to-day operations. +- **Data source** represents each data source you connect to. For each data source, we store important information like: + - What kind of data source it is (PostgreSQL, Cassandra, etc.) + - How to connect to it (connection details and passwords) + - Special features the data source supports (like transactions) +- **Namespace** is like a subfolder within your data source that groups related tables together. In PostgreSQL these are called schemas, in Cassandra they're called keyspaces. You can have multiple levels of namespaces, similar to having folders within folders. +- **Table** is where your actual data lives. For each table, we keep track of: + - What columns it has + - What type of data each column can store + - Whether columns can be empty (null) +- **View namespace** is a special folder for views. Unlike regular namespaces that are tied to one data source, view namespaces can work with multiple data sources at once. +- **View** is like a virtual table that can: + - Show your data in a simpler way (like hiding technical columns in ScalarDB tables) + - Combine data from different sources using SQL queries + - Each view, like tables, has its own columns with specific types and rules about empty values. + +### Supported data types + +ScalarDB Analytics supports a wide range of data types across different data sources. The universal data catalog maps these data types to a common set of types to ensure compatibility and consistency across sources. The following list shows the supported data types in ScalarDB Analytics: + +- `BYTE` +- `SMALLINT` +- `INT` +- `BIGINT` +- `FLOAT` +- `DOUBLE` +- `DECIMAL` +- `TEXT` +- `BLOB` +- `BOOLEAN` +- `DATE` +- `TIME` +- `TIMESTAMP` +- `TIMESTAMPTZ` +- `DURATION` +- `INTERVAL` + +### Catalog information mappings by data source + +When registering a data source to ScalarDB Analytics, the catalog information of the data source, that is, namespaces, tables, and columns, are resolved and registered to the universal data catalog. To resolve the catalog information of the data source, a particular object on the data sources side are mapped to the universal data catalog object. This mapping is consists of two parts: catalog-level mappings and data-type mappings. In the following sections, we describe how ScalarDB Analytics maps the catalog level and data type from each data source into the universal data catalog. + +#### Catalog-level mappings + +The catalog-level mappings are the mappings of the namespace names, table names, and column names from the data sources to the universal data catalog. To see the catalog-level mappings in each data source, select a data source. + + + + The catalog information of ScalarDB is automatically resolved by ScalarDB Analytics. The catalog-level objects are mapped as follows: + + - The ScalarDB namespace is mapped to the namespace. Therefore, the namespace of the ScalarDB data source is always single level, consisting of only the namespace name. + - The ScalarDB table is mapped to the table. + - The ScalarDB column is mapped to the column. + + + + + The catalog information of PostgreSQL is automatically resolved by ScalarDB Analytics. The catalog-level objects are mapped as follows: + + - The PostgreSQL schema is mapped to the namespace. Therefore, the namespace of the PostgreSQL data source is always single level, consisting of only the schema name. + - Only user-defined schemas are mapped to namespaces. The following system schemas are ignored: + - `information_schema` + - `pg_catalog` + - The PostgreSQL table is mapped to the table. + - The PostgreSQL column is mapped to the column. + + + + The catalog information of MySQL is automatically resolved by ScalarDB Analytics. The catalog-level objects are mapped as follows: + + - The MySQL database is mapped to the namespace. Therefore, the namespace of the MySQL data source is always single level, consisting of only the database name. + - Only user-defined databases are mapped to namespaces. The following system databases are ignored: + - `mysql` + - `sys` + - `information_schema` + - `performance_schema` + - The MySQL table is mapped to the table. + - The MySQL column is mapped to the column. + + + + The catalog information of Oracle is automatically resolved by ScalarDB Analytics. The catalog-level objects are mapped as follows: + + - The Oracle schema is mapped to the namespace. Therefore, the namespace of the Oracle data source is always single level, consisting of only schema name. + - Only user-defined schemas are mapped to namespaces. The following system schemas are ignored: + - `ANONYMOUS` + - `APPQOSSYS` + - `AUDSYS` + - `CTXSYS` + - `DBSNMP` + - `DGPDB_INT` + - `DBSFWUSER` + - `DVF` + - `DVSYS` + - `GGSYS` + - `GSMADMIN_INTERNAL` + - `GSMCATUSER` + - `GSMROOTUSER` + - `GSMUSER` + - `LBACSYS` + - `MDSYS` + - `OJVMSYS` + - `ORDDATA` + - `ORDPLUGINS` + - `ORDSYS` + - `OUTLN` + - `REMOTE_SCHEDULER_AGENT` + - `SI_INFORMTN_SCHEMA` + - `SYS` + - `SYS$UMF` + - `SYSBACKUP` + - `SYSDG` + - `SYSKM` + - `SYSRAC` + - `SYSTEM` + - `WMSYS` + - `XDB` + - `DIP` + - `MDDATA` + - `ORACLE_OCM` + - `XS$NULL` + + + + The catalog information of SQL Server is automatically resolved by ScalarDB Analytics. The catalog-level objects are mapped as follows: + + - The SQL Server database and schema are mapped to the namespace together. Therefore, the namespace of the SQL Server data source is always two-level, consisting of the database name and the schema name. + - Only user-defined databases are mapped to namespaces. The following system databases are ignored: + - `sys` + - `guest` + - `INFORMATION_SCHEMA` + - `db_accessadmin` + - `db_backupoperator` + - `db_datareader` + - `db_datawriter` + - `db_ddladmin` + - `db_denydatareader` + - `db_denydatawriter` + - `db_owner` + - `db_securityadmin` + - Only user-defined schemas are mapped to namespaces. The following system schemas are ignored: + - `master` + - `model` + - `msdb` + - `tempdb` + - The SQL Server table is mapped to the table. + - The SQL Server column is mapped to the column. + + + + Since DynamoDB is schema-less, you need to specify the catalog information explicitly when registering a DynamoDB data source by using the following format JSON: + + ```json + { + "namespaces": [ + { + "name": "", + "tables": [ + { + "name": "", + "columns": [ + { + "name": "", + "type": "" + }, + ... + ] + }, + ... + ] + }, + ... + ] + } + ``` + + In the specified JSON, you can use any arbitrary namespace names, but the table names must match the table names in DynamoDB and column name and type must match field names and types in DynamoDB. + + + + +#### Data-type mappings + +The native data types of the underlying data sources are mapped to the data types in ScalarDB Analytics. To see the data-type mappings in each data source, select a data source. + + + + | **ScalarDB Data Type** | **ScalarDB Analytics Data Type** | + |:------------------------------|:---------------------------------| + | `BOOLEAN` | `BOOLEAN` | + | `INT` | `INT` | + | `BIGINT` | `BIGINT` | + | `FLOAT` | `FLOAT` | + | `DOUBLE` | `DOUBLE` | + | `TEXT` | `TEXT` | + | `BLOB` | `BLOB` | + | `DATE` | `DATE` | + | `TIME` | `TIME` | + | `TIMESTAMP` | `TIMESTAMP` | + | `TIMESTAMPTZ` | `TIMESTAMPTZ` | + + + | **PostgreSQL Data Type** | **ScalarDB Analytics Data Type** | + |:------------------------------|:---------------------------------| + | `integer` | `INT` | + | `bigint` | `BIGINT` | + | `real` | `FLOAT` | + | `double precision` | `DOUBLE` | + | `smallserial` | `SMALLINT` | + | `serial` | `INT` | + | `bigserial` | `BIGINT` | + | `char` | `TEXT` | + | `varchar` | `TEXT` | + | `text` | `TEXT` | + | `bpchar` | `TEXT` | + | `boolean` | `BOOLEAN` | + | `bytea` | `BLOB` | + | `date` | `DATE` | + | `time` | `TIME` | + | `time with time zone` | `TIME` | + | `time without time zone` | `TIME` | + | `timestamp` | `TIMESTAMP` | + | `timestamp with time zone` | `TIMESTAMPTZ` | + | `timestamp without time zone` | `TIMESTAMP` | + + + | **MySQL Data Type** | **ScalarDB Analytics Data Type** | + |:-----------------------|:---------------------------------| + | `bit` | `BOOLEAN` | + | `bit(1)` | `BOOLEAN` | + | `bit(x)` if *x >= 2* | `BLOB` | + | `tinyint` | `SMALLINT` | + | `tinyint(1)` | `BOOLEAN` | + | `boolean` | `BOOLEAN` | + | `smallint` | `SMALLINT` | + | `smallint unsigned` | `INT` | + | `mediumint` | `INT` | + | `mediumint unsigned` | `INT` | + | `int` | `INT` | + | `int unsigned` | `BIGINT` | + | `bigint` | `BIGINT` | + | `float` | `FLOAT` | + | `double` | `DOUBLE` | + | `real` | `DOUBLE` | + | `char` | `TEXT` | + | `varchar` | `TEXT` | + | `text` | `TEXT` | + | `binary` | `BLOB` | + | `varbinary` | `BLOB` | + | `blob` | `BLOB` | + | `date` | `DATE` | + | `time` | `TIME` | + | `datetime` | `TIMESTAMP` | + | `timestamp` | `TIMESTAMPTZ` | + + + | **Oracle Data Type** | **ScalarDB Analytics Data Type** | + |:-----------------------------------|:---------------------------------| + | `NUMBER` if *scale = 0* | `BIGINT` | + | `NUMBER` if *scale > 0* | `DOUBLE` | + | `FLOAT` if *precision ≤ 53* | `DOUBLE` | + | `BINARY_FLOAT` | `FLOAT` | + | `BINARY_DOUBLE` | `DOUBLE` | + | `CHAR` | `TEXT` | + | `NCHAR` | `TEXT` | + | `VARCHAR2` | `TEXT` | + | `NVARCHAR2` | `TEXT` | + | `CLOB` | `TEXT` | + | `NCLOB` | `TEXT` | + | `BLOB` | `BLOB` | + | `BOOLEAN` | `BOOLEAN` | + | `DATE` | `DATE` | + | `TIMESTAMP` | `TIMESTAMPTZ` | + | `TIMESTAMP WITH TIME ZONE` | `TIMESTAMPTZ` | + | `TIMESTAMP WITH LOCAL TIME ZONE` | `TIMESTAMP` | + | `RAW` | `BLOB` | + + + | **SQL Server Data Type** | **ScalarDB Analytics Data Type** | + |:---------------------------|:---------------------------------| + | `bit` | `BOOLEAN` | + | `tinyint` | `SMALLINT` | + | `smallint` | `SMALLINT` | + | `int` | `INT` | + | `bigint` | `BIGINT` | + | `real` | `FLOAT` | + | `float` | `DOUBLE` | + | `float(n)` if *n ≤ 24* | `FLOAT` | + | `float(n)` if *n ≥ 25* | `DOUBLE` | + | `binary` | `BLOB` | + | `varbinary` | `BLOB` | + | `char` | `TEXT` | + | `varchar` | `TEXT` | + | `nchar` | `TEXT` | + | `nvarchar` | `TEXT` | + | `ntext` | `TEXT` | + | `text` | `TEXT` | + | `date` | `DATE` | + | `time` | `TIME` | + | `datetime` | `TIMESTAMP` | + | `datetime2` | `TIMESTAMP` | + | `smalldatetime` | `TIMESTAMP` | + | `datetimeoffset` | `TIMESTAMPTZ` | + + + | **DynamoDB Data Type** | **ScalarDB Analytics Data Type** | + |:-------------------------|:---------------------------------| + | `Number` | `BYTE` | + | `Number` | `SMALLINT` | + | `Number` | `INT` | + | `Number` | `BIGINT` | + | `Number` | `FLOAT` | + | `Number` | `DOUBLE` | + | `Number` | `DECIMAL` | + | `String` | `TEXT` | + | `Binary` | `BLOB` | + | `Boolean` | `BOOLEAN` | + +:::warning + +It is important to ensure that the field values of `Number` types are parsable as a specified data type for ScalarDB Analytics. For example, if a column that corresponds to a `Number`-type field is specified as an `INT` type, its value must be an integer. If the value is not an integer, an error will occur when running a query. + +::: + + + + +## Query engine + +A query engine is an independent component along with the universal data catalog, which is responsible for executing queries against the data sources registered in the universal data catalog and returning the results to the user. ScalarDB Analytics does not currently provide a built-in query engine. Instead, it is designed to be integrated with existing query engines, normally provided as a plugin of the query engine. + +When you run a query, the ScalarDB Analytics query engine plugin works as follows: + +1. Fetches the catalog metadata by calling the universal data catalog API, like the data source location, the table object identifier, and the table schema. +2. Sets up the data source connectors to the data sources by using the catalog metadata. +3. Provides the query optimization information to the query engine based on the catalog metadata. +4. Reads the data from the data sources by using the data source connectors. + +ScalarDB Analytics manages these processes internally. You can simply run a query against the universal data catalog by using the query engine API in the same way that you would normally run a query. + +ScalarDB Analytics currently supports Apache Spark as its query engine. For details on how to use ScalarDB Analytics with Spark, see [Run Analytical Queries Through ScalarDB Analytics](./run-analytical-queries.mdx). diff --git a/versioned_docs/version-3.X/scalardb-analytics/run-analytical-queries.mdx b/versioned_docs/version-3.X/scalardb-analytics/run-analytical-queries.mdx new file mode 100644 index 00000000..4f4b26aa --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-analytics/run-analytical-queries.mdx @@ -0,0 +1,453 @@ +--- +tags: + - Enterprise Option +displayed_sidebar: docsEnglish +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +# Run Analytical Queries Through ScalarDB Analytics + +This guide explains how to develop ScalarDB Analytics applications. For details on the architecture and design, see [ScalarDB Analytics Design](./design.mdx) + +ScalarDB Analytics currently uses Spark as an execution engine and provides a Spark custom catalog plugin to provide a unified view of ScalarDB-managed and non-ScalarDB-managed data sources as Spark tables. This allows you to execute arbitrary Spark SQL queries seamlessly. + +## Preparation + +This section describes the prerequisites, setting up ScalarDB Analytics in the Spark configuration, and adding the ScalarDB Analytics dependency. + +### Prerequisites + +ScalarDB Analytics works with Apache Spark 3.4 or later. If you don't have Spark installed yet, please download the Spark distribution from [Apache's website](https://spark.apache.org/downloads.html). + +:::note + +Apache Spark are built with either Scala 2.12 or Scala 2.13. ScalarDB Analytics supports both versions. You need to be sure which version you are using so that you can select the correct version of ScalarDB Analytics later. You can refer to [Version Compatibility](#version-compatibility) for more details. + +::: + +### Set up ScalarDB Analytics in the Spark configuration + +The following sections describe all available configuration options for ScalarDB Analytics. These configurations control: + +- How ScalarDB Analytics integrates with Spark +- How data sources are connected and accessed +- How license information is provided + +For example configurations in a practical scenario, see [the sample application configuration](../scalardb-samples/scalardb-analytics-spark-sample/README.mdx#scalardb-analytics-configuration). + +#### Spark plugin configurations + +| Configuration Key | Required | Description | +|:-----------------|:---------|:------------| +| `spark.jars.packages` | No | A comma-separated list of Maven coordinates for the required dependencies. User need to include the ScalarDB Analytics package you are using, otherwise, specify it as the command line argument when running the Spark application. For details about the Maven coordinates of ScalarDB Analytics, refer to [Add ScalarDB Analytics dependency](#add-the-scalardb-analytics-dependency). | +| `spark.sql.extensions` | Yes | Must be set to `com.scalar.db.analytics.spark.extension.ScalarDbAnalyticsExtensions`. | +| `spark.sql.catalog.` | Yes | Must be set to `com.scalar.db.analytics.spark.ScalarDbAnalyticsCatalog`. | + +You can specify any name for ``. Be sure to use the same catalog name throughout your configuration. + +#### License configurations + +| Configuration Key | Required | Description | +| :--------------------------------------------------- | :------- | :---------------------------------------------------------------------------------------------------------------------------- | +| `spark.sql.catalog..license.key` | Yes | JSON string of the license key for ScalarDB Analytics | +| `spark.sql.catalog..license.cert_pem` | Yes | A string of PEM-encoded certificate of ScalarDB Analytics license. Either `cert_pem` or `cert_path` must be set. | +| `spark.sql.catalog..license.cert_path` | Yes | A path to the PEM-encoded certificate of ScalarDB Analytics license. Either `cert_pem` or `cert_path` must be set. | + +#### Data source configurations + +ScalarDB Analytics supports multiple types of data sources. Each type requires specific configuration parameters: + + + + +:::note + +ScalarDB Analytics supports ScalarDB as a data source. This table describes how to configure ScalarDB as a data source. + +::: + +| Configuration Key | Required | Description | +| :---------------------------------------------------------------------------- | :------- | :---------------------------------------------- | +| `spark.sql.catalog..data_source..type` | Yes | Always set to `scalardb` | +| `spark.sql.catalog..data_source..config_path` | Yes | The path to the configuration file for ScalarDB | + +:::tip + +You can use an arbitrary name for ``. + +::: + + + + +| Configuration Key | Required | Description | +| :------------------------------------------------------------------------- | :------- | :------------------------------------- | +| `spark.sql.catalog..data_source..type` | Yes | Always set to `mysql` | +| `spark.sql.catalog..data_source..host` | Yes | The host name of the MySQL server | +| `spark.sql.catalog..data_source..port` | Yes | The port number of the MySQL server | +| `spark.sql.catalog..data_source..username` | Yes | The username of the MySQL server | +| `spark.sql.catalog..data_source..password` | Yes | The password of the MySQL server | +| `spark.sql.catalog..data_source..database` | No | The name of the database to connect to | + +:::tip + +You can use an arbitrary name for ``. + +::: + + + + +| Configuration Key | Required | Description | +| :------------------------------------------------------------------------- | :------- | :--------------------------------------- | +| `spark.sql.catalog..data_source..type` | Yes | Always set to `postgresql` or `postgres` | +| `spark.sql.catalog..data_source..host` | Yes | The host name of the PostgreSQL server | +| `spark.sql.catalog..data_source..port` | Yes | The port number of the PostgreSQL server | +| `spark.sql.catalog..data_source..username` | Yes | The username of the PostgreSQL server | +| `spark.sql.catalog..data_source..password` | Yes | The password of the PostgreSQL server | +| `spark.sql.catalog..data_source..database` | Yes | The name of the database to connect to | + +:::tip + +You can use an arbitrary name for ``. + +::: + + + + +| Configuration Key | Required | Description | +| :----------------------------------------------------------------------------- | :------- | :------------------------------------ | +| `spark.sql.catalog..data_source..type` | Yes | Always set to `oracle` | +| `spark.sql.catalog..data_source..host` | Yes | The host name of the Oracle server | +| `spark.sql.catalog..data_source..port` | Yes | The port number of the Oracle server | +| `spark.sql.catalog..data_source..username` | Yes | The username of the Oracle server | +| `spark.sql.catalog..data_source..password` | Yes | The password of the Oracle server | +| `spark.sql.catalog..data_source..service_name` | Yes | The service name of the Oracle server | + +:::tip + +You can use an arbitrary name for ``. + +::: + + + + +| Configuration Key | Required | Description | +| :------------------------------------------------------------------------- | :------- | :----------------------------------------------------------------------------------------------------- | +| `spark.sql.catalog..data_source..type` | Yes | Always set to `sqlserver` or `mssql` | +| `spark.sql.catalog..data_source..host` | Yes | The host name of the SQL Server server | +| `spark.sql.catalog..data_source..port` | Yes | The port number of the SQL Server server | +| `spark.sql.catalog..data_source..username` | Yes | The username of the SQL Server server | +| `spark.sql.catalog..data_source..password` | Yes | The password of the SQL Server server | +| `spark.sql.catalog..data_source..database` | No | The name of the database to connect to | +| `spark.sql.catalog..data_source..secure` | No | Whether to use a secure connection to the SQL Server server. Set to `true` to use a secure connection. | + +:::tip + +You can use an arbitrary name for ``. + +::: + + + + +| Configuration Key | Required | Description | +|:---------------------------------------------------------------------------|:------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `spark.sql.catalog..data_source..type` | Yes | Always set to `dynamodb` | +| `spark.sql.catalog..data_source..region` | Either `region` or `endpoint` must be set | The AWS region of the DynamoDB instance | +| `spark.sql.catalog..data_source..endpoint` | Either `region` or `endpoint` must be set | The AWS endpoint of the DynamoDB instance | +| `spark.sql.catalog..data_source..schema` | Yes | A JSON object representing the schema of the catalog. For details on the format, see [Catalog-level mappings](./design.mdx#catalog-level-mappings). | + + +:::tip + +You can use an arbitrary name for ``. + +::: + + + + +#### Example configuration + +Below is an example configuration for ScalarDB Analytics that demonstrates how to set up a catalog named `scalardb` with multiple data sources: + +```conf +# Spark plugin configurations +spark.jars.packages com.scalar-labs:scalardb-analytics-spark-all-_: +spark.sql.extensions com.scalar.db.analytics.spark.extension.ScalarDbAnalyticsExtensions +spark.sql.catalog.scalardb com.scalar.db.analytics.spark.ScalarDbAnalyticsCatalog + +# License configurations +spark.sql.catalog.scalardb.license.key +spark.sql.catalog.scalardb.license.cert_pem + +# Data source configurations +spark.sql.catalog.scalardb.data_source.scalardb.type scalardb +spark.sql.catalog.scalardb.data_source.scalardb.config_path /path/to/scalardb.properties + +spark.sql.catalog.scalardb.data_source.mysql_source.type mysql +spark.sql.catalog.scalardb.data_source.mysql_source.host localhost +spark.sql.catalog.scalardb.data_source.mysql_source.port 3306 +spark.sql.catalog.scalardb.data_source.mysql_source.username root +spark.sql.catalog.scalardb.data_source.mysql_source.password password +spark.sql.catalog.scalardb.data_source.mysql_source.database mydb +``` + +The following describes what you should change the content in the angle brackets to: + +- ``: The license key for ScalarDB Analytics +- ``: The PEM-encoded certificate of ScalarDB Analytics license +- ``: The major and minor version of Spark you are using (such as 3.4) +- ``: The major and minor version of Scala that matches your Spark installation (such as 2.12 or 2.13) +- ``: The version of ScalarDB Analytics + +### Add the ScalarDB Analytics dependency + +ScalarDB Analytics is hosted in the Maven Central Repository. The name of the package is `scalardb-analytics-spark-all-_:`, where: + +- ``: The major and minor version of Spark you are using (such as 3.4) +- ``: The major and minor version of Scala that matches your Spark installation (such as 2.12 or 2.13) +- ``: The version of ScalarDB Analytics + +For details about version compatibility, refer to [Version Compatibility](#version-compatibility). + +You can add this dependency to your project by configuring the build settings of your project. For example, if you are using Gradle, you can add the following to your `build.gradle` file: + +```groovy +dependencies { + implementation 'com.scalar-labs:scalardb-analytics-spark-all-_:' +} +``` + +:::note + +If you want bundle your application in a single fat JAR file by using plugins like Gradle Shadow plugin or Maven Shade plugin, you need to exclude ScalarDB Analytics from the fat JAR file by choosing the appropriate configuration, such as `provided` or `shadow`, depending on the plugin you are using. + +::: + +## Develop a Spark application + +In this section, you will learn how to develop a Spark application that uses ScalarDB Analytics in Java. + +There are three ways to develop Spark applications with ScalarDB Analytics: + +1. **Spark driver application**: A traditional Spark application that runs within the cluster +2. **Spark Connect application**: A remote application that uses the Spark Connect protocol +3. **JDBC application**: A remote application that uses the JDBC interface + +:::note + +Depending on your environment, you may not be able to use all the methods mentioned above. For details about supported features and deployment options, refer to [Supported managed Spark services and their application types](./deployment.mdx#supported-managed-spark-services-and-their-application-types). + +::: + +With all these methods, you can refer to tables in ScalarDB Analytics using the same table identifier format. For details about how ScalarDB Analytics maps catalog information from data sources, refer to [Catalog information mappings by data source](./design.mdx#catalog-information-mappings-by-data-source). + + + + +You can use a commonly used `SparkSession` class for ScalarDB Analytics. Additionally, you can use any type of cluster deployment that Spark supports, such as YARN, Kubernetes, standalone, or local mode. + +To read data from tables in ScalarDB Analytics, you can use the `spark.sql` or `spark.read.table` function in the same way as when reading a normal Spark table. + +First, you need to set up your Java project. For example, if you are using Gradle, you can add the following to your `build.gradle` file: + +```groovy +dependencies { + implementation 'com.scalar-labs:scalardb-analytics-spark-_:' +} +``` + +Below is an example of a Spark Driver application: + +```java +import org.apache.spark.sql.SparkSession; + +public class MyApp { + public static void main(String[] args) { + // Create a SparkSession + try (SparkSession spark = SparkSession.builder().getOrCreate()) { + // Read data from a table in ScalarDB Analytics + spark.sql("SELECT * FROM my_catalog.my_data_source.my_namespace.my_table").show(); + } + } +} +``` + +Then, you can build and run your application by using the `spark-submit` command. + +:::note + +You may need to build a fat JAR file for your application, as is usual for normal Spark applications. + +::: + +```console +spark-submit --class MyApp --master local[*] my-spark-application-all.jar +``` + +:::tip + +You can also use other CLI tools that Spark provides, such as `spark-sql` and `spark-shell`, to interact with ScalarDB Analytics tables. + +::: + + + + +You can use [Spark Connect](https://spark.apache.org/spark-connect/) to interact with ScalarDB Analytics. By using Spark Connect, you can access a remote Spark cluster and read data in the same way as a Spark Driver application. The following briefly describes how to use Spark Connect. + +First, you need to start a Spark Connect server in the remote Spark cluster by running the following command: + +```console +./sbin/start-connect-server.sh --packages org.apache.spark:spark-connect_:,com.scalar-labs:scalardb-analytics-spark-all-_: +``` + +The following describes what you should change the content in the angle brackets to: + +- ``: The major and minor version of Scala that matches your Spark installation (such as 2.12 or 2.13) +- ``: The full version of Spark you are using (such as 3.5.3) +- ``: The major and minor version of Spark you are using (such as 3.5) +- ``: The version of ScalarDB Analytics + +:::note + +The versions of the packages must match the versions of Spark and ScalarDB Analytics that you are using. + +::: + +You also need to include the Spark Connect client package in your application. For example, if you are using Gradle, you can add the following to your `build.gradle` file: + +```kotlin +implementation("org.apache.spark:spark-connect-client-jvm_2.12:3.5.3") +``` + +Then, you can write a Spark Connect client application to connect to the server and read data. + +```java +import org.apache.spark.sql.SparkSession; + +public class MyApp { + public static void main(String[] args) { + try (SparkSession spark = SparkSession.builder() + .remote("sc://:") + .getOrCreate()) { + + // Read data from a table in ScalarDB Analytics + spark.sql("SELECT * FROM my_catalog.my_data_source.my_namespace.my_table").show(); + } + } +} +``` + +You can run your Spark Connect client application as a normal Java application by running the following command: + +```console +java -jar my-spark-connect-client.jar +``` + +For details about how you can use Spark Connect, refer to the [Spark Connect documentation](https://spark.apache.org/docs/latest/spark-connect-overview.html). + + + + +Unfortunately, Spark Thrift JDBC server does not support the Spark features that are necessary for ScalarDB Analytics, so you cannot use JDBC to read data from ScalarDB Analytics in your Apache Spark environment. JDBC application is referred to here because some managed Spark services provide different ways to interact with a Spark cluster via the JDBC interface. For more details, refer to [Supported application types](./deployment.mdx#supported-managed-spark-services-and-their-application-types). + + + + +## Catalog information mapping + +ScalarDB Analytics manages its own catalog, containing data sources, namespaces, tables, and columns. That information is automatically mapped to the Spark catalog. In this section, you will learn how ScalarDB Analytics maps its catalog information to the Spark catalog. + +For details about how information in the raw data sources is mapped to the ScalarDB Analytics catalog, refer to [Catalog information mappings by data source](./design.mdx#catalog-information-mappings-by-data-source). + +### Catalog level mapping + +Each catalog level object in the ScalarDB Analytics catalog is mapped to a Spark catalog. The following table shows how the catalog levels are mapped: + +#### Data source tables + +Tables from data sources in the ScalarDB Analytics catalog are mapped to Spark tables. The following format is used to represent the identity of the Spark tables that correspond to ScalarDB Analytics tables: + +```console +... +``` + +The following describes what you should change the content in the angle brackets to: + +- ``: The name of the catalog. +- ``: The name of the data source. +- ``: The names of the namespaces. If the namespace names are multi-level, they are concatenated with a dot (`.`) as the separator. +- ``: The name of the table. + +For example, if you have a ScalarDB catalog named `my_catalog` that contains a data source named `my_data_source` and a schema named `my_schema`, you can refer to the table named `my_table` in that schema as `my_catalog.my_data_source.my_schema.my_table`. + +#### Views + +Views in ScalarDB Analytics are provided as tables in the Spark catalog, not views. The following format is used to represent the identity of the Spark tables that correspond to ScalarDB Analytics views: + +```console +.view.. +``` + +The following describes what you should change the content in the angle brackets to: + +- ``: The name of the catalog. +- ``: The names of the view namespaces. If the view namespace names are multi-level, they are concatenated with a dot (`.`) as the separator. +- ``: The name of the view. + +For example, if you have a ScalarDB catalog named `my_catalog` and a view namespace named `my_view_namespace`, you can refer to the view named `my_view` in that namespace as `my_catalog.view.my_view_namespace.my_view`. + +:::note + +`view` is prefixed to avoid conflicts with the data source table identifier. + +::: + +##### WAL-interpreted views + +As explained in [ScalarDB Analytics Design](./design.mdx), ScalarDB Analytics provides a functionality called WAL-interpreted views, which is a special type of views. These views are automatically created for tables of ScalarDB data sources to provide a user-friendly view of the data by interpreting WAL-metadata in the tables. + +Since the data source name and the namespace names of the original ScalarDB tables are used as the view namespace names for WAL-interpreted views, if you have a ScalarDB table named `my_table` in a namespace named `my_namespace` of a data source named `my_data_source`, you can refer to the WAL-interpreted view of the table as `my_catalog.view.my_data_source.my_namespace.my_table`. + +### Data-type mapping + +ScalarDB Analytics maps data types in its catalog to Spark data types. The following table shows how the data types are mapped: + +| ScalarDB Data Type | Spark Data Type | +| :----------------- | :----------------- | +| `BYTE` | `Byte` | +| `SMALLINT` | `Short` | +| `INT` | `Integer` | +| `BIGINT` | `Long` | +| `FLOAT` | `Float` | +| `DOUBLE` | `Double` | +| `DECIMAL` | `Decimal` | +| `TEXT` | `String` | +| `BLOB` | `Binary` | +| `BOOLEAN` | `Boolean` | +| `DATE` | `Date` | +| `TIME` | `TimestampNTZ` | +| `TIMESTAMP` | `TimestampNTZ` | +| `TIMESTAMPTZ` | `Timestamp` | +| `DURATION` | `CalendarInterval` | +| `INTERVAL` | `CalendarInterval` | + +## Version compatibility + +Since Spark and Scala may be incompatible among different minor versions, ScalarDB Analytics offers different artifacts for various Spark and Scala versions, named in the format `scalardb-analytics-spark-all-_`. Make sure that you select the artifact matching the Spark and Scala versions you're using. For example, if you're using Spark 3.5 with Scala 2.13, you must specify `scalardb-analytics-spark-all-3.5_2.13`. + +Regarding the Java version, ScalarDB Analytics supports Java 8 or later. + +The following is a list of Spark and Scalar versions supported by each version of ScalarDB Analytics. + +| ScalarDB Analytics Version | ScalarDB Version | Spark Versions Supported | Scala Versions Supported | Minimum Java Version | +|:---------------------------|:-----------------|:-------------------------|:-------------------------|:---------------------| +| 3.16 | 3.16 | 3.5, 3.4 | 2.13, 2.12 | 8 | +| 3.15 | 3.15 | 3.5, 3.4 | 2.13, 2.12 | 8 | diff --git a/versioned_docs/version-3.X/scalardb-benchmarks/README.mdx b/versioned_docs/version-3.X/scalardb-benchmarks/README.mdx new file mode 100644 index 00000000..010d3e9d --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-benchmarks/README.mdx @@ -0,0 +1,236 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# ScalarDB Benchmarking Tools + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +This tutorial describes how to run benchmarking tools for ScalarDB. Database benchmarking is helpful for evaluating how databases perform against a set of standards. + +## Benchmark workloads + +- TPC-C +- YCSB (Workloads A, C, and F) +- Multi-storage YCSB (Workloads C and F) + - This YCSB variant is for a multi-storage environment that uses ScalarDB. + - Workers in a multi-storage YCSB execute the same number of read and write operations in two namespaces: `ycsb_primary` and `ycsb_secondary`. + +## Prerequisites + +- One of the following Java Development Kits (JDKs): + - [Oracle JDK](https://www.oracle.com/java/technologies/downloads/) LTS version 8 + - [OpenJDK](https://openjdk.org/install/) LTS version 8 +- Gradle +- [Kelpie](https://github.com/scalar-labs/kelpie) + - Kelpie is a framework for performing end-to-end testing, such as system benchmarking and verification. Get the latest version from [Kelpie Releases](https://github.com/scalar-labs/kelpie), and unzip the archive file. +- A client to run the benchmarking tools +- A target database + - For a list of databases that ScalarDB supports, see [Databases](../requirements.mdx#databases). + +:::note + +Currently, only JDK 8 can be used when running the benchmarking tools. + +::: + +## Set up the benchmarking tools + +The following sections describe how to set up the benchmarking tools. + +### Clone the ScalarDB benchmarks repository + +Open **Terminal**, then clone the ScalarDB benchmarks repository by running the following command: + +```console +git clone https://github.com/scalar-labs/scalardb-benchmarks +``` + +Then, go to the directory that contains the benchmarking files by running the following command: + +```console +cd scalardb-benchmarks +``` + +### Build the tools + +To build the benchmarking tools, run the following command: + +```console +./gradlew shadowJar +``` + +### Load the schema + +Before loading the initial data, the tables must be defined by using the [ScalarDB Schema Loader](../schema-loader.mdx). You can download the ScalarDB Schema Loader on the [ScalarDB Releases](https://github.com/scalar-labs/scalardb/releases) page. Select the Schema Loader based on how you access ScalarDB: +- **Using the ScalarDB Core library (Community edition)?:** Choose `scalardb-schema-loader-.jar` for the version of ScalarDB that you're using. Then, save the `.jar` file in the `scalardb-benchmarks` root directory. +- **Using ScalarDB Cluster (Enterprise edition)?:** Choose `scalardb-cluster-schema-loader--all.jar` for the version of ScalarDB Cluster that you're using. Then, save the `.jar` file in the `scalardb-benchmarks` root directory. + +In addition, you need a properties file for accessing ScalarDB via the Java CRUD interface. For details about configuring the ScalarDB properties file, see [ScalarDB Configurations](../configurations.mdx) or [ScalarDB Cluster Client Configurations](../scalardb-cluster/developer-guide-for-scalardb-cluster-with-java-api.mdx#client-configurations). + +After applying the schema and configuring the properties file, select a benchmark and follow the instructions to create the tables. + + + + To create tables for TPC-C benchmarking ([`tpcc-schema.json`](https://github.com/scalar-labs/scalardb-benchmarks/blob/master/tpcc-schema.json)), run the following command, replacing the contents in the angle brackets as described: + + ```console + java -jar scalardb-schema-loader-.jar --config -f tpcc-schema.json --coordinator + ``` + + If you are using ScalarDB Cluster, run the following command instead: + + ```console + java -jar scalardb-cluster-schema-loader--all.jar --config -f tpcc-schema.json --coordinator + ``` + + + To create tables for YCSB benchmarking ([`ycsb-schema.json`](https://github.com/scalar-labs/scalardb-benchmarks/blob/master/ycsb-schema.json)), run the following command, replacing the contents in the angle brackets as described: + + ```console + java -jar scalardb-schema-loader-.jar --config -f ycsb-schema.json --coordinator + ``` + + If you are using ScalarDB Cluster, run the following command instead: + + ```console + java -jar scalardb-cluster-schema-loader--all.jar --config -f ycsb-schema.json --coordinator + ``` + + + To create tables for multi-storage YCSB benchmarking ([`ycsb-multi-storage-schema.json`](https://github.com/scalar-labs/scalardb-benchmarks/blob/master/ycsb-multi-storage-schema.json)), run the following command, replacing the contents in the angle brackets as described: + + ```console + java -jar scalardb-schema-loader-.jar --config -f ycsb-multi-storage-schema.json --coordinator + ``` + + If you are using ScalarDB Cluster, run the following command instead: + + ```console + java -jar scalardb-cluster-schema-loader--all.jar --config -f ycsb-multi-storage-schema.json --coordinator + ``` + + + +### Prepare a benchmarking configuration file + +To run a benchmark, you must first prepare a benchmarking configuration file. The configuration file requires at least the locations of the workload modules to run and the database configuration. + +The following is an example configuration for running the TPC-C benchmark. The ScalarDB properties file specified for `config_file` should be the properties file that you created as one of the steps in [Load the schema](#load-the-schema). + +:::note + +Alternatively, instead of using the ScalarDB properties file, you can specify each database configuration item in the `.toml` file. If `config_file` is specified, all other configurations under `[database_config]` will be ignored even if they are uncommented. + +::: + +```toml +[modules] +[modules.preprocessor] +name = "com.scalar.db.benchmarks.tpcc.TpccLoader" +path = "./build/libs/scalardb-benchmarks-all.jar" +[modules.processor] +name = "com.scalar.db.benchmarks.tpcc.TpccBench" +path = "./build/libs/scalardb-benchmarks-all.jar" +[modules.postprocessor] +name = "com.scalar.db.benchmarks.tpcc.TpccReporter" +path = "./build/libs/scalardb-benchmarks-all.jar" + +[database_config] +config_file = "" +#contact_points = "localhost" +#contact_port = 9042 +#username = "cassandra" +#password = "cassandra" +#storage = "cassandra" +``` + +You can define parameters to pass to modules in the configuration file. For details, see the sample configuration files below and available parameters in [Common parameters](#common-parameters): + +- **TPC-C:** [`tpcc-benchmark-config.toml`](https://github.com/scalar-labs/scalardb-benchmarks/blob/master/tpcc-benchmark-config.toml) +- **YCSB:** [`ycsb-benchmark-config.toml`](https://github.com/scalar-labs/scalardb-benchmarks/blob/master/ycsb-benchmark-config.toml) +- **Multi-storage YCSB:** [`ycsb-multi-storage-benchmark-config.toml`](https://github.com/scalar-labs/scalardb-benchmarks/blob/master/ycsb-multi-storage-benchmark-config.toml) + +## Run a benchmark + +Select a benchmark, and follow the instructions to run the benchmark. + + + + To run the TPC-C benchmark, run the following command, replacing `` with the path to the Kelpie directory: + + ```console + //bin/kelpie --config tpcc-benchmark-config.toml + ``` + + + To run the YCSB benchmark, run the following command, replacing `` with the path to the Kelpie directory: + + ```console + //bin/kelpie --config ycsb-benchmark-config.toml + ``` + + + To run the multi-storage YCSB benchmark, run the following command, replacing `` with the path to the Kelpie directory: + + ```console + //bin/kelpie --config ycsb-multi-storage-benchmark-config.toml + ``` + + + +In addition, the following options are available: + +- `--only-pre`. Only loads the data. +- `--only-process`. Only runs the benchmark. +- `--except-pre` Runs a job without loading the data. +- `--except-process`. Runs a job without running the benchmark. + +## Common parameters + +| Name | Description | Default | +|:---------------|:--------------------------------------------------------|:----------| +| `concurrency` | Number of threads for benchmarking. | `1` | +| `run_for_sec` | Duration of benchmark (in seconds). | `60` | +| `ramp_for_sec` | Duration of ramp-up time before benchmark (in seconds). | `0` | + +## Workload-specific parameters + +Select a benchmark to see its available workload parameters. + + + + | Name | Description | Default | + |:-----------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:----------| + | `num_warehouses` | Number of warehouses (scale factor) for benchmarking. | `1` | + | `load_concurrency` | Number of threads for loading. | `1` | + | `load_start_warehouse` | Start ID of loading warehouse. This option can be useful with `--skip-item-load` when loading large-scale data with multiple clients or adding additional warehouses. | `1` | + | `load_end_warehouse` | End ID of loading warehouse. You can use either `--num-warehouses` or `--end-warehouse` to specify the number of loading warehouses. | `1` | + | `skip_item_load` | Whether or not to skip loading item table. | `false` | + | `use_table_index` | Whether or not to use a generic table-based secondary index instead of ScalarDB's secondary index. | `false` | + | `np_only` | Run benchmark with only new-order and payment transactions (50% each). | `false` | + | `rate_new_order` | Percentage of new-order transactions. When specifying this percentage based on your needs, you must specify the percentages for all other rate parameters. In that case, the total of all rate parameters must equal 100 percent. | N/A | + | `rate_payment` | Percentage of payment transactions. When specifying this percentage based on your needs, you must specify the percentages for all other rate parameters. In that case, the total of all rate parameters must equal 100 percent. | N/A | + | `rate_order_status` | Percentage of order-status transactions. When specifying this percentage based on your needs, you must specify the percentages for all other rate parameters. In that case, the total of all rate parameters must equal 100 percent. | N/A | + | `rate_delivery` | Percentage of delivery transactions. When specifying this percentage based on your needs, you must specify the percentages for all other rate parameters. In that case, the total of all rate parameters must equal 100 percent. | N/A | + | `rate_stock_level` | Percentage of stock-level transactions. When specifying this percentage based on your needs, you must specify the percentages for all other rate parameters. In that case, the total of all rate parameters must equal 100 percent. | N/A | + | `backoff` | Sleep time in milliseconds inserted after a transaction is aborted due to a conflict. | `0` | + + + | Name | Description | Default | + |:------------------------|:----------------------------------------------------------------------------------|:----------------------------------------------| + | `load_concurrency` | Number of threads for loading. | `1` | + | `load_batch_size` | Number of put records in a single loading transaction. | `1` | + | `load_overwrite` | Whether or not to overwrite when loading records. | `false` | + | `ops_per_tx` | Number of operations in a single transaction. | `2` (Workloads A and C)
`1` (Workload F) | + | `record_count` | Number of records in the target table. | `1000` | + | `use_read_modify_write` | Whether or not to use read-modify-writes instead of blind writes in Workload A. | `false`[^rmw] | + + [^rmw]: The default value is `false` for `use_read_modify_write` since Workload A doesn't assume that the transaction reads the original record first. However, if you're using Consensus Commit as the transaction manager, you must set `use_read_modify_write` to `true`. This is because ScalarDB doesn't allow a blind write for an existing record. + + diff --git a/versioned_docs/version-3.X/scalardb-cluster-dotnet-client-sdk/common-reference.mdx b/versioned_docs/version-3.X/scalardb-cluster-dotnet-client-sdk/common-reference.mdx new file mode 100644 index 00000000..a71540f7 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster-dotnet-client-sdk/common-reference.mdx @@ -0,0 +1,194 @@ +--- +tags: + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# ScalarDB Cluster .NET Client SDK Reference + +This reference provides details on how the ScalarDB Cluster .NET Client SDK works. + +## Client configuration + +The client can be configured by using the following: + +- A settings file, like `appsettings.json` or a custom JSON file +- Environment variables +- The `ScalarDbOptions` object + +If you use the SDK with ASP.NET Core, you can configure an app in more ways. For details, see [Configuration in ASP.NET Core](https://learn.microsoft.com/en-us/aspnet/core/fundamentals/configuration/?view=aspnetcore-8.0). + +For a list of options that you can configure, see [Available options](common-reference.mdx#available-options). + +### Using a settings file + +The SDK supports both the standard `appsettings.json` and custom JSON setting files. To configure the client in a JSON file, add the `ScalarDbOptions` section and configure the options that you need. For example: + +```json +{ + "ScalarDbOptions": { + "Address": "http://localhost:60053", + "HopLimit": 10 + } +} +``` + +Then, create a configured `TransactionFactory` object as follows: + +```c# +// If appsettings.json is used, call the Create() method without parameters. +var factory = TransactionFactory.Create(); + +// Or, if a custom file is used, call the Create() method that is passed in the path to the custom file as a parameter. +factory = TransactionFactory.Create("scalardb-options.json"); +``` + +If you use the SDK with ASP.NET Core, the settings from `appsettings.json` will be applied automatically when the registered transaction managers and/or `ScalarDbContext` are created. If you want to use a custom JSON file, add it to the configuration framework as follows: + +```c# +var builder = WebApplication.CreateBuilder(args); + +// ... + +builder.Configuration.AddJsonFile("scalardb-options.json"); +``` + +:::warning + +Because the custom JSON file is applied after all standard configuration providers, the values from the custom file will override values from other sources. + +::: + +### Using environment variables + +To configure the client to use environment variables, you can use the prefix `ScalarDbOptions__`. For example: + +```console +export ScalarDbOptions__Address="http://localhost:60053" +export ScalarDbOptions__HopLimit=10 +``` + +:::warning + +Values from environment variables will override values from settings files. + +::: + +### Using the `ScalarDbOptions` object + +You can configure the client at runtime by using the `ScalarDbOptions` object as follows: + +```c# +var options = new ScalarDbOptions() +{ + Address = "http://localhost:60053", + HopLimit = 10 +}; + +var factory = TransactionFactory.Create(options); +``` + +You can also initialize the `ScalarDbOptions` object with values from JSON files and/or environment variables, and then set any remaining values at runtime as follows: + +```c# +// If appsettings.json is used, call the Load() method without parameters. +var options = ScalarDbOptions.Load(); + +// Or, if a custom file is used, call the Load() method that is passed in the path to the custom file as a parameter. +options = ScalarDbOptions.Load("scalardb-options.json"); + +options.HopLimit = 10; + +var factory = TransactionFactory.Create(options); +``` + +If you use the SDK with ASP.NET Core, a lambda function of `AddScalarDb` and/or `AddScalarDbContext` can be used as follows: + +```c# +var builder = WebApplication.CreateBuilder(args); + +//... + +builder.Services.AddScalarDb(options => +{ + options.Address = "http://localhost:60053"; + options.HopLimit = 10; +}); + +builder.Services.AddScalarDbContext(options => +{ + options.Address = "http://localhost:60053"; + options.HopLimit = 10; +}); +``` + +By using this configuration, the `ScalarDbOptions` object that is passed to the lambda function (named `options` in the example above) is initialized with values from the JSON files, environment variables, and other sources. + +### Available options + +The following options are available: + +| Name | Description | Default | +|-----------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------| +| `Address` | **Required:** Address of the cluster in the following format: `://:`. ``: `https` if wire encryption (TLS) is enabled; `http` otherwise. ``: The FQDN or the IP address of the cluster. ``: The port number (`60053` by default) of the cluster. | - | +| `HopLimit` | Number of hops for a request to the cluster. The purpose of `HopLimit` is to prevent infinite loops within the cluster. Each time a request is forwarded to another cluster node, `HopLimit` decreases by one. If `HopLimit` reaches zero, the request will be rejected. | `3` | +| `RetryCount` | How many times a client can try to connect to the cluster if it's unavailable. | `10` | +| `AuthEnabled` | Whether authentication and authorization are enabled. | `false` | +| `Username` | Username for authentication and authorization. | | +| `Password` | Password for authentication. If this isn't set, authentication is conducted without a password. | | +| `AuthTokenExpirationTime` | Time after which the authentication token should be refreshed. If the time set for `AuthTokenExpirationTime` is greater than the expiration time on the cluster, the authentication token will be refreshed when an authentication error is received. If the authentication token is successfully refreshed, the authentication error won't be propagated to the client code. Instead, the operation that has failed with the authentication error will be retried automatically. If more than one operation is running in parallel, all these operations will fail once with the authentication error before the authentication token is refreshed. | `00:00:00` (The authentication token expiration time received from the cluster is used.) | +| `TlsRootCertPem` | Custom CA root certificate (PEM data) for TLS communication. | | +| `TlsRootCertPath` | File path to the custom CA root certificate for TLS communication. | | +| `TlsOverrideAuthority` | Custom authority for TLS communication. This doesn't change what host is actually connected. This is mainly intended for testing. For example, you can specify the hostname presented in the cluster's certificate (the `scalar.db.cluster.node.tls.cert_chain_path` parameter of the cluster). If there's more than one hostname in the cluster's certificate, only the first hostname will be checked. | | +| `LogSensitiveData` | If set to `true`, information like username, password, and authentication token will be logged as is without masking when logging gRPC requests and responses. | `false` | +| `GrpcRequestTimeout` | Timeout for gRPC requests. Internally, the timeout's value is used to calculate and set a deadline for each gRPC request to the cluster. If the set deadline is exceeded, the request is cancelled and `DeadlineExceededException` is thrown. If the timeout is set to `0`, no deadline will be set. | `00:01:00` | +| `GrpcMaxReceiveMessageSize` | The maximum message size in bytes that can be received by the client. When set to `0`, the message size is unlimited. | `4 MB` | +| `GrpcMaxSendMessageSize` | The maximum message size in bytes that can be sent from the client. When set to `0`, the message size is unlimited. | `0` (Unlimited) | + +## How ScalarDB column types are converted to and from .NET types + +When using [LINQ](getting-started-with-linq.mdx#set-up-classes) or extension methods for the [Transactional API](getting-started-with-scalardb-tables-as-csharp-classes.mdx#create-classes-for-all-scalardb-tables), [SQL API](getting-started-with-distributed-sql-transactions.mdx#execute-sql-queries), or [Administrative API](getting-started-with-scalardb-tables-as-csharp-classes.mdx#use-the-administrative-api), a column's value received from the cluster is automatically converted to a corresponding .NET type. Likewise, a value of a .NET property is automatically converted to a corresponding cluster's type when an object is being saved to the cluster. + +In the following table, you can find how types are converted: + +| ScalarDB type | .NET type | C# alias | +|---------------|----------------------------|----------| +| TEXT | System.String | string | +| INT | System.Int32 | int | +| BIGINT | System.Int64 | long | +| FLOAT | System.Single | float | +| DOUBLE | System.Double | double | +| BOOLEAN | System.Boolean | bool | +| BLOB | Google.Protobuf.ByteString | | +| DATE | NodaTime.LocalDate | | +| TIME | NodaTime.LocalTime | | +| TIMESTAMP | NodaTime.LocalDateTime | | +| TIMESTAMPTZ | NodaTime.Instant | | + +:::note + +The ScalarDB Cluster .NET Client SDK uses [Google.Protobuf](https://www.nuget.org/packages/Google.Protobuf) for `BLOB` type and [NodaTime](https://www.nuget.org/packages/NodaTime) for time-related types. + +::: + +:::warning + +The precision of time-related types in .NET is greater than supported by ScalarDB. Therefore, you should be careful when saving time-related values received from external sources. The ScalarDB Cluster .NET Client SDK includes `WithScalarDbPrecision` extension methods that you can use to lower the precision of time-related values in the following manner: + +```c# +using ScalarDB.Client.Extensions; + +// ... + +var updatedAt = Instant.FromDateTimeUtc(DateTime.UtcNow) + .WithScalarDbPrecision(); + +// using NodaTime to get current instant +updatedAt = clockInstance.GetCurrentInstant() + .WithScalarDbPrecision(); +``` + +For details about value ranges and precision in ScalarDB, see [Data-type mapping between ScalarDB and other databases](../schema-loader.mdx#data-type-mapping-between-scalardb-and-other-databases). + +::: diff --git a/versioned_docs/version-3.X/scalardb-cluster-dotnet-client-sdk/exception-handling.mdx b/versioned_docs/version-3.X/scalardb-cluster-dotnet-client-sdk/exception-handling.mdx new file mode 100644 index 00000000..1767360f --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster-dotnet-client-sdk/exception-handling.mdx @@ -0,0 +1,175 @@ +--- +tags: + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Exception Handling in the ScalarDB Cluster .NET Client SDK + +When executing a transaction, you will also need to handle exceptions properly. + +:::warning + +If you don't handle exceptions properly, you may face anomalies or data inconsistency. + +::: + +:::note + +The Transactional API is used in this example, but exceptions can be handled in the same way when using the SQL API or `ScalarDbContext`. + +::: + +The following sample code shows how to handle exceptions: + +```c# +using System.ComponentModel.DataAnnotations.Schema; +using ScalarDB.Client; +using ScalarDB.Client.DataAnnotations; +using ScalarDB.Client.Exceptions; +using ScalarDB.Client.Extensions; + +var options = new ScalarDbOptions { Address = "http://:"}; + +var factory = TransactionFactory.Create(options); +using var manager = factory.GetTransactionManager(); + +var retryCount = 0; +TransactionException? lastException = null; + +while (true) +{ + if (retryCount++ > 0) + { + // Retry the transaction three times maximum in this sample code + if (retryCount > 3) + // Throw the last exception if the number of retries exceeds the maximum + throw lastException!; + + // Sleep 100 milliseconds before retrying the transaction in this sample code + await Task.Delay(100); + } + + // Begin a transaction + var tran = await manager.BeginAsync(); + try + { + // Execute CRUD operations in the transaction + var getKeys = new Dictionary { { nameof(Item.Id), 1 } }; + var result = await tran.GetAsync(getKeys); + + var scanKeys = new Dictionary { { nameof(Item.Id), 1 } }; + await foreach (var item in tran.ScanAsync(scanKeys, null)) + Console.WriteLine($"{item.Id}, {item.Name}, {item.Price}"); + + await tran.InsertAsync(new Item { Id = 1, Name = "Watermelon", Price = 4500 }); + await tran.DeleteAsync(new Item { Id = 1 }); + + // Commit the transaction + await tran.CommitAsync(); + + return; + } + catch (UnsatisfiedConditionException) + { + // You need to handle `UnsatisfiedConditionException` only if a mutation operation specifies + // a condition. This exception indicates the condition for the mutation operation is not met. + // InsertAsync/UpdateAsync implicitlly sets IfNotExists/IfExists condition + + try + { + await tran.RollbackAsync(); + } + catch (TransactionException ex) + { + // Rolling back the transaction failed. As the transaction should eventually recover, you + // don't need to do anything further. You can simply log the occurrence here + Console.WriteLine($"Rollback error: {ex}"); + } + + // You can handle the exception here, according to your application requirements + + return; + } + catch (UnknownTransactionStatusException) + { + // If you catch `UnknownTransactionStatusException` when committing the transaction, it + // indicates that the status of the transaction, whether it has succeeded or not, is + // unknown. In such a case, you need to check if the transaction is committed successfully + // or not and retry it if it failed. How to identify a transaction status is delegated to users + return; + } + catch (TransactionException ex) + { + // For other exceptions, you can try retrying the transaction. + + // For `TransactionConflictException` and `TransactionNotFoundException`, + // you can basically retry the transaction. However, for the other exceptions, + // the transaction may still fail if the cause of the exception is nontransient. + // In such a case, you will exhaust the number of retries and throw the last exception + + try + { + await tran.RollbackAsync(); + } + catch (TransactionException e) + { + // Rolling back the transaction failed. As the transaction should eventually recover, + // you don't need to do anything further. You can simply log the occurrence here + Console.WriteLine($"Rollback error: {e}"); + } + + lastException = ex; + } +} + +[Table("order_service.items")] +public class Item +{ + [PartitionKey] + [Column("item_id", Order = 0)] + public int Id { get; set; } + + [Column("name", Order = 1)] + public string Name { get; set; } = String.Empty; + + [Column("price", Order = 2)] + public int Price { get; set; } +} + +``` + +:::note + +In the sample code, the transaction is retried a maximum of three times and sleeps for 100 milliseconds before it is retried. You can choose a retry policy, such as exponential backoff, according to your application requirements. + +::: + +### Exception details + +The table below shows transaction exceptions that can occur when communicating with the cluster: + +| Exception | Operations | Description | +|-----------------------------------|--------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| AuthenticationErrorException | All | The authentication failed because of a wrong username and/or password when calling the cluster. | +| AuthorizationErrorException | Put, Insert, Update, Delete, Mutate, Execute, Administrative | The authorization failed because of a lack of permissions. | +| HopLimitExceededException | All | The hop limit was exceeded. This occurs when the routing information between cluster nodes is inconsistent. The error is usually resolved in a short amount of time, so you can retry the transaction from the beginning after some time has passed since encountering this error. | +| IllegalArgumentException | All | The argument in the request message is invalid. | +| IllegalStateException | All | The RPC was called in an invalid state. | +| InternalErrorException | All | The operation failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | +| TransactionConflictException | All except Begin, Join, Rollback | A transaction conflict occurred. If you encounter this error, please retry the transaction from the beginning. | +| TransactionNotFoundException | All except Begin, Join | The transaction associated with the specified transaction ID was not found. This error indicates that the transaction has expired or the routing information has been updated due to cluster topology changes. In this case, please retry the transaction from the beginning. | +| UnavailableException | All | ScalarDB Cluster is unavailable even after trying to connect multiple times. | +| UnknownTransactionStatusException | Commit | The status of the transaction is unknown (it is uncertain whether the transaction was successfully committed or not). In this situation, you need to check whether the transaction was successfully committed, and if not, to retry it. You are responsible for determining the transaction status. You may benefit from creating a transaction status table and updating it in conjunction with other application data. Doing so may help you determine the status of a transaction from the table itself. | +| UnsatisfiedConditionException | Put, Insert, Update, Delete, Mutate | The mutation condition is not satisfied. | + +If you encounter an exception, you should roll back the transaction, except in the case of `Begin`. After rolling back the transaction, you can retry the transaction from the beginning for the exceptions that can be resolved by retrying. + +Besides the exceptions listed above, you may encounter exceptions thrown by the gRPC library. In such cases, you can check the `RpcException` property for more information. + +Also, `ScalarDbContext` will throw a `TransactionException` type exception in the following cases: + +- If `BeginTransaction` or `JoinTransaction` were called when there was already an active transaction +- If `CommitTransaction` or `RollbackTransaction` were called without an active transaction +- If `PrepareTransaction` or `ValidateTransaction` were called without an active two-phase commit transaction diff --git a/versioned_docs/version-3.X/scalardb-cluster-dotnet-client-sdk/getting-started-with-admin-api.mdx b/versioned_docs/version-3.X/scalardb-cluster-dotnet-client-sdk/getting-started-with-admin-api.mdx new file mode 100644 index 00000000..c7a81560 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster-dotnet-client-sdk/getting-started-with-admin-api.mdx @@ -0,0 +1,128 @@ +--- +tags: + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Getting Started with the Administrative API in the ScalarDB Cluster .NET Client SDK + +The ScalarDB Cluster .NET Client SDK supports the Administrative API of ScalarDB Cluster. By using this API, you can manage ScalarDB Cluster from .NET applications. + +:::note + +Although we recommend using asynchronous methods as in the following examples, you can use synchronous methods instead. + +::: + +## Install the SDK + +Install the same major and minor version of the [SDK](https://www.nuget.org/packages/ScalarDB.Client) as ScalarDB Cluster into the .NET project. You can do this by using the built-in NuGet package manager, replacing `.` with the version that you're using: + +```console +dotnet add package ScalarDB.Client --version '..*' +``` + +## Create a settings file + +Create a `scalardb-options.json` file and add the following, replacing `` with the FQDN or the IP address, and `` with the port number (`60053` by default) of your cluster: + +```json +{ + "ScalarDbOptions": { + "Address": "http://:", + "HopLimit": 10 + } +} +``` + +For details about settings files and other ways to configure the client, see [Client configuration](common-reference.mdx#client-configuration). + +## Get a transaction manager + +You need to get an object for interacting with the Administrative API. To get the object, you can use `TransactionFactory` as follows: + +```c# +// Pass the path to the settings file created in the previous step. +var factory = TransactionFactory.Create("scalardb-options.json"); + +using var admin = factory.GetTransactionAdmin(); +``` + +## Manage ScalarDB Cluster + +The following operations can be performed by using the ScalarDB Cluster .NET Client SDK. + +### Create a new namespace + +```c# +await admin.CreateNamespaceAsync("ns", ifNotExists: true); +``` + +### Drop a namespace + +```c# +await admin.DropNamespaceAsync("ns", ifExists: true); +``` + +### Check if a namespace exists + +```c# +var namespaceExists = await admin.IsNamespacePresentAsync("ns"); +``` + +### Create a new table + +```c# +// ... +using ScalarDB.Client.Builders.Admin; +using ScalarDB.Client.Core; + +// ... + +var tableMetadata = + new TableMetadataBuilder() + .AddPartitionKey("pk", DataType.Int) + .AddClusteringKey("ck", DataType.Double) + .AddSecondaryIndex("index", DataType.Float) + .AddColumn("ordinary", DataType.Text) + .Build(); + +await admin.CreateTableAsync("ns", "table_name", tableMetadata, ifNotExists: true); +``` + +### Drop a table + +```c# +await admin.DropTableAsync("ns", "table_name", ifExists: true); +``` + +### Checking if a table exists + +```c# +var tableExists = await admin.IsTablePresentAsync("ns", "table_name"); +``` + +### Get the names of existing tables + +```c# +var tablesList = await admin.GetTableNamesAsync("ns"); +``` + +### Create the Coordinator table + +```c# +await admin.CreateCoordinatorTablesAsync(); +``` + +### Drop the Coordinator table + +```c# +await admin.DropCoordinatorTablesAsync(); +``` + +### Check if the Coordinator table exist + +```c# +var exists = await admin.AreCoordinatorTablesPresentAsync(); +``` diff --git a/versioned_docs/version-3.X/scalardb-cluster-dotnet-client-sdk/getting-started-with-aspnet-and-di.mdx b/versioned_docs/version-3.X/scalardb-cluster-dotnet-client-sdk/getting-started-with-aspnet-and-di.mdx new file mode 100644 index 00000000..b3188720 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster-dotnet-client-sdk/getting-started-with-aspnet-and-di.mdx @@ -0,0 +1,84 @@ +--- +tags: + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Getting Started with ASP.NET Core and Dependency Injection in the ScalarDB Cluster .NET Client SDK + +The ScalarDB Cluster .NET Client SDK supports dependency injection (DI) in frameworks like ASP.NET Core. + +## Install the SDK + +Install the same major and minor version of the [SDK](https://www.nuget.org/packages/ScalarDB.Client) as ScalarDB Cluster into the .NET project. You can do this by using the built-in NuGet package manager, replacing `.` with the version that you're using: + +```console +dotnet add package ScalarDB.Client --version '..*' +``` + +## Add client settings + +Add the `ScalarDbOptions` section to the `appsettings.json` file of your ASP.NET Core app, replacing `` with the FQDN or the IP address, and `` with the port number (`60053` by default) of your cluster: + +```json +{ + "ScalarDbOptions": { + "Address": "http://:", + "HopLimit": 10 + } +} +``` + +For details about settings files and other ways to configure the client, see [Client configuration](common-reference.mdx#client-configuration). + +## Set up the transaction managers + +You can register the ScalarDB transaction managers in the DI container as follows: + +```c# +using ScalarDB.Client.Extensions; + +//... + +var builder = WebApplication.CreateBuilder(args); + +//... + +builder.Services.AddScalarDb(); +``` + +:::note + +The ScalarDB transaction managers will be registered as transient services. For details about service lifetimes, see [.NET dependency injection - Service lifetimes](https://learn.microsoft.com/en-us/dotnet/core/extensions/dependency-injection#service-lifetimes). + +::: + +After registering the transaction managers, they can be injected into the controller's constructor as follows: + +```c# +[ApiController] +public class OrderController: ControllerBase +{ + private readonly IDistributedTransactionManager _manager; + private readonly ISqlTransactionManager _sqlManager; + private readonly ITwoPhaseCommitTransactionManager _twoPhaseManager; + private readonly ISqlTwoPhaseCommitTransactionManager _sqlTwoPhaseManager; + private readonly IDistributedTransactionAdmin _admin; + + public OrderController(IDistributedTransactionManager manager, + ISqlTransactionManager sqlManager, + ITwoPhaseCommitTransactionManager twoPhaseManager, + ISqlTwoPhaseCommitTransactionManager sqlTwoPhaseManager, + IDistributedTransactionAdmin admin) + { + _manager = manager; + _sqlManager = sqlManager; + _twoPhaseManager = twoPhaseManager; + _sqlTwoPhaseManager = sqlTwoPhaseManager; + _admin = admin; + } +} +``` + +Although these examples are for WebApi projects, the examples will work in a similar way in GrpcService projects. diff --git a/versioned_docs/version-3.X/scalardb-cluster-dotnet-client-sdk/getting-started-with-auth.mdx b/versioned_docs/version-3.X/scalardb-cluster-dotnet-client-sdk/getting-started-with-auth.mdx new file mode 100644 index 00000000..196a2ac1 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster-dotnet-client-sdk/getting-started-with-auth.mdx @@ -0,0 +1,67 @@ +--- +tags: + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Getting Started with Authentication and Authorization by Using ScalarDB Cluster .NET Client SDK + +The ScalarDB Cluster .NET Client SDK supports [authentication and authorization](../scalardb-cluster/scalardb-auth-with-sql.mdx), which allows you to authenticate and authorize your requests to ScalarDB Cluster. + +## Install the SDK + +Install the same major and minor version of the [SDK](https://www.nuget.org/packages/ScalarDB.Client) as ScalarDB Cluster into the .NET project. You can do this by using the built-in NuGet package manager, replacing `.` with the version that you're using: + +```console +dotnet add package ScalarDB.Client --version '..*' +``` + +## Set credentials in the settings file + +You need to set credentials in the settings file as follows, replacing the contents in the angle brackets as described: + +```json +{ + "ScalarDbOptions": { + "Address": "http://:", + "HopLimit": 10, + "AuthEnabled": true, + "Username": "", + "Password": "" + } +} +``` + +For details about settings files and other ways to configure the client, see [Client configuration](common-reference.mdx#client-configuration). + +## Get a transaction manager + +You need to get a transaction manager or transaction admin object by using `TransactionFactory` as follows. Be sure to replace `` with `GetTransactionManager()`, `GetTwoPhaseCommitTransactionManager()`, `GetSqlTransactionManager()`, or `GetSqlTwoPhaseCommitTransactionManager()`. + +```c# +// Pass the path to the settings file. +var factory = TransactionFactory.Create("scalardb-options.json"); + +// To get a transaction manager +using var manager = factory.(); + +// To get a transaction admin +using var admin = factory.GetTransactionAdmin(); +``` + +A transaction manager or transaction admin object created from `TransactionFactory` with the provided credentials will automatically log in to ScalarDB Cluster and can communicate with it. + +## Wire encryption + +[Wire encryption](../scalardb-cluster/scalardb-auth-with-sql.mdx#wire-encryption) is also supported. It can be turned on by setting `Address` to the URL starting with `https` as follows: + +```json +{ + "ScalarDbOptions": { + "Address": "https://:" + } +} +``` + +For details about settings files and other ways to configure the client, see [Client configuration](common-reference.mdx#client-configuration). diff --git a/versioned_docs/version-3.X/scalardb-cluster-dotnet-client-sdk/getting-started-with-distributed-sql-transactions.mdx b/versioned_docs/version-3.X/scalardb-cluster-dotnet-client-sdk/getting-started-with-distributed-sql-transactions.mdx new file mode 100644 index 00000000..628ea26d --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster-dotnet-client-sdk/getting-started-with-distributed-sql-transactions.mdx @@ -0,0 +1,192 @@ +--- +tags: + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Getting Started with Distributed SQL Transactions in the ScalarDB Cluster .NET Client SDK + +The ScalarDB Cluster .NET Client SDK supports the distributed SQL transaction functionality of ScalarDB Cluster. The SDK includes transaction and manager abstractions for easier communication within a cluster. + +:::note + +Although we recommend using asynchronous methods, as in the following examples, you can use synchronous methods instead. + +::: + +For details about distributed non-SQL transactions, see [Getting Started with Distributed Transactions in the ScalarDB Cluster .NET Client SDK](getting-started-with-distributed-transactions.mdx). + +## Install the SDK + +Install the same major and minor version of the [SDK](https://www.nuget.org/packages/ScalarDB.Client) as ScalarDB Cluster into the .NET project. You can do this by using the built-in NuGet package manager, replacing `.` with the version that you're using: + +```console +dotnet add package ScalarDB.Client --version '..*' +``` + +## Create a settings file + +Create a `scalardb-options.json` file and add the following, replacing `` with the FQDN or the IP address, and `` with the port number (`60053` by default) of your cluster: + +```json +{ + "ScalarDbOptions": { + "Address": "http://:", + "HopLimit": 10 + } +} +``` + +For details about settings files and other ways to configure the client, see [Client configuration](common-reference.mdx#client-configuration). + +## Get a transaction manager + +You need to get a transaction manager object for distributed SQL transactions. To get the transaction manager object, you can use `TransactionFactory` as follows: + +```c# +// Pass the path to the settings file created in the previous step. +var factory = TransactionFactory.Create("scalardb-options.json"); + +using var manager = factory.GetSqlTransactionManager(); +``` + +## Execute SQL queries + +To execute a SQL statement, you need an `ISqlStatement` object, which can be created by using a builder as follows: + +```c# +using ScalarDB.Client.Builders.Sql; + +// ... + +var sqlStatement = + new SqlStatementBuilder() + .SetSql("SELECT * FROM order_service.statements WHERE item_id = :item_id") + .AddParam("item_id", 2) + .Build(); +``` + +A single SQL statement can be executed directly by using the transaction manager as follows: + +```c# +var resultSet = await manager.ExecuteAsync(sqlStatement); +``` + +The result from the `ExecuteAsync` method will contain records received from the cluster. The value of the specific column can be retrieved in the following manner: + +```c# +foreach (var record in resultSet.Records) +{ + // Getting an integer value from the "item_id" column. + // If it fails, an exception will be thrown. + var itemId = record.GetValue("item_id"); + + // Trying to get a string value from the "order_id" column. + // If it fails, no exception will be thrown. + if (record.TryGetValue("order_id", out var orderId)) + Console.WriteLine($"order_id: {orderId}"); + + // Checking if the "count" column is null. + if (record.IsNull("count")) + Console.WriteLine("'count' is null"); +} +``` + +For details about which type should be used in `GetValue` and `TryGetValue`, see [How ScalarDB Column Types Are Converted to and from .NET Types](common-reference.mdx#how-scalardb-column-types-are-converted-to-and-from-net-types). + +## Execute SQL queries in a transaction + +To execute multiple SQL statements as part of a single transaction, you need a transaction object. + +You can create a transaction object by using the transaction manager as follows: + +```c# +var transaction = await manager.BeginAsync(); +``` + +You can also resume a transaction that has already been started as follows: + +```c# +var transaction = manager.Resume(transactionIdString); +``` + +:::note + +The `Resume` method doesn't have an asynchronous version because it only creates a transaction object. Because of this, resuming a transaction by using the wrong ID is possible. + +::: + +The transaction has the same `ExecuteAsync` method as the transaction manager. That method can be used to execute SQL statements. + +When a transaction is ready to be committed, you can call the `CommitAsync` method of the transaction as follows: + +```c# +await transaction.CommitAsync(); +``` + +To roll back the transaction, you can use the `RollbackAsync` method: + +```c# +await transaction.RollbackAsync(); +``` + +## Get Metadata + +You can retrieve ScalarDB's metadata with the Metadata property as follows: + +```c# +// namespaces, tables metadata +var namespaceNames = new List(); + +await foreach (var ns in manager.Metadata.GetNamespacesAsync()) +{ + namespaceNames.Add(ns.Name); + Console.WriteLine($"Namespace: {ns.Name}"); + + await foreach (var tbl in ns.GetTablesAsync()) + { + Console.WriteLine($" Table: {tbl.Name}"); + + Console.WriteLine($" Columns:"); + foreach (var col in tbl.Columns) + Console.WriteLine($" {col.Name} [{col.DataType}]"); + + Console.WriteLine($" PartitionKey:"); + foreach (var col in tbl.PartitionKey) + Console.WriteLine($" {col.Name}"); + + Console.WriteLine($" ClusteringKey:"); + foreach (var col in tbl.ClusteringKey) + Console.WriteLine($" {col.Name} [{col.ClusteringOrder}]"); + + Console.WriteLine($" Indexes:"); + foreach (var index in tbl.Indexes) + Console.WriteLine($" {index.ColumnName}"); + + Console.WriteLine(); + } +} + +// users metadata +await foreach (var user in manager.Metadata.GetUsersAsync()) +{ + Console.WriteLine($"User: {user.Name} [IsSuperuser: {user.IsSuperuser}]"); + + foreach (var nsName in namespaceNames) + { + Console.WriteLine($" Namespace: {nsName}"); + + Console.WriteLine($" Privileges:"); + foreach (var privilege in await user.GetPrivilegesAsync(nsName)) + Console.WriteLine($" {privilege}"); + } + + Console.WriteLine(); +} +``` + +:::note + +To use LINQ methods with `IAsyncEnumerable`, you can install [System.Linq.Async](https://www.nuget.org/packages/System.Linq.Async/) package. + +::: diff --git a/versioned_docs/version-3.X/scalardb-cluster-dotnet-client-sdk/getting-started-with-distributed-transactions.mdx b/versioned_docs/version-3.X/scalardb-cluster-dotnet-client-sdk/getting-started-with-distributed-transactions.mdx new file mode 100644 index 00000000..30582abc --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster-dotnet-client-sdk/getting-started-with-distributed-transactions.mdx @@ -0,0 +1,329 @@ +--- +tags: + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Getting Started with Distributed Transactions in the ScalarDB Cluster .NET Client SDK + +The ScalarDB Cluster .NET Client SDK supports the distributed transaction functionality of ScalarDB Cluster. The SDK includes transaction and manager abstractions for easier communication within a cluster. + +:::note + +Although we recommend using asynchronous methods as in the following examples, you can use synchronous versions instead. + +::: + +For details about distributed SQL transactions, see [Getting Started with Distributed SQL Transactions in the ScalarDB Cluster .NET Client SDK](getting-started-with-distributed-sql-transactions.mdx). + +## Install the SDK + +Install the same major and minor version of the [SDK](https://www.nuget.org/packages/ScalarDB.Client) as ScalarDB Cluster into the .NET project. You can do this by using the built-in NuGet package manager, replacing `.` with the version that you're using: + +```console +dotnet add package ScalarDB.Client --version '..*' +``` + +## Create a settings file + +Create a `scalardb-options.json` file and add the following, replacing `` with the FQDN or the IP address, and `` with the port number (`60053` by default) of your cluster: + +```json +{ + "ScalarDbOptions": { + "Address": "http://:", + "HopLimit": 10 + } +} +``` + +For details about settings files and other ways to configure the client, see [Client configuration](common-reference.mdx#client-configuration). + +## Get a transaction manager + +You need to get a transaction manager for distributed transactions. To get the transaction manager, you can use `TransactionFactory` as follows: + +```c# +// Pass the path to the settings file created in the previous step. +var factory = TransactionFactory.Create("scalardb-options.json"); + +using var manager = factory.GetTransactionManager(); +``` + +## Manage transactions + +To execute multiple CRUD operations as part of a single transaction, first, you need to begin a transaction. You can begin a transaction by using the transaction manager as follows: + +```c# +var transaction = await manager.BeginAsync(); +``` + +You can also resume a transaction that is already being executed as follows: + +```c# +var transaction = manager.Resume(transactionIdString); +``` + +:::note + +The `Resume` method doesn't have an asynchronous version because it only creates a transaction object. Because of this, resuming a transaction by using the wrong ID is possible. + +::: + +When a transaction is ready to be committed, you can call the `CommitAsync` method of the transaction as follows: + +```c# +await transaction.CommitAsync(); +``` + +To roll back the transaction, you can use the `RollbackAsync` method: + +```c# +await transaction.RollbackAsync(); +``` + +## Execute CRUD operations + +A transaction has `GetAsync`, `ScanAsync`, `InsertAsync`, `UpsertAsync`, `UpdateAsync`, `DeleteAsync`, and `MutateAsync` methods to execute CRUD operations against the cluster. As a parameter, these methods have an operation object. An operation object can be created by using the builders listed in this section. + +:::note + +CRUD operations can be executed in a one-shot transaction manner without needing to explicitly create a transaction. For that, a manager object has the same CRUD methods as a transaction object. + +::: + +To use builders, add the following namespace to the `using` section: + +```c# +using ScalarDB.Client.Builders; +``` + +:::note + +The cluster does not support parallel execution of commands inside one transaction, so make sure to use `await` for asynchronous methods. + +::: + +### `GetAsync` method example + +To retrieve a single record, you can use the `GetAsync` method as follows: + +```c# +var get = + new GetBuilder() + .SetNamespaceName("ns") + .SetTableName("statements") + .AddPartitionKey("order_id", "1") + .AddClusteringKey("item_id", 2) + .SetProjections("item_id", "count") + .Build(); + +var getResult = await transaction.GetAsync(get); +``` + +It is possible to retrieve a record by using an index instead of a partition key. To do that, you need to set the type of operation to `GetWithIndex` as follows: + +```c# +// ... +using ScalarDB.Client.Core; + +// ... + +var get = + new GetBuilder() + // ... + .SetGetType(GetOperationType.GetWithIndex) + .AddPartitionKey("index_column", "1") + .Build(); +``` + +You can also specify arbitrary conditions that a retrieved record must meet, or it won't be returned. The conditions can be set as conjunctions of conditions as follows: + +```c# +var get = + new GetBuilder() + // ... + .AddConjunction(c => c.AddCondition("cost", 1000, Operator.LessThan)) + .AddConjunction(c => + { + c.AddCondition("cost", 10000, Operator.LessThan); + c.AddCondition("in_stock", true, Operator.Equal); + }) + .Build(); +``` + +In the above example, a record will be returned only if its `cost` is less than `1000`, or if its `cost` is less than `10000` and `in_stock` is true. + +#### Handle `IResult` objects + +The `GetAsync` and `ScanAsync` methods return `IResult` objects. An `IResult` object contains columns of the retrieved record. The value of the specific column can be retrieved in the following manner: + +```c# +// Getting an integer value from the "item_id" column. +// If it fails, an exception will be thrown. +var itemId = result.GetValue("item_id"); + +// Trying to get a string value from the "order_id" column. +// If it fails, no exception will be thrown. +if (result.TryGetValue("order_id", out var orderId)) + Console.WriteLine($"order_id: {orderId}"); + +// Checking if the "count" column is null. +if (result.IsNull("count")) + Console.WriteLine("'count' is null"); +``` + +For details about which type should be used in `GetValue` and `TryGetValue`, see [How ScalarDB Column Types Are Converted to and from .NET Types](common-reference.mdx#how-scalardb-column-types-are-converted-to-and-from-net-types). + +### `ScanAsync` method example + +To retrieve a range of records, you can use the `ScanAsync` method as follows: + +```c# +var scan = + new ScanBuilder() + .SetNamespaceName("ns") + .SetTableName("statements") + .AddPartitionKey("order_id", "1") + .AddStartClusteringKey("item_id", 2) + .SetStartInclusive(true) + .AddEndClusteringKey("item_id", 8) + .SetEndInclusive(true) + .SetProjections("item_id", "count") + .Build(); + +var scanResult = await transaction.ScanAsync(scan); +``` + +It is possible to retrieve a record by using an index instead of a partition key. To do that, you need to set the type of operation to `ScanWithIndex` as follows: + +```c# +// ... +using ScalarDB.Client.Core; + +// ... + +var scan = + new ScanBuilder() + // ... + .SetScanType(ScanOperationType.ScanWithIndex) + .AddPartitionKey("index_column", "1") + .Build(); +``` + +The arbitrary conditions that a retrieved record must meet can also be set for a scan operation in the same way as for a [get operation](getting-started-with-distributed-transactions.mdx#getasync-method-example). + +### `InsertAsync` method example + +To insert a new record, you can use the `InsertAsync` method as follows: + +```c# +var insert = + new InsertBuilder() + .SetNamespaceName("ns") + .SetTableName("statements") + .AddPartitionKey("order_id", "1") + .AddClusteringKey("item_id", 2) + .AddColumn("count", 11) + .Build(); + +await transaction.InsertAsync(insert); +``` + +### `UpsertAsync` method example + +To upsert a record (update an existing record or insert a new one), you can use the `UpsertAsync` method as follows: + +```c# +var upsert = + new UpsertBuilder() + .SetNamespaceName("ns") + .SetTableName("statements") + .AddPartitionKey("order_id", "1") + .AddClusteringKey("item_id", 2) + .AddColumn("count", 11) + .Build(); + +await transaction.UpsertAsync(upsert); +``` + +### `UpdateAsync` method example + +To update an existing record, you can use the `UpdateAsync` method as follows: + +```c# +// ... +using ScalarDB.Client.Core; + +// ... + +var update = + new UpdateBuilder() + .SetNamespaceName("ns") + .SetTableName("statements") + .AddPartitionKey("order_id", "1") + .AddClusteringKey("item_id", 2) + .AddColumn("count", 11) + .AddCondition("processed", false, Operator.Equal) + .Build(); + +await transaction.UpdateAsync(update); +``` + +### `DeleteAsync` method example + +To delete a record, you can use the `DeleteAsync` method as follows: + +```c# +// ... +using ScalarDB.Client.Core; + +// ... + +var delete = + new DeleteBuilder() + .SetNamespaceName("ns") + .SetTableName("statements") + .AddPartitionKey("order_id", "1") + .AddClusteringKey("item_id", 2) + .AddCondition("processed", false, Operator.Equal) + .Build(); + +await transaction.DeleteAsync(delete); +``` + +### `MutateAsync` method example + +The `MutateAsync` method allows you to execute more than one mutation operation in a single call to the cluster. You can do this in the following manner: + +```c# +// ... +using ScalarDB.Client.Core; + +// ... + +var mutations = new IMutation[] + { + new InsertBuilder() + // ... + .Build(), + new UpsertBuilder() + // ... + .Build(), + new UpdateBuilder() + // ... + .Build(), + new DeleteBuilder() + // ... + .Build() + }; + +await transaction.MutateAsync(mutations); +``` + +:::note + +To modify data by using the `InsertAsync`, `UpsertAsync`, `UpdateAsync`, `DeleteAsync`, or `MutateAsync` method, the data must be retrieved first by using the `GetAsync` or `ScanAsync` method. + +::: diff --git a/versioned_docs/version-3.X/scalardb-cluster-dotnet-client-sdk/getting-started-with-linq.mdx b/versioned_docs/version-3.X/scalardb-cluster-dotnet-client-sdk/getting-started-with-linq.mdx new file mode 100644 index 00000000..5acee088 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster-dotnet-client-sdk/getting-started-with-linq.mdx @@ -0,0 +1,369 @@ +--- +tags: + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Getting Started with LINQ in the ScalarDB Cluster .NET Client SDK + +The ScalarDB Cluster .NET Client SDK supports querying the cluster with LINQ and some Entity Framework-like functionality. + +:::note + +This SDK doesn't support [Entity Framework](https://learn.microsoft.com/en-us/ef/). Instead, this SDK implements functionality that is similar to Entity Framework. + +::: + +:::note + +SQL support must be enabled on the cluster to use LINQ. + +::: + +## Install the SDK + +Install the same major and minor version of the [SDK](https://www.nuget.org/packages/ScalarDB.Client) as ScalarDB Cluster into the .NET project. You can do this by using the built-in NuGet package manager, replacing `.` with the version that you're using: + +```console +dotnet add package ScalarDB.Client --version '..*' +``` + +## Add client settings + +Add the `ScalarDbOptions` section to the `appsettings.json` file of your ASP.NET Core app, replacing `` with the FQDN or the IP address, and `` with the port number (`60053` by default) of your cluster: + +```json +{ + "ScalarDbOptions": { + "Address": "http://:", + "HopLimit": 10 + } +} +``` + +For details about settings files and other ways to configure the client, see [Client configuration](common-reference.mdx#client-configuration). + +## Set up classes + +After confirming that SQL support is enabled, create a C# class for each ScalarDB table that you want to use. For example: + +```c# +using System.ComponentModel.DataAnnotations.Schema; +using ScalarDB.Client.DataAnnotations; + +// ... + +[Table("ns.statements")] +public class Statement +{ + [PartitionKey] + [Column("statement_id", Order = 0)] + public int Id { get; set; } + + [SecondaryIndex] + [Column("order_id", Order = 1)] + public string OrderId { get; set; } = String.Empty; + + [SecondaryIndex] + [Column("item_id", Order = 2)] + public int ItemId { get; set; } + + [Column("count", Order = 3)] + public int Count { get; set; } +} + +[Table("order_service.items")] +public class Item +{ + [PartitionKey] + [Column("item_id", Order = 0)] + public int Id { get; set; } + + [Column("name", Order = 1)] + public string Name { get; set; } = String.Empty; + + [Column("price", Order = 2)] + public int Price { get; set; } +} +``` + +If a partition key, clustering key, or secondary index consists of more than one column, the `Order` property of `ColumnAttribute` will decide the order inside the key or index. + +For details about which types should be used for properties, see [How ScalarDB Column Types Are Converted to and from .NET Types](common-reference.mdx#how-scalardb-column-types-are-converted-to-and-from-net-types). + +Create a context class that has properties for all the tables you want to use. For example: + +```c# + public class MyDbContext: ScalarDbContext + { + public ScalarDbSet Statements { get; set; } + public ScalarDbSet Items { get; set; } + } +``` + +After all the classes are created, you need to register the created context in the dependency injection container. For example: + +```c# +using ScalarDB.Client.Extensions; + +//... + +var builder = WebApplication.CreateBuilder(args); + +//... + +builder.Services.AddScalarDbContext(); +``` + +:::note + +The context class will be registered as a transient service. For details about service lifetimes, see [.NET dependency injection - Service lifetimes](https://learn.microsoft.com/en-us/dotnet/core/extensions/dependency-injection#service-lifetimes). + +::: + +The context can be injected into the controller's constructor as follows: + +```c# +[ApiController] +public class OrderController: ControllerBase +{ + private readonly MyDbContext _myDbContext; + + public OrderController(MyDbContext myDbContext) + { + _myDbContext = myDbContext; + } +} +``` + +## Use LINQ to query properties + +After receiving `MyDbContext` in your controller, you can query its properties by using LINQ. For example: + +### Use query syntax + +```c# +from stat in _myDbContext.Statements +join item in _myDbContext.Items on stat.ItemId equals item.Id +where stat.Count > 2 && item.Name.Contains("apple") +orderby stat.Count descending, stat.ItemId +select new { item.Name, stat.Count }; +``` + +### Use method syntax + +```c# +_myDbContext.Statements + .Where(stat => stat.OrderId == "1") + .Skip(1) + .Take(2); +``` + +### Use the `First` method to get one `Statement` by its partition key + +```c# +_myDbContext.Statements.First(stat => stat.OrderId == "1"); +``` + +### Use the `DefaultIfEmpty` method to perform left outer join + +```c# +from stat in _myDbContext.Statements +join item in _myDbContext.Items on stat.ItemId equals item.Id into items +from i in items.DefaultIfEmpty() +select new { ItemName = i != null ? i.Name : "" } +``` + +The following methods are supported: + +- `Select` +- `Where` +- `Join` +- `GroupJoin` +- `First`/`FirstOrDefault` +- `Skip` +- `Take` +- `OrderBy`/`OrderByDescending` +- `ThenBy`/`ThenByDescending` + +The following `String` methods are supported inside the predicates of `Where` and `First`/`FirstOrDefault` methods: + +- `Contains` +- `StartsWith` +- `EndsWith` + +Unsupported LINQ methods can be used after the supported methods. For example: + +```c# +_myDbContext.Statements + .Where(stat => stat.OrderId == "1") // Will be executed remotely on the cluster. + .Distinct() // Will be executed locally in the app. + .Where(stat => stat.ItemId < 5); // Will be executed locally. +``` + +:::note + +If `Skip` is specified before `Take` or `First`/`FirstOrDefault`, the number that is passed to `Skip` will be added to the `LIMIT` number in the SQL query. By itself, `Skip` won't change the resulting SQL query. + +::: + +## Limitations when using LINQ against `ScalarDbSet{T}` objects + +- All method calls are supported inside `Select`. For example: + +```c# +.Select(stat => convertToSomething(stat.ItemId)) +//... +.Select(stat => stat.ItemId * getSomeNumber()) +``` + +- Method calls, except for calls against the querying object, are also supported inside `Where` and `First`/`FirstOrDefault`. For example: + +```c# +.Where(stat => stat.ItemId == getItemId()) // is OK +//... +.Where(stat => stat.ItemId.ToString() == "1") // is not supported +``` + +- All method calls are supported inside the result-selecting lambda of `Join` and `GroupJoin`. For example: + +```c# +.Join(_myDbContext.Items, + stat => stat.ItemId, + item => item.Id, + (stat, item) => new { ItemName = convertToSomething(item.Name), + ItemCount = stat.Count.ToString() }) +``` + +- Method calls are not supported inside the key-selecting lambdas of `Join` and `GroupJoin`. +- Custom equality comparers are not supported. The `comparer` argument in `Join` and `GroupJoin` methods will be ignored if the argument has been passed. +- More than one `from` directly in one query is not supported, except when the `DefaultIfEmpty` method is used to perform left outer join. Each subsequent `from` is considered to be a separate query. + +```c# +var firstQuery = from stat in _myDbContext.Statements + where stat.Count > 2 + select new { stat.Count }; + +var secondQuery = from item in _myDbContext.Items + where item.Price > 6 + select new { item.Name }; + +var finalQuery = from first in firstQuery + from second in secondQuery + select new { first.Count, second.Name }; + +// 1. firstQuery will be executed against the cluster. +// 2. secondQuery will be executed against the cluster for each object (row) from 1. +// 3. finalQuery will be executed locally with the results from 1 and 2. +var result = finalQuery.ToArray(); +``` + +- Method calls are not supported inside `OrderBy`/`OrderByDescending` or `ThenBy`/`ThenByDescending`. +- Only overloads of `Contains`, `StartsWith`, and `EndsWith` methods that have a single string argument are supported inside `Where` and `First`/`FirstOrDefault`. + +## Modify data in a cluster by using `ScalarDbContext` + +The properties of the class inherited from `ScalarDbContext` can be used to modify data. + +### Add a new object by using the `AddAsync` method + +```c# +var statement = new Statement + { + OrderId = "2", + ItemId = 4, + Count = 8 + }; +await _myDbContext.Statements.AddAsync(statement); +``` + +### Update an object by using the `UpdateAsync` method + +```c# +var statement = _myDbContext.Statements.First(stat => stat.Id == 1); + +// ... + +statement.Count = 10; +await _myDbContext.Statements.UpdateAsync(statement); +``` + +### Remove an object by using the `RemoveAsync` method + +```c# +var statement = _myDbContext.Statements.First(stat => stat.Id == 1); + +// ... + +await _myDbContext.Statements.RemoveAsync(statement); +``` + +## Manage transactions + +LINQ queries and `AddAsync`, `UpdateAsync`, and `RemoveAsync` methods can be executed without an explicitly started transaction. However, to execute multiple queries and methods as part of a single transaction, the transaction must be explicitly started and committed. `ScalarDbContext` supports both ordinary transactions and transactions with the two-phase commit interface in ScalarDB. + +### Begin a new transaction + +```c# +await _myDbContext.BeginTransactionAsync(); +``` + +### Begin a new transaction with the two-phase commit interface + +```c# +await _myDbContext.BeginTwoPhaseCommitTransactionAsync(); +``` + +### Get the ID of a currently active transaction + +```c# +var transactionId = _myDbContext.CurrentTransactionId; +``` + +### Join an existing transaction with the two-phase commit interface + +```c# +await _myDbContext.JoinTwoPhaseCommitTransactionAsync(transactionId); +``` + +### Resume an existing transaction + +```c# +await _myDbContext.ResumeTransaction(transactionId); +``` + +### Resume an existing transaction with the two-phase commit interface + +```c# +await _myDbContext.ResumeTwoPhaseCommitTransaction(transactionId); +``` + +:::note + +The `ResumeTransaction`/`ResumeTwoPhaseCommitTransaction` methods don't have asynchronous versions because they only initialize the transaction data in the `ScalarDbContext` inheriting object without querying the cluster. Because of this, resuming a transaction by using the wrong ID is possible. + +::: + +### Commit a transaction (ordinary or two-phase commit) + +```c# +await _myDbContext.CommitTransactionAsync(); +``` + +### Roll back a transaction (ordinary or two-phase commit) + +```c# +await _myDbContext.RollbackTransactionAsync(); +``` + +### Prepare a transaction with the two-phase commit interface for the commit + +```c# +await _myDbContext.PrepareTransactionAsync(); +``` + +### Validate a transaction with the two-phase commit interface before the commit + +```c# +await _myDbContext.ValidateTransactionAsync(); +``` diff --git a/versioned_docs/version-3.X/scalardb-cluster-dotnet-client-sdk/getting-started-with-scalardb-tables-as-csharp-classes.mdx b/versioned_docs/version-3.X/scalardb-cluster-dotnet-client-sdk/getting-started-with-scalardb-tables-as-csharp-classes.mdx new file mode 100644 index 00000000..3b6357ab --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster-dotnet-client-sdk/getting-started-with-scalardb-tables-as-csharp-classes.mdx @@ -0,0 +1,204 @@ +--- +tags: + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Getting Started with Tables as C# Classes in the ScalarDB Cluster .NET Client SDK + +The ScalarDB Cluster .NET Client SDK helps you write code to access a cluster by abstracting ScalarDB tables as C# objects. After defining a class that represents a table in the cluster, you can ensure that a column name or its type won't be mixed up when querying the cluster. In addition, if a table's structure changes, you can apply the changes to the code by using the refactoring feature in your IDE. + +:::note + +Although we recommend using asynchronous methods, as in the following examples, you can use synchronous methods instead. + +::: + +## Install the SDK + +Install the same major and minor version of the [SDK](https://www.nuget.org/packages/ScalarDB.Client) as ScalarDB Cluster into the .NET project. You can do this by using the built-in NuGet package manager, replacing `.` with the version that you're using: + +```console +dotnet add package ScalarDB.Client --version '..*' +``` + +## Create classes for all ScalarDB tables + +To work with ScalarDB tables as C# objects, you must create a class for each table that you want to use. For example: + +```c# +using System.ComponentModel.DataAnnotations.Schema; +using ScalarDB.Client.DataAnnotations; + +// ... + +[Table("ns.statements")] +public class Statement +{ + [PartitionKey] + [Column("order_id", Order = 0)] + public string OrderId { get; set; } = String.Empty; + + [ClusteringKey] + [Column("item_id", Order = 1)] + public int ItemId { get; set; } + + [Column("count", Order = 2)] + public int Count { get; set; } +} +``` + +For details about which types should be used for properties, see [How ScalarDB Column Types Are Converted to and from .NET Types](common-reference.mdx#how-scalardb-column-types-are-converted-to-and-from-net-types). + +## Execute CRUD operations + +After creating a class for each table, you can use the classes as objects by using the generic `GetAsync`, `ScanAsync`, `InsertAsync`, `UpdateAsync`, `DeleteAsync`, `UpsertAsync`, or `MutateAsync` method of `ITransactionCrudOperable`. + +To use these generic methods, add the following namespace to the `using` section: + +```c# +using ScalarDB.Client.Extensions; +``` + +### Get one object by using the `GetAsync` method + +```c# +var keys = new Dictionary + { + { nameof(Statement.OrderId), "1" } + }; +var statement = await transaction.GetAsync(keys); + +Console.WriteLine($"ItemId: {statement.ItemId}, Count: {statement.Count}"); +``` + +### Get multiple objects by using the `ScanAsync` method + +```c# +var startKeys = new Dictionary + { + { nameof(Statement.OrderId), "1" }, + { nameof(Statement.ItemId), 3 } + }; +var endKeys = new Dictionary + { + { nameof(Statement.ItemId), 6} + }; + +await foreach (var s in transaction.ScanAsync(startKeys, endKeys)) + Console.WriteLine($"ItemId: {s.ItemId}, Count: {s.Count}"); +``` + +:::note + +To use LINQ methods with `IAsyncEnumerable`, you can install [System.Linq.Async](https://www.nuget.org/packages/System.Linq.Async/) package. + +::: + +### Insert a new object by using the `InsertAsync` method + +```c# +var statement = new Statement + { + OrderId = "2", + ItemId = 4, + Count = 8 + }; +await transaction.InsertAsync(statement); +``` + +### Update an object by using the `UpdateAsync` method + +```c# +// ... +statement.ItemId = 4; +statement.Count = 8; + +await transaction.UpdateAsync(statement); +``` + +### Delete an object by using the `DeleteAsync` method + +```c# +// ... +await transaction.DeleteAsync(statement); +``` + +### Upsert an object by using the `UpsertAsync` method + +```c# +var statement = new Statement + { + OrderId = "2", + ItemId = 4, + Count = 8 + }; +await transaction.UpsertAsync(statement); +``` + +### Upsert and delete multiple objects at once by using the `MutateAsync` method + +```c# +var statement = new Statement + { + OrderId = "2", + ItemId = 4, + Count = 16 + }; + +// ... + +await transaction.MutateAsync(objectsToUpsert: new[] { statement }, + objectsToDelete: new[] { statement2 }); +``` + +:::note + +To modify objects by using the `UpdateAsync`, `DeleteAsync`, `UpsertAsync`, or `MutateAsync` method, the objects must be retrieved first by using the `GetAsync` or `ScanAsync` method. + +::: + +## Use the Administrative API + +C# objects also can be used with the Administrative API. To use generic Administrative API methods, add the following namespace to the `using` section: + +```c# +using ScalarDB.Client.Extensions; +``` + +### Create a new namespace + +```c# +await admin.CreateNamespaceAsync(); +``` + +### Drop an existing namespace + +```c# +await admin.DropNamespaceAsync(); +``` + +### Check if a namespace exists + +```c# +var namespaceExists = await admin.IsNamespacePresentAsync(); +``` + +### Create a new table + +```c# +await admin.CreateTableAsync(); +``` + +### Drop an existing table + +```c# +await admin.DropTableAsync(); +``` + +### Check if a table exists + +```c# +var tableExists = await admin.IsTablePresentAsync(); +``` diff --git a/versioned_docs/version-3.X/scalardb-cluster-dotnet-client-sdk/getting-started-with-two-phase-commit-transactions.mdx b/versioned_docs/version-3.X/scalardb-cluster-dotnet-client-sdk/getting-started-with-two-phase-commit-transactions.mdx new file mode 100644 index 00000000..7a684b2c --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster-dotnet-client-sdk/getting-started-with-two-phase-commit-transactions.mdx @@ -0,0 +1,142 @@ +--- +tags: + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Getting Started with Distributed Transactions with a Two-Phase Commit Interface in the ScalarDB Cluster .NET Client SDK + +The ScalarDB Cluster .NET Client SDK supports transactions with the two-phase commit interface in ScalarDB. The SDK includes transaction and manager abstractions for enhanced communication within a cluster. + +:::note + +Although we recommend using asynchronous methods as in the following examples, you can use synchronous methods instead. + +::: + +## About transactions with the two-phase commit interface + +By using the SDK, you can execute transactions with the two-phase commit interface that span multiple applications. For example, if you have multiple microservices, you can create a transaction manager in each of them and execute a transaction that spans those microservices. + +In transactions with the two-phase commit interface, there are two roles—coordinator and a participant—that collaboratively execute a single transaction. + +The coordinator process first begins a transaction and sends the ID of the transaction to all the participants, and the participant processes join the transaction. After executing CRUD or SQL operations, the coordinator process and the participant processes commit the transaction by using the two-phase interface. + +## Install the SDK + +Install the same major and minor version of the [SDK](https://www.nuget.org/packages/ScalarDB.Client) as ScalarDB Cluster into the .NET project. You can do this by using the built-in NuGet package manager, replacing `.` with the version that you're using: + +```console +dotnet add package ScalarDB.Client --version '..*' +``` + +## Create a settings file + +Create a `scalardb-options.json` file and add the following, replacing `` with the FQDN or the IP address, and `` with the port number (`60053` by default) of your cluster: + +```json +{ + "ScalarDbOptions": { + "Address": "http://:", + "HopLimit": 10 + } +} +``` + +For details about settings files and other ways to configure the client, see [Client configuration](common-reference.mdx#client-configuration). + +## Get a transaction manager (for coordinator and participants) + +You need to get a transaction manager for distributed transactions with the two-phase commit interface. To get the transaction manager, you can use `TransactionFactory` as follows: + +```c# +// Pass the path to the settings file created in the previous step. +var factory = TransactionFactory.Create("scalardb-options.json"); + +using var manager = factory.GetTwoPhaseCommitTransactionManager(); +``` + +Alternatively, you can use SQL instead of CRUD operations for transactions with the two-phase commit interface by specifying the following transaction manager: + +```c# +using var manager = factory.GetSqlTwoPhaseCommitTransactionManager(); +``` + +## Begin a transaction (for coordinator) + +You can begin a transaction with the two-phase commit interface in the coordinator as follows: + +```c# +var transaction = await manager.BeginAsync(); +``` + +The ID of the started transaction can be obtained with the following code: + +```c# +var transactionId = transaction.Id; +``` + +## Join a transaction (for participants) + +You can join a transaction with the two-phase commit interface in a participant as follows: + +```c# +var transaction = await manager.JoinAsync(transactionId); +``` + +## Resume a transaction (for coordinator and participants) + +Usually, a transaction with the two-phase commit interface involves multiple request and response exchanges. In scenarios where you need to work with a transaction that has been begun or joined in the previous request, you can resume such transaction as follows: + +```c# +var transaction = manager.Resume(transactionId); +``` + +:::note + +The `Resume` method doesn't have an asynchronous version because it only creates a transaction object. Because of this, resuming a transaction by using the wrong ID is possible. + +::: + +## Roll back a transaction + +If a transaction fails to commit, you can roll back the transaction as follows: + +```c# +await transaction.RollbackAsync(); +``` + +## Commit a transaction (for coordinator and participants) + +After completing CRUD or SQL operations, you must commit the transaction. However, for transactions with the two-phase commit interface, you must prepare the transaction in the coordinator and all the participants first. + +```c# +await transaction.PrepareAsync(); +``` + +Next, depending on the concurrency control protocol, you may need to validate the transaction in the coordinator and all the participants as follows: + +```c# +await transaction.ValidateAsync(); +``` + +Finally, you can commit the transaction in the coordinator and all the participants as follows: + +```c# +await transaction.CommitAsync(); +``` + +If the coordinator or any of the participants failed to prepare or validate the transaction, you will need to call `RollbackAsync` in the coordinator and all the participants. + +In addition, if the coordinator and all the participants failed to commit the transaction, you will need to call `RollbackAsync` in the coordinator and all the participants. + +However, if the coordinator or only some of the participants failed to commit the transaction, the transaction will be regarded as committed as long as the coordinator or any one of the participants has succeeded in committing the transaction. + +## Execute CRUD operations + +The two-phase commit interface of the transaction has the same methods for CRUD operations as ordinary transactions. For details, see [Execute CRUD operations](getting-started-with-distributed-transactions.mdx#execute-crud-operations). + +## Execute SQL statements + +The two-phase commit interface of the SQL transaction has the same methods for executing SQL queries as ordinary SQL transactions. For details, see [Execute SQL queries](getting-started-with-distributed-sql-transactions.mdx#execute-sql-queries). diff --git a/versioned_docs/version-3.X/scalardb-cluster-dotnet-client-sdk/index.mdx b/versioned_docs/version-3.X/scalardb-cluster-dotnet-client-sdk/index.mdx new file mode 100644 index 00000000..f2562631 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster-dotnet-client-sdk/index.mdx @@ -0,0 +1,22 @@ +--- +tags: + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# ScalarDB Cluster .NET Client SDK Overview + +The ScalarDB Cluster .NET Client SDK enables applications to connect to ScalarDB Cluster by using gRPC. + +To use the ScalarDB Cluster .NET Client SDK, see the following getting started guides: + +* [Getting Started with Distributed Transactions](getting-started-with-distributed-transactions.mdx) +* [Getting Started with Distributed SQL Transactions](getting-started-with-distributed-sql-transactions.mdx) +* [Getting Started with the Administrative API](getting-started-with-admin-api.mdx) +* [Getting Started with ScalarDB Tables as C# Classes](getting-started-with-scalardb-tables-as-csharp-classes.mdx) +* [Getting Started with ASP.NET Core and Dependency Injection](getting-started-with-aspnet-and-di.mdx) +* [Getting Started with LINQ](getting-started-with-linq.mdx) +* [Getting Started with Distributed Transactions with a Two-Phase Commit Interface](getting-started-with-two-phase-commit-transactions.mdx) +* [Getting Started with Authentication and Authorization](getting-started-with-auth.mdx) +* [Exception Handling](exception-handling.mdx) diff --git a/versioned_docs/version-3.X/scalardb-cluster/authorize-with-abac.mdx b/versioned_docs/version-3.X/scalardb-cluster/authorize-with-abac.mdx new file mode 100644 index 00000000..7bffd58c --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster/authorize-with-abac.mdx @@ -0,0 +1,27 @@ +--- +tags: + - Enterprise Premium Option + - Private Preview +displayed_sidebar: docsEnglish +--- + +# Control User Access in a Fine-Grained Manner + +:::info + +- This feature is currently available only to customers in Japan. If you're a customer in Japan, please see the Japanese version of this page. +- If you need more details about this feature in English, please [contact support](https://www.scalar-labs.com/support). + +::: + +ScalarDB Cluster can authorize users in a fine-grained manner with a mechanism called attributed-based access control (ABAC). This page explains how to use ABAC in ScalarDB Cluster. + +## What is ABAC? + +ABAC is a fine-grained access control mechanism in ScalarDB Cluster, allowing for record-level access control instead of just table-level access control, done through [simple authorization](./scalardb-auth-with-sql.mdx). With ABAC, a user can access a particular record only if the user's attributes and the record's attributes match. For example, you can restrict access to some highly confidential records to only users with the required privileges. This mechanism is also useful when multiple applications share the same table but need to access different segments based on their respective privileges. + +## Why use ABAC? + +Enterprise databases often provide row-level security or similar alternatives to allow for controlling access to rows in a database table. However, if a system comprises several databases, you need to configure each database one by one in the same way. If different kinds of databases are used, you have to configure each database by understanding the differences in the capabilities of each database. Such configuration causes too much burden and is error-prone. With ABAC, you can just configure it once, even though you manage several databases under ScalarDB. + +Row-level security features in most databases often require you to implement matching logic through functions like stored procedures. This can sometimes lead to writing lots of code to achieve the desired logic, which can become burdensome. In contrast, ABAC allows you to configure matching logic by using attributes known as tags. With ABAC, you only need to define these tags and assign them to users and records, eliminating the need for coding. Tags consist of several components that enable you to specify matching logic in a flexible and straightforward manner. diff --git a/versioned_docs/version-3.X/scalardb-cluster/compatibility.mdx b/versioned_docs/version-3.X/scalardb-cluster/compatibility.mdx new file mode 100644 index 00000000..5aafdca9 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster/compatibility.mdx @@ -0,0 +1,49 @@ +--- +tags: + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# ScalarDB Cluster Compatibility Matrix + +This document shows the compatibility of ScalarDB Cluster versions among client SDK versions. + +## ScalarDB Cluster compatibility with client SDKs + +| ScalarDB Cluster version | ScalarDB Cluster Java Client SDK version | ScalarDB Cluster .NET Client SDK version | +|:-------------------------|:-----------------------------------------|:-----------------------------------------| +| 3.16 | 3.9 - 3.16 | 3.12* - 3.16 | +| 3.15 | 3.9 - 3.15 | 3.12* - 3.15 | +| 3.14 | 3.9 - 3.14 | 3.12* - 3.14 | +| 3.13 | 3.9 - 3.13 | 3.12* - 3.13 | +| 3.12 | 3.9 - 3.12 | 3.12* | +| 3.11 | 3.9 - 3.11 | Not supported | +| 3.10 | 3.9 - 3.10 | Not supported | +| 3.9 | 3.9 | Not supported | + +\* This version is in private preview, which means that future versions might have backward-incompatible updates. + +:::note + +- You can consider the client tools (for example, [ScalarDB Cluster SQL CLI](developer-guide-for-scalardb-cluster-with-java-api.mdx#sql-cli) and [ScalarDB Cluster Schema Loader](developer-guide-for-scalardb-cluster-with-java-api.mdx#schema-loader-for-cluster)) to be the same as the ScalarDB Cluster Java Client SDK. In other words, you can apply the same compatibility rules to client tools as the ScalarDB Cluster Java Client SDK. +- When you access backend databases by using ScalarDB Data Loader, you must use a version of ScalarDB Data Loader that is compatible with the version of ScalarDB Cluster that you're using. In this case, the supported version of ScalarDB Data Loader is the same as the version of the ScalarDB Cluster Java Client SDK shown in the matrix above. Note that ScalarDB Data Loader doesn't access ScalarDB Cluster directly. +- If you use a new feature that ScalarDB Cluster provides in a new minor version, you may need to use the same or a later version of the client tools or re-create (or update) existing schemas. For details, please refer to the relevant documentation about each feature. + +::: + +## Version skew policy + +:::note + +Versions are expressed as `x.y.z`, where `x` represents the major version, `y` represents the minor version, and `z` represents the patch version. This format follows [Semantic Versioning](https://semver.org/). + +::: + +- If the **major** versions are different between ScalarDB Cluster and a client SDK, they are **not** compatible and are **not** supported. +- If the **major** versions are the same and the **minor** versions are different between ScalarDB Cluster and a client SDK, the version of ScalarDB Cluster must be greater than or equal to the client SDK version. For example: + - **Supported:** Combination of ScalarDB Cluster 3.13 and client SDK 3.11 + - **Not supported:** Combination of ScalarDB Cluster 3.11 and client SDK 3.13 +- If the **major** versions and the **minor** versions are the same, you can use different **patch** versions between ScalarDB Cluster and a client SDK. For example: + - **Supported:** Combination of ScalarDB Cluster 3.13.2 and client SDK 3.13.0 + - **Supported:** Combination of ScalarDB Cluster 3.13.0 and client SDK 3.13.2 diff --git a/versioned_docs/version-3.X/scalardb-cluster/deployment-patterns-for-microservices.mdx b/versioned_docs/version-3.X/scalardb-cluster/deployment-patterns-for-microservices.mdx new file mode 100644 index 00000000..d86a2d55 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster/deployment-patterns-for-microservices.mdx @@ -0,0 +1,72 @@ +--- +tags: + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# ScalarDB Cluster Deployment Patterns for Microservices + +When building microservice applications that use ScalarDB Cluster, there are two patterns you can choose for how to deploy ScalarDB Cluster: shared-cluster pattern and separated-cluster pattern. +This document first explains those patterns, how they differ, and the basic guidelines on which one to choose in which cases. + +Also, this document assumes that your microservice applications are created based on the database-per-service pattern, where each microservice manages its database, and a microservice needs to access another microservice's database via APIs between the microservices. + +## ScalarDB Cluster deployment patterns + +In the shared-cluster pattern, microservices share one ScalarDB Cluster instance, which is a cluster of ScalarDB Cluster nodes, in a system, so they access the same ScalarDB Cluster instance to interact with their databases. On the other hand, in the separated-cluster pattern, microservices use several ScalarDB Cluster instances. Typically, one microservice accesses one ScalarDB Cluster instance to interact with its database. + +The following diagram shows the patterns. (MS stands for microservice.) + +![ScalarDB Cluster deployment patterns for microservices.](images/scalardb-deployment-patterns.png) + +:::note + +You also need to manage the Coordinator table in either pattern in addition to the databases required for microservices. + +::: + +## Pros and cons + +One obvious difference is the amount of resources for ScalarDB Cluster instances. With the separated-cluster pattern, you need more resources to manage your applications. This also incurs more maintenance burden and costs. + +In addition, the ScalarDB Cluster APIs that you would need to use are different. Specifically, for the shared-cluster pattern, you need to use the [one-phase commit interface](../api-guide.mdx#transactional-api), where only one microservice needs to call `commit` to commit a transaction after microservices read and write records. For the separated-cluster pattern, you need to use the [two-phase commit interface](../two-phase-commit-transactions.mdx), where all the microservices first need to call `prepare` and then call `commit` if all the prepare calls are successful. Therefore, microservices with the separated-cluster pattern will likely be more complex than microservices with the shared-cluster pattern because they need to handle transactions and their errors in a more fine-grained manner. + +Moreover, the level of resource isolation is different. Microservices should be well-isolated for better maintainability and development efficiency, but the shared-cluster pattern brings weaker resource isolation. Weak resource isolation might also bring weak security. However, security risks can be mitigated by using the security features of ScalarDB Cluster, like authentication and authorization. + +Similarly, there is a difference in how systems are administrated. Specifically, in the shared-cluster pattern, a team must be tasked with managing a ScalarDB Cluster instance on behalf of the other teams. Typically, the central data team can manage it, but issues may arise if no such team exists. With the separated-cluster pattern, administration is more balanced but has a similar issue for the Coordinator table. The issue can be addressed by having a microservice for coordination and making a team manage the microservice. + +The following is a summary of the pros and cons of the patterns. + +### Shared-cluster pattern + +- **Pros:** + - Simple transaction and error handling because of the one-phase commit interface. (Backup operations for databases can also be simple.) + - Less resource usage because it uses one ScalarDB Cluster instance. +- **Cons:** + - Weak resource isolation between microservices. + - Unbalanced administration. (One team needs to manage a ScalarDB Cluster instance on behalf of the others.) + +### Separated-cluster pattern + +- **Pros:** + - Better resource isolation. + - More balanced administration. (A team manages one microservice and one ScalarDB Cluster instance. Also, a team must be tasked with managing the Coordinator table.) +- **Cons:** + - Complex transaction and error handling due to the two-phase commit interface. (Backup operations for databases can also be complex.) + - More resource usage because of several ScalarDB Cluster instances. + +## Which pattern to choose + +Using the shared-cluster pattern is recommended whenever possible. Although the shared-cluster pattern has some disadvantages, as described above, its simplicity and ease of management outweigh those disadvantages. Moreover, since ScalarDB Cluster stores all critical states in their underlying databases and does not hold any critical states in its memory, it can be seen as just a path to the databases. Therefore, we believe a system with the shared-cluster pattern still complies with the database-per-service pattern and does not violate the microservice philosophy much. + +If the cons of the shared-cluster pattern are not acceptable, you can still use the separated-cluster pattern. However, you should use that pattern only if you properly understand the mechanism and usage of the two-phase commit interface. Otherwise, you might face some issues, like database anomalies. + +## Limitations + +ScalarDB provides several APIs, such as CRUD, SQL, and Spring Data JDBC. Although the CRUD and SQL interfaces support both the shared-cluster and separated-cluster patterns, the Spring Data JDBC interface does not support the shared-cluster pattern. This is because its one-phase commit interface currently assumes an application is monolithic, where it is not divided into microservices that interact with each other. The Spring Data JDBC interface supports the two-phase commit interface and the separated-cluster pattern, just as the other APIs do. + +## See also + +- [Transactions with a Two-Phase Commit Interface](../two-phase-commit-transactions.mdx) + diff --git a/versioned_docs/version-3.X/scalardb-cluster/developer-guide-for-scalardb-cluster-with-java-api.mdx b/versioned_docs/version-3.X/scalardb-cluster/developer-guide-for-scalardb-cluster-with-java-api.mdx new file mode 100644 index 00000000..b4b0830a --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster/developer-guide-for-scalardb-cluster-with-java-api.mdx @@ -0,0 +1,261 @@ +--- +tags: + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Developer Guide for ScalarDB Cluster with the Java API + +ScalarDB Cluster provides a Java API for developing applications. +This document explains how to use the Java API. + +## Add ScalarDB Cluster Java Client SDK to your build + +The ScalarDB Cluster Java Client SDK is available in the [Maven Central Repository](https://mvnrepository.com/artifact/com.scalar-labs/scalardb-cluster-java-client-sdk). + +To add a dependency on the ScalarDB Cluster Java Client SDK by using Gradle, use the following: + +```gradle +dependencies { + implementation 'com.scalar-labs:scalardb-cluster-java-client-sdk:3.16.0' +} +``` + +To add a dependency by using Maven, use the following: + +```xml + + com.scalar-labs + scalardb-cluster-java-client-sdk + 3.16.0 + +``` + +## Client modes + +The ScalarDB Cluster Java Client SDK supports two client modes: `indirect` and `direct-kubernetes`. The following describes the client modes. + +### `indirect` client mode + +This mode simply sends a request to any cluster node (typically via a load balancer, such as Envoy), and the cluster node receiving the request routes the request to the appropriate cluster node that has the transaction state. + +![ScalarDB Cluster architecture](images/indirect-client-mode.png) + +The advantage of this mode is that we can keep the client thin. +The disadvantage is that we need an additional hop to reach the correct cluster node, which may affect performance. + +You can use this connection mode even if your application is running on a different Kubernetes cluster and your application can't access the Kubernetes API and each cluster node. +If your application is running on the same Kubernetes cluster as your ScalarDB Cluster nodes, you can use the `direct-kubernetes` client mode. + +### `direct-kubernetes` client mode + +In this mode, the client uses the membership logic (using the Kubernetes API) and the distribution logic (consistent hashing algorithm) to find the right cluster node that has the transaction state. +The client then sends a request to the cluster node directly. + +![ScalarDB Cluster architecture](images/direct-kubernetes-client-mode.png) + +The advantage of this mode is that we can reduce the hop count to reach the proper cluster node, which will improve the performance. +The disadvantage of this mode is that we need to make the client fat because the client needs to have membership logic and request-routing logic. + +Since this connection mode needs to access the Kubernetes API and each cluster node, you can use this connection mode only if your application is running on the same Kubernetes cluster as your ScalarDB Cluster nodes. +If your application is running on a different Kubernetes cluster, use the `indirect` client mode. + +For details about how to deploy your application on Kubernetes with `direct-kubernetes` client mode, see [Deploy your client application on Kubernetes with `direct-kubernetes` mode](../helm-charts/how-to-deploy-scalardb-cluster.mdx#deploy-your-client-application-on-kubernetes-with-direct-kubernetes-mode). + +## ScalarDB Cluster Java API + +The ScalarDB Cluster Java Client SDK provides a Java API for applications to access ScalarDB Cluster. The following diagram shows the architecture of the ScalarDB Cluster Java API. + +``` + +------------------+ + | User/Application | + +------------------+ + ↓ Java API + +--------------+ + | ScalarDB API | + +--------------+ + ↓ gRPC + +------------------+ + | ScalarDB Cluster | + +------------------+ + ↓ DB vendor–specific protocol + +----+ + | DB | + +----+ +``` + +Using the ScalarDB Cluster Java API is almost the same as using the ScalarDB Java API except the client configurations and Schema Loader are different. +For details, see [ScalarDB Java API Guide](../api-guide.mdx). + +The following section describes the Schema Loader for ScalarDB Cluster. + +### Schema Loader for Cluster + +To load a schema via ScalarDB Cluster, you need to use the dedicated Schema Loader for ScalarDB Cluster (Schema Loader for Cluster). +Using the Schema Loader for Cluster is basically the same as using the [ScalarDB Schema Loader](../schema-loader.mdx) except the name of the JAR file is different. +You can download the Schema Loader for Cluster from [ScalarDB Releases](https://github.com/scalar-labs/scalardb/releases/tag/v3.16.0). +After downloading the JAR file, you can run Schema Loader for Cluster with the following command: + +```console +java -jar scalardb-cluster-schema-loader-3.16.0-all.jar --config --schema-file --coordinator +``` + +You can also pull the Docker image from the [Scalar container registry](https://github.com/orgs/scalar-labs/packages/container/package/scalardb-cluster-schema-loader) by running the following command, replacing the contents in the angle brackets as described: + +```console +docker run --rm -v :/scalardb.properties -v :/schema.json ghcr.io/scalar-labs/scalardb-cluster-schema-loader:3.16.0 --config /scalardb.properties --schema-file /schema.json --coordinator +``` + +## ScalarDB Cluster SQL + +ScalarDB Cluster SQL can be accessed via JDBC and Spring Data JDBC for ScalarDB in Java as follows: + +``` + +-----------------------------------------+ + | User/Application | + +-----------------------------------------+ + ↓ ↓ Java API +Java API ↓ +-------------------------------+ + (JDBC) ↓ | Spring Data JDBC for ScalarDB | + ↓ +-------------------------------+ ++----------------------------------------------+ +| ScalarDB JDBC (ScalarDB SQL) | ++----------------------------------------------+ + ↓ gRPC + +----------------------+ + | ScalarDB Cluster SQL | + +----------------------+ + ↓ DB vendor–specific protocol + +----+ + | DB | + +----+ +``` + +This section describes how to use ScalarDB Cluster SQL though JDBC and Spring Data JDBC for ScalarDB. + +### ScalarDB Cluster SQL via JDBC + +Using ScalarDB Cluster SQL via JDBC is almost the same using [ScalarDB JDBC](../scalardb-sql/jdbc-guide.mdx) except for how to add the JDBC driver to your project. + +In addition to adding the ScalarDB Cluster Java Client SDK as described in [Add ScalarDB Cluster Java Client SDK to your build](#add-scalardb-cluster-java-client-sdk-to-your-build), you need to add the following dependencies to your project: + +To add the dependencies on the ScalarDB Cluster JDBC driver by using Gradle, use the following: + +```gradle +dependencies { + implementation 'com.scalar-labs:scalardb-sql-jdbc:3.16.0' + implementation 'com.scalar-labs:scalardb-cluster-java-client-sdk:3.16.0' +} +``` + +To add the dependencies by using Maven, use the following: + +```xml + + + com.scalar-labs + scalardb-sql-jdbc + 3.16.0 + + + com.scalar-labs + scalardb-cluster-java-client-sdk + 3.16.0 + + +``` + +Other than that, using ScalarDB Cluster SQL via JDBC is the same as using ScalarDB JDBC. +For details about ScalarDB JDBC, see [ScalarDB JDBC Guide](../scalardb-sql/jdbc-guide.mdx). + +### ScalarDB Cluster SQL via Spring Data JDBC for ScalarDB + +Similar to ScalarDB Cluster SQL via JDBC, using ScalarDB Cluster SQL via Spring Data JDBC for ScalarDB is almost the same as using [Spring Data JDBC for ScalarDB](../scalardb-sql/spring-data-guide.mdx) except for how to add it to your project. + +In addition to adding the ScalarDB Cluster Java Client SDK as described in [Add ScalarDB Cluster Java Client SDK to your build](#add-scalardb-cluster-java-client-sdk-to-your-build), you need to add the following dependencies to your project: + +To add the dependencies by using Gradle, use the following: + +```gradle +dependencies { + implementation 'com.scalar-labs:scalardb-sql-spring-data:3.16.0' + implementation 'com.scalar-labs:scalardb-cluster-java-client-sdk:3.16.0' +} +``` + +To add the dependencies by using Maven, use the following: + +```xml + + + com.scalar-labs + scalardb-sql-spring-data + 3.16.0 + + + com.scalar-labs + scalardb-cluster-java-client-sdk + 3.16.0 + + +``` + +Other than that, using ScalarDB Cluster SQL via Spring Data JDBC for ScalarDB is the same as using Spring Data JDBC for ScalarDB. +For details about Spring Data JDBC for ScalarDB, see [Guide of Spring Data JDBC for ScalarDB](../scalardb-sql/spring-data-guide.mdx). + +### SQL CLI + +Like other SQL databases, ScalarDB SQL also provides a CLI tool where you can issue SQL statements interactively in a command-line shell. + +You can download the SQL CLI for Cluster from [ScalarDB Releases](https://github.com/scalar-labs/scalardb/releases/tag/v3.16.0). After downloading the JAR file, you can run the SQL CLI with the following command: + +```console +java -jar scalardb-cluster-sql-cli-3.16.0-all.jar --config +``` + +You can also pull the Docker image from the [Scalar container registry](https://github.com/orgs/scalar-labs/packages/container/package/scalardb-cluster-sql-cli) by running the following command, replacing the contents in the angle brackets as described: + +```console +docker run --rm -it -v :/scalardb-sql.properties ghcr.io/scalar-labs/scalardb-cluster-sql-cli:3.16.0 --config /scalardb-sql.properties +``` + +#### Usage + +You can see the CLI usage with the `-h` option as follows: + +```console +java -jar scalardb-cluster-sql-cli-3.16.0-all.jar -h +Usage: scalardb-sql-cli [-hs] -c=PROPERTIES_FILE [-e=COMMAND] [-f=FILE] + [-l=LOG_FILE] [-o=] [-p=PASSWORD] + [-u=USERNAME] +Starts ScalarDB SQL CLI. + -c, --config=PROPERTIES_FILE + A configuration file in properties format. + -e, --execute=COMMAND A command to execute. + -f, --file=FILE A script file to execute. + -h, --help Display this help message. + -l, --log=LOG_FILE A file to write output. + -o, --output-format= + Format mode for result display. You can specify + table/vertical/csv/tsv/xmlattrs/xmlelements/json/a + nsiconsole. + -p, --password=PASSWORD A password to connect. + -s, --silent Reduce the amount of informational messages + displayed. + -u, --username=USERNAME A username to connect. +``` + +## Further reading + +If you want to use ScalarDB Cluster in programming languages other than Java, you can use the ScalarDB Cluster gRPC API. +For details about the ScalarDB Cluster gRPC API, refer to the following: + +* [ScalarDB Cluster gRPC API Guide](scalardb-cluster-grpc-api-guide.mdx) +* [ScalarDB Cluster SQL gRPC API Guide](scalardb-cluster-sql-grpc-api-guide.mdx) + +JavaDocs are also available: + +* [ScalarDB Cluster Java Client SDK](https://javadoc.io/doc/com.scalar-labs/scalardb-cluster-java-client-sdk/3.16.0/index.html) +* [ScalarDB Cluster Common](https://javadoc.io/doc/com.scalar-labs/scalardb-cluster-common/3.16.0/index.html) +* [ScalarDB Cluster RPC](https://javadoc.io/doc/com.scalar-labs/scalardb-cluster-rpc/3.16.0/index.html) diff --git a/versioned_docs/version-3.X/scalardb-cluster/encrypt-data-at-rest.mdx b/versioned_docs/version-3.X/scalardb-cluster/encrypt-data-at-rest.mdx new file mode 100644 index 00000000..e7e49d7e --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster/encrypt-data-at-rest.mdx @@ -0,0 +1,325 @@ +--- +tags: + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Encrypt Data at Rest + +import WarningLicenseKeyContact from '/src/components/en-us/_warning-license-key-contact.mdx'; + +This document explains how to encrypt data at rest in ScalarDB. + +## Overview + +ScalarDB can encrypt data stored through it. The encryption feature is similar to transparent data encryption (TDE) in major database systems; therefore, it is transparent to applications. ScalarDB encrypts data before writing it to the backend databases and decrypts it when reading from them. + +Currently, ScalarDB supports column-level encryption, allowing specific columns in a table to be encrypted. + +## Configurations + +To enable the encryption feature, you need to configure `scalar.db.cluster.encryption.enabled` to `true` in the ScalarDB Cluster node configuration file. + +| Name | Description | Default | +|----------------------------------------|-----------------------------------------|---------| +| `scalar.db.cluster.encryption.enabled` | Whether ScalarDB encrypts data at rest. | `false` | + +:::note + +Since encryption is transparent to the client, you don't need to change the client configuration. + +::: + +:::note + +If you enable the encryption feature, you will also need to set `scalar.db.cross_partition_scan.enabled` to `true` for the system namespace (`scalardb` by default) because it performs cross-partition scans internally. + +::: + +The other configurations depend on the encryption implementation you choose. Currently, ScalarDB supports the following encryption implementations: + +- HashiCorp Vault encryption +- Self-encryption + +The following sections explain how to configure each encryption implementation. + +### HashiCorp Vault encryption + +In HashiCorp Vault encryption, ScalarDB uses the [encryption as a service](https://developer.hashicorp.com/vault/tutorials/encryption-as-a-service/eaas-transit) of HashiCorp Vault to encrypt and decrypt data. In this implementation, ScalarDB delegates the management of encryption keys, as well as the encryption and decryption of data, to HashiCorp Vault. + +To use HashiCorp Vault encryption, you need to set the property `scalar.db.cluster.encryption.type` to `vault` in the ScalarDB Cluster node configuration file: + +| Name | Description | Default | +|-------------------------------------|-------------------------------------------------------------|---------| +| `scalar.db.cluster.encryption.type` | Should be set to `vault` to use HashiCorp Vault encryption. | | + +You also need to configure the following properties: + +| Name | Description | Default | +|------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------| +| `scalar.db.cluster.encryption.vault.key_type` | The key type. Currently, `aes128-gcm96`, `aes256-gcm96`, and `chacha20-poly1305` are supported. For details about the key types, see [Key types](https://developer.hashicorp.com/vault/docs/secrets/transit#key-types). | `aes128-gcm96` | +| `scalar.db.cluster.encryption.vault.associated_data_required` | Whether associated data is required for AEAD encryption. | `false` | +| `scalar.db.cluster.encryption.vault.address` | The address of the HashiCorp Vault server. | | +| `scalar.db.cluster.encryption.vault.token` | The token to authenticate with HashiCorp Vault. | | +| `scalar.db.cluster.encryption.vault.namespace` | The namespace of the HashiCorp Vault. This configuration is optional. | | +| `scalar.db.cluster.encryption.vault.transit_secrets_engine_path` | The path of the transit secrets engine. | `transit` | +| `scalar.db.cluster.encryption.vault.column_batch_size` | The number of columns to be included in a single request to the HashiCorp Vault server. | `64` | + +### Self-encryption + +In self-encryption, ScalarDB manages data encryption keys (DEKs) and performs encryption and decryption. ScalarDB generates a DEK for each table when creating the table and stores it in Kubernetes Secrets. + +To use self-encryption, you need to set the property `scalar.db.cluster.encryption.type` to `self` in the ScalarDB Cluster node configuration file: + +| Name | Description | Default | +|-------------------------------------|-------------------------------------------------|---------| +| `scalar.db.cluster.encryption.type` | Should be set to `self` to use self-encryption. | | + +You also need to configure the following properties: + +| Name | Description | Default | +|-------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------| +| `scalar.db.cluster.encryption.self.key_type` | The key type. Currently, `AES128_GCM`, `AES256_GCM`, `AES128_EAX`, `AES256_EAX`, `AES128_CTR_HMAC_SHA256`, `AES256_CTR_HMAC_SHA256`, `CHACHA20_POLY1305`, and `XCHACHA20_POLY1305` are supported. For details about the key types, see [Choose a key type](https://developers.google.com/tink/aead#choose_a_key_type). | `AES128_GCM` | +| `scalar.db.cluster.encryption.self.associated_data_required` | Whether associated data is required for AEAD encryption. | `false` | +| `scalar.db.cluster.encryption.self.kubernetes.secret.namespace_name` | The namespace name of the Kubernetes Secrets. | `default` | +| `scalar.db.cluster.encryption.self.data_encryption_key_cache_expiration_time` | The expiration time of the DEK cache in milliseconds. | `60000` (60 seconds) | + +### Delete the DEK when dropping a table + +By default, ScalarDB does not delete the data encryption key (DEK) associated with a table when the table is dropped. However, you can configure ScalarDB to delete the DEK when dropping a table. To enable this, set the property `scalar.db.cluster.encryption.delete_data_encryption_key_on_drop_table.enabled` to `true` in the ScalarDB Cluster node configuration file: + +| Name | Description | Default | +|---------------------------------------------------------------------------------|------------------------------------------------------------------|---------| +| `scalar.db.cluster.encryption.delete_data_encryption_key_on_drop_table.enabled` | Whether to delete the DEK when dropping a table. | `false` | + +## Limitations + +There are some limitations to the encryption feature: + +- Primary-key columns (partition-key columns and clustering-key columns) cannot be encrypted. +- Secondary-index columns cannot be encrypted. +- Encrypted columns cannot be specified in the WHERE clauses or ORDER BY clauses. +- Encrypted columns are stored in the underlying database as the BLOB type, so encrypted columns that are larger than the maximum size of the BLOB type cannot be stored. For the maximum size of the BLOB type, see [Data-type mapping between ScalarDB and other databases](../schema-loader.mdx#data-type-mapping-between-scalardb-and-other-databases). + +## Wire encryption + +If you enable the encryption feature, enabling wire encryption to protect your data is strongly recommended, especially in production environments. For details about wire encryption, see [Encrypt Wire Communications](encrypt-wire-communications.mdx). + +## Tutorial - Encrypt data by configuring HashiCorp Vault encryption + +This tutorial explains how to encrypt data stored through ScalarDB by using HashiCorp Vault encryption. + +### Prerequisites + +- OpenJDK LTS version (8, 11, 17, or 21) from [Eclipse Temurin](https://adoptium.net/temurin/releases/) +- [Docker](https://www.docker.com/get-started/) 20.10 or later with [Docker Compose](https://docs.docker.com/compose/install/) V2 or later + +:::note + +This tutorial has been tested with OpenJDK from Eclipse Temurin. ScalarDB itself, however, has been tested with JDK distributions from various vendors. For details about the requirements for ScalarDB, including compatible JDK distributions, please see [Requirements](../requirements.mdx). + +::: + + + +### Step 1. Install HashiCorp Vault + +Install HashiCorp Vault by referring to the official HashiCorp documentation, [Install Vault](https://developer.hashicorp.com/vault/tutorials/getting-started/getting-started-install). + +### Step 2. Create the ScalarDB Cluster configuration file + +Create the following configuration file as `scalardb-cluster-node.properties`, replacing `` and `` with your ScalarDB license key and license check certificate values. For more information about the license key and certificate, see [How to Configure a Product License Key](../scalar-licensing/index.mdx). + +```properties +scalar.db.storage=jdbc +scalar.db.contact_points=jdbc:postgresql://postgresql:5432/postgres +scalar.db.username=postgres +scalar.db.password=postgres +scalar.db.cluster.node.standalone_mode.enabled=true +scalar.db.cross_partition_scan.enabled=true +scalar.db.sql.enabled=true + +# Encryption configurations +scalar.db.cluster.encryption.enabled=true +scalar.db.cluster.encryption.type=vault +scalar.db.cluster.encryption.vault.address=http://vault:8200 +scalar.db.cluster.encryption.vault.token=root + +# License key configurations +scalar.db.cluster.node.licensing.license_key= +scalar.db.cluster.node.licensing.license_check_cert_pem= +``` + +### Step 3. Create the Docker Compose configuration file + +Create the following configuration file as `docker-compose.yaml`. + +```yaml +services: + vault: + container_name: "vault" + image: "hashicorp/vault:1.17.3" + ports: + - 8200:8200 + environment: + - VAULT_DEV_ROOT_TOKEN_ID=root + - VAULT_DEV_LISTEN_ADDRESS=0.0.0.0:8200 + cap_add: + - IPC_LOCK + + postgresql: + container_name: "postgresql" + image: "postgres:15" + ports: + - 5432:5432 + environment: + - POSTGRES_PASSWORD=postgres + healthcheck: + test: ["CMD-SHELL", "pg_isready || exit 1"] + interval: 1s + timeout: 10s + retries: 60 + start_period: 30s + + scalardb-cluster-standalone: + container_name: "scalardb-cluster-node" + image: "ghcr.io/scalar-labs/scalardb-cluster-node-byol-premium:3.16.0" + ports: + - 60053:60053 + - 9080:9080 + volumes: + - ./scalardb-cluster-node.properties:/scalardb-cluster/node/scalardb-cluster-node.properties + depends_on: + postgresql: + condition: service_healthy +``` + +### Step 4. Start the HashiCorp Vault server + +Run the following command to start the HashiCorp Vault server in development mode. + +```console +docker compose up vault -d +``` + +Once the HashiCorp Vault server is running, set its environment variables by running the following commands. + +```console +export VAULT_ADDR="http://127.0.0.1:8200" +export VAULT_TOKEN=root +``` + +### Step 5. Enable the transit secrets engine on the HashiCorp Vault server + +Run the following command to enable the transit secrets engine on the HashiCorp Vault server. + +```console +vault secrets enable transit +``` + +### Step 6. Start PostgreSQL and ScalarDB Cluster + +Run the following command to start PostgreSQL and ScalarDB Cluster in standalone mode. + +```console +docker compose up postgresql scalardb-cluster-standalone -d +``` + +It may take a few minutes for ScalarDB Cluster to fully start. + +### Step 7. Connect to ScalarDB Cluster + +To connect to ScalarDB Cluster, this tutorial uses the SQL CLI, a tool for connecting to ScalarDB Cluster and executing SQL queries. You can download the SQL CLI from the [ScalarDB releases page](https://github.com/scalar-labs/scalardb/releases). + +Create a configuration file named `scalardb-cluster-sql-cli.properties`. This file will be used to connect to ScalarDB Cluster by using the SQL CLI. + +```properties +scalar.db.sql.connection_mode=cluster +scalar.db.sql.cluster_mode.contact_points=indirect:localhost +``` + +Then, start the SQL CLI by running the following command. + +```console +java -jar scalardb-cluster-sql-cli-3.16.0-all.jar --config scalardb-cluster-sql-cli.properties +``` + +To begin, create the Coordinator tables required for ScalarDB transaction execution. + +```sql +CREATE COORDINATOR TABLES IF NOT EXISTS; +``` + +Now you're ready to use the database with the encryption feature enabled in ScalarDB Cluster. + +### Step 8. Create a table + +Before creating a table, you need to create a namespace. + +```sql +CREATE NAMESPACE ns; +``` + +Next, create a table. + +```sql +CREATE TABLE ns.tbl ( + id INT PRIMARY KEY, + col1 TEXT ENCRYPTED, + col2 INT ENCRYPTED, + col3 INT); +``` + +By using the `ENCRYPTED` keyword, the data in the specified columns will be encrypted. In this example, the data in `col1` and `col2` will be encrypted. + +### Step 9. Insert data into the table + +To insert data into the table, execute the following SQL query. + +```sql +INSERT INTO ns.tbl (id, col1, col2, col3) VALUES (1, 'data1', 123, 456); +``` + +To verify the inserted data, run the following SQL query. + +```sql +SELECT * FROM ns.tbl; +``` + +```console ++----+-------+------+------+ +| id | col1 | col2 | col3 | ++----+-------+------+------+ +| 1 | data1 | 123 | 456 | ++----+-------+------+------+ +``` + +### Step 10. Verify data encryption + +To verify that the data is encrypted, connect directly to PostgreSQL and check the data. + +:::warning + +Reading or writing data from the backend database directly is not supported in ScalarDB. In such a case, ScalarDB cannot guarantee data consistency. This guide accesses the backend database directly for testing purposes, however, you cannot do this in a production environment. + +::: + +Run the following command to connect to PostgreSQL. + +```console +docker exec -it postgresql psql -U postgres +``` + +Next, execute the following SQL query to check the data in the table. + +```sql +SELECT id, col1, col2, col3 FROM ns.tbl; +``` + +You should see a similar output as below, which confirms that the data in `col1` and `col2` are encrypted. + +```console + id | col1 | col2 | col3 +----+--------------------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------+------ + 1 | \x7661756c743a76313a6b6f76455062316a676e6a4a596b643743765539315a49714d625564545a61697152666c7967367837336e66 | \x7661756c743a76313a4b6244543162764678676d44424b526d7037794f5176423569616e615635304c473079664354514b3866513d | 456 +``` diff --git a/versioned_docs/version-3.X/scalardb-cluster/encrypt-wire-communications.mdx b/versioned_docs/version-3.X/scalardb-cluster/encrypt-wire-communications.mdx new file mode 100644 index 00000000..88dd7a97 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster/encrypt-wire-communications.mdx @@ -0,0 +1,64 @@ +--- +tags: + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Encrypt Wire Communications + +ScalarDB can encrypt wire communications by using Transport Layer Security (TLS). This document explains the configurations for wire encryption in ScalarDB. + +The wire encryption feature encrypts: + +* The communications between the ScalarDB Cluster node and clients. +* The communications between all the ScalarDB Cluster nodes (the cluster's internal communications). + +This feature uses TLS support in gRPC. For details, see the official gRPC [Security Policy](https://github.com/grpc/grpc-java/blob/master/SECURITY.md). + +:::note + +Enabling wire encryption between the ScalarDB Cluster nodes and the underlying databases in production environments is strongly recommended. For instructions on how to enable wire encryption between the ScalarDB Cluster nodes and the underlying databases, please refer to the product documentation for your underlying databases. + +::: + +## Configurations + +This section describes the available configurations for wire encryption. + +### Enable wire encryption in the ScalarDB Cluster nodes + +To enable wire encryption in the ScalarDB Cluster nodes, you need to set `scalar.db.cluster.tls.enabled` to `true`. + +| Name | Description | Default | +|---------------------------------|-------------------------------------------|---------| +| `scalar.db.cluster.tls.enabled` | Whether wire encryption (TLS) is enabled. | `false` | + +You also need to set the following configurations: + +| Name | Description | Default | +|-----------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------| +| `scalar.db.cluster.tls.ca_root_cert_pem` | The custom CA root certificate (PEM data) for TLS communication. | | +| `scalar.db.cluster.tls.ca_root_cert_path` | The custom CA root certificate (file path) for TLS communication. | | +| `scalar.db.cluster.tls.override_authority` | The custom authority for TLS communication. This doesn't change what host is actually connected. This is intended for testing, but may safely be used outside of tests as an alternative to DNS overrides. For example, you can specify the hostname presented in the certificate chain file that you set for `scalar.db.cluster.node.tls.cert_chain_path`. | | +| `scalar.db.cluster.node.tls.cert_chain_path` | The certificate chain file used for TLS communication. | | +| `scalar.db.cluster.node.tls.private_key_path` | The private key file used for TLS communication. | | + +To specify the certificate authority (CA) root certificate, you should set either `scalar.db.cluster.tls.ca_root_cert_pem` or `scalar.db.cluster.tls.ca_root_cert_path`. If you set both, `scalar.db.cluster.tls.ca_root_cert_pem` will be used. + +### Enable wire encryption on the client side + +To enable wire encryption on the client side by using the ScalarDB Cluster Java client SDK, you need to set `scalar.db.cluster.tls.enabled` to `true`. + +| Name | Description | Default | +|---------------------------------|-------------------------------------------|---------| +| `scalar.db.cluster.tls.enabled` | Whether wire encryption (TLS) is enabled. | `false` | + +You also need to set the following configurations: + +| Name | Description | Default | +|--------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------| +| `scalar.db.cluster.tls.ca_root_cert_pem` | The custom CA root certificate (PEM data) for TLS communication. | | +| `scalar.db.cluster.tls.ca_root_cert_path` | The custom CA root certificate (file path) for TLS communication. | | +| `scalar.db.cluster.tls.override_authority` | The custom authority for TLS communication. This doesn't change what host is actually connected. This is intended for testing, but may safely be used outside of tests as an alternative to DNS overrides. For example, you can specify the hostname presented in the certificate chain file that you set for `scalar.db.cluster.node.tls.cert_chain_path`. | | + +To specify the CA root certificate, you should set either `scalar.db.cluster.tls.ca_root_cert_pem` or `scalar.db.cluster.tls.ca_root_cert_path`. If you set both, `scalar.db.cluster.tls.ca_root_cert_pem` will be used. diff --git a/versioned_docs/version-3.X/scalardb-cluster/getting-started-with-scalardb-cluster-dotnet.mdx b/versioned_docs/version-3.X/scalardb-cluster/getting-started-with-scalardb-cluster-dotnet.mdx new file mode 100644 index 00000000..d06000d8 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster/getting-started-with-scalardb-cluster-dotnet.mdx @@ -0,0 +1,439 @@ +--- +tags: + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Getting Started with ScalarDB Cluster via .NET + +This tutorial describes how to create a sample application that uses [ScalarDB Cluster](./index.mdx) through the .NET API. + +## Overview + +This tutorial illustrates the process of creating a sample e-commerce application, where items can be ordered and paid for with a line of credit by using ScalarDB. + +:::note + +Since the focus of the sample application is to demonstrate using ScalarDB, application-specific error handling, authentication processing, and similar functions are not included in the sample application. For details about exception handling, see [Exception Handling in the ScalarDB Cluster .NET Client SDK](../scalardb-cluster-dotnet-client-sdk/exception-handling.mdx). + +::: + +The following diagram shows the system architecture of the sample application: + +```mermaid +stateDiagram-v2 + state "Sample application using the .NET API" as SA + state "Kubernetes Cluster" as KC + state "Service (Envoy)" as SE + state "Pod" as P1 + state "Pod" as P2 + state "Pod" as P3 + state "Envoy" as E1 + state "Envoy" as E2 + state "Envoy" as E3 + state "Service (ScalarDB Cluster)" as SSC + state "ScalarDB Cluster" as SC1 + state "ScalarDB Cluster" as SC2 + state "ScalarDB Cluster" as SC3 + state "PostgreSQL" as PSQL + SA --> SE + state KC { + SE --> E1 + SE --> E2 + SE --> E3 + state P1 { + E1 --> SSC + E2 --> SSC + E3 --> SSC + } + SSC --> SC1 + SSC --> SC2 + SSC --> SC3 + state P2 { + SC1 --> PSQL + SC1 --> SC2 + SC1 --> SC3 + SC2 --> PSQL + SC2 --> SC1 + SC2 --> SC3 + SC3 --> PSQL + SC3 --> SC1 + SC3 --> SC2 + } + state P3 { + PSQL + } + } +``` + +### What you can do in this sample application + +The sample application supports the following types of transactions: + +- Get customer information. +- Place an order by using a line of credit. + - Checks if the cost of the order is below the customer's credit limit. + - If the check passes, records the order history and updates the amount the customer has spent. +- Get order information by order ID. +- Get order information by customer ID. +- Make a payment. + - Reduces the amount the customer has spent. + +## Prerequisites for this sample application + +- [.NET SDK 8.0](https://dotnet.microsoft.com/en-us/download/dotnet/8.0) +- ScalarDB Cluster running on a Kubernetes cluster + - We assume that you have a ScalarDB Cluster running on a Kubernetes cluster that you deployed by following the instructions in [Set Up ScalarDB Cluster on Kubernetes by Using a Helm Chart](setup-scalardb-cluster-on-kubernetes-by-using-helm-chart.mdx). + +:::note + +.NET SDK 8.0 is the version used to create the sample application. For information about all supported versions, see [Requirements](../requirements.mdx#net) + +::: + +## Set up ScalarDB Cluster + +The following sections describe how to set up the sample e-commerce application. + +### Clone the ScalarDB samples repository + +Open **Terminal**, then clone the ScalarDB samples repository by running the following command: + +```console +git clone https://github.com/scalar-labs/scalardb-samples +``` + +Then, go to the directory that contains the sample application by running the following command: + +```console +cd scalardb-samples/scalardb-dotnet-samples/scalardb-cluster-sample +``` + +### Update the referenced version of the ScalarDB.Client package + +To use ScalarDB Cluster, open `ScalarDbClusterSample.csproj` in your preferred text editor. Then, update the version of the referenced `ScalarDB.Client` package, replacing `.` with the version of the deployed ScalarDB Cluster: + +```xml + +``` + +### Modify `scalardb-options.json` + +You need to modify `scalardb-options.json` to connect to ScalarDB Cluster as well. But before doing so, you need to get the `EXTERNAL-IP` address of the Envoy service resource (`scalardb-cluster-envoy`). To get the service resource, run the following command: + +```console +kubectl get svc scalardb-cluster-envoy +``` + +You should see a similar output as below, with different values for `CLUSTER-IP`, `PORT(S)`, and `AGE`: + +```console +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +scalardb-cluster-envoy LoadBalancer 10.105.121.51 localhost 60053:30641/TCP 16h +``` + +In this case, the `EXTERNAL-IP` address is `localhost`. + +Open `scalardb-options.json` by running the following command: + +```console +vim scalardb-options.json +``` + +Then, modify `scalardb-options.json` as follows: + +```json +{ + "ScalarDbOptions": { + "Address": "http://localhost:60053" + } +} +``` + +### Load the initial data + +Before running the sample application, you need to load the initial data by running the following command: + +```console +dotnet run LoadInitialData +``` + +#### Schema details + +Running the command above will also apply the schema. All the tables are created in the `sample` namespace. + +- `sample.customers`: a table that manages customer information + - `credit_limit`: the maximum amount of money that the lender will allow the customer to spend from their line of credit + - `credit_total`: the amount of money that the customer has spent from their line of credit +- `sample.orders`: a table that manages order information +- `sample.statements`: a table that manages order statement information +- `sample.items`: a table that manages information for items to be ordered + +The Entity Relationship Diagram for the schema is as follows: + +![ERD](images/getting-started-ERD.png) + +#### Initial data + +After the initial data has loaded, the following records should be stored in the tables. + +**`sample.customers` table** + +| customer_id | name | credit_limit | credit_total | +|-------------|---------------|--------------|--------------| +| 1 | Yamada Taro | 10000 | 0 | +| 2 | Yamada Hanako | 10000 | 0 | +| 3 | Suzuki Ichiro | 10000 | 0 | + +**`sample.items` table** + +| item_id | name | price | +|---------|--------|-------| +| 1 | Apple | 1000 | +| 2 | Orange | 2000 | +| 3 | Grape | 2500 | +| 4 | Mango | 5000 | +| 5 | Melon | 3000 | + +## Execute transactions and retrieve data in the sample application + +The following sections describe how to execute transactions and retrieve data in the sample e-commerce application. + +### Get customer information + +Start with getting information about the customer whose ID is `1` by running the following command: + +```console +dotnet run GetCustomerInfo 1 +``` + +You should see the following output: + +```console +{ + "id": 1, + "name": "Yamada Taro", + "credit_limit": 10000, + "credit_total": 0 +} +``` + +### Place an order + +Then, have customer ID `1` place an order for three apples and two oranges by running the following command: + +:::note + +The order format in this command is `dotnet run PlaceOrder :,:,..."`. + +::: + +```console +dotnet run PlaceOrder 1 1:3,2:2 +``` + +You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: + +```console +{ + "order_id": "5a22150b-1944-403f-b02c-77183e705d1b" +} +``` + +### Check order details + +Check details about the order by running the following command, replacing `` with the UUID for the `order_id` that was shown after running the previous command: + +```console +dotnet run GetOrder +``` + +You should see a similar output as below, with different UUIDs for `order_id` and `timestamp`: + +```console +{ + "order": { + "order_id": "5a22150b-1944-403f-b02c-77183e705d1b", + "timestamp": 1743143358216, + "customer_id": 1, + "customer_name": "Yamada Taro", + "statements": [ + { + "item_id": 1, + "item_name": "Apple", + "price": 1000, + "count": 3, + "total": 3000 + }, + { + "item_id": 2, + "item_name": "Orange", + "price": 2000, + "count": 2, + "total": 4000 + } + ], + "total": 7000 + } +} +``` + +### Place another order + +Place an order for one melon that uses the remaining amount in `credit_total` for customer ID `1` by running the following command: + +```console +dotnet run PlaceOrder 1 5:1 +``` + +You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: + +```console +{ + "order_id": "79fcd778-94ba-4e8b-b993-cdb88a6186a8" +} +``` + +### Check order history + +Get the history of all orders for customer ID `1` by running the following command: + +```console +dotnet run GetOrders 1 +``` + +You should see a similar output as below, with different UUIDs for `order_id` and `timestamp`, which shows the history of all orders for customer ID `1`: + +```console +{ + "orders": [ + { + "order_id": "5a22150b-1944-403f-b02c-77183e705d1b", + "timestamp": 1743143358216, + "customer_id": 1, + "customer_name": "Yamada Taro", + "statements": [ + { + "item_id": 1, + "item_name": "Apple", + "price": 1000, + "count": 3, + "total": 3000 + }, + { + "item_id": 2, + "item_name": "Orange", + "price": 2000, + "count": 2, + "total": 4000 + } + ], + "total": 7000 + }, + { + "order_id": "79fcd778-94ba-4e8b-b993-cdb88a6186a8", + "timestamp": 1743143505436, + "customer_id": 1, + "customer_name": "Yamada Taro", + "statements": [ + { + "item_id": 5, + "item_name": "Melon", + "price": 3000, + "count": 1, + "total": 3000 + } + ], + "total": 3000 + } + ] +} +``` + +### Check credit total + +Get the credit total for customer ID `1` by running the following command: + +```console +dotnet run GetCustomerInfo 1 +``` + +You should see the following output, which shows that customer ID `1` has reached their `credit_limit` in `credit_total` and cannot place anymore orders: + +```console +{ + "id": 1, + "name": "Yamada Taro", + "credit_limit": 10000, + "credit_total": 10000 +} +``` + +Try to place an order for one grape and one mango by running the following command: + +```console +dotnet run PlaceOrder 1 3:1,4:1 +``` + +You should see a similar output as below, which shows that the order failed because the `credit_total` amount would exceed the `credit_limit` amount. + +```console +Unhandled exception: System.Exception: Credit limit exceeded (17500 > 10000) + at ScalarDbClusterSample.Sample.PlaceOrder(Int32 customerId, IReadOnlyDictionary`2 itemCounts) in /scalar-labs/scalardb-samples/scalardb-dotnet-samples/scalardb-cluster-sample/Sample.cs:line 254 + at ScalarDbClusterSample.Commands.PlaceOrderCommand.<>c.<b__6_0>d.MoveNext() in /scalar-labs/scalardb-samples/scalardb-dotnet-samples/scalardb-cluster-sample/Commands/PlaceOrderCommand.cs:line 47 +--- End of stack trace from previous location --- +... +``` + +### Make a payment + +To continue making orders, customer ID `1` must make a payment to reduce the `credit_total` amount. + +Make a payment by running the following command: + +```console +dotnet run Repayment 1 8000 +``` + +Then, check the `credit_total` amount for customer ID `1` by running the following command: + +```console +dotnet run GetCustomerInfo 1 +``` + +You should see the following output, which shows that a payment was applied to customer ID `1`, reducing the `credit_total` amount: + +```console +{ + "id": 1, + "name": "Yamada Taro", + "credit_limit": 10000, + "credit_total": 2000 +} +``` + +Now that customer ID `1` has made a payment, place an order for one grape and one melon by running the following command: + +```console +dotnet run PlaceOrder 1 3:1,4:1 +``` + +You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: + +```console +{ + "order_id": "ecd68f46-e248-4f2e-b581-620e9019bf5b" +} +``` + +## See also + +For details about developing applications that use ScalarDB Cluster with the .NET API, refer to the following: + +- [ScalarDB Cluster .NET Client SDK Overview](../scalardb-cluster-dotnet-client-sdk/index.mdx) + +For details about the ScalarDB Cluster gRPC API, refer to the following: + +- [ScalarDB Cluster gRPC API Guide](scalardb-cluster-grpc-api-guide.mdx) +- [ScalarDB Cluster SQL gRPC API Guide](scalardb-cluster-sql-grpc-api-guide.mdx) diff --git a/versioned_docs/version-3.X/scalardb-cluster/getting-started-with-scalardb-cluster-graphql.mdx b/versioned_docs/version-3.X/scalardb-cluster/getting-started-with-scalardb-cluster-graphql.mdx new file mode 100644 index 00000000..30bf1fb3 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster/getting-started-with-scalardb-cluster-graphql.mdx @@ -0,0 +1,335 @@ +--- +tags: + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Getting Started with ScalarDB Cluster GraphQL + +import JavadocLink from '/src/theme/JavadocLink.js'; + +This tutorial describes how to use ScalarDB Cluster GraphQL. + +## Prerequisites + +- One of the following Java Development Kits (JDKs): + - [Oracle JDK](https://www.oracle.com/java/technologies/downloads/) LTS version (8, 11, 17, or 21) + - [OpenJDK](https://openjdk.org/install/) LTS version (8, 11, 17, or 21) +- ScalarDB Cluster running on a Kubernetes cluster + - We assume that you have a ScalarDB Cluster running on a Kubernetes cluster that you deployed by following the instructions in [Set Up ScalarDB Cluster on Kubernetes by Using a Helm Chart](setup-scalardb-cluster-on-kubernetes-by-using-helm-chart.mdx). + +## Sample application + +This tutorial illustrates the process of creating an electronic money application, where money can be transferred between accounts. + +The following diagram shows the system architecture of the sample application: + +``` + +----------------------------------------------------------------------------------------------------------------------------------------+ + | [Kubernetes Cluster] | + | | + | [Pod] [Pod] [Pod] | + | | + | +-------+ | + | +---> | Envoy | ---+ | + | | +-------+ | | + | | | | + +------------------------+ | +---------+ | +-------+ | +--------------------+ | + | Schema Loader | --+-> | Service | ---+---> | Envoy | ---+---------> | Service | ---+ | + | (indirect client mode) | | | (Envoy) | | +-------+ | | (ScalarDB Cluster) | | | + +------------------------+ | +---------+ | | +--------------------+ | +-----------------------+ | + | | +-------+ | | +---> | ScalarDB Cluster Node | ---+ | + | +---> | Envoy | ---+ | | +-----------------------+ | | + | +-------+ | | | | + | | | +-----------------------+ | +------------+ | + | +---+---> | ScalarDB Cluster Node | ---+---> | PostgreSQL | | + | | | +-----------------------+ | +------------+ | + | | | | | + | | | +-----------------------+ | | + | | +---> | ScalarDB Cluster Node | ---+ | + | | +-----------------------+ | + +------------+ | +----------------------------+ | | + | Browser | ------+---------------------------------------> | Service | ---+ | + | (GraphiQL) | | | (ScalarDB Cluster GraphQL) | | + +------------+ | +----------------------------+ | + | | + +----------------------------------------------------------------------------------------------------------------------------------------+ +``` + +## Step 1. Create `schema.json` + +The following is a simple example schema. + +Create `schema.json`, and add the following to the file: + +```json +{ + "emoney.account": { + "transaction": true, + "partition-key": [ + "id" + ], + "clustering-key": [], + "columns": { + "id": "TEXT", + "balance": "INT" + } + } +} +``` + +## Step 2. Create `database.properties` + +You need to create `database.properties` for the Schema Loader for ScalarDB Cluster. +But first, you need to get the `EXTERNAL-IP` address of the service resource of Envoy (`scalardb-cluster-envoy`). + +To see the `EXTERNAL-IP` address, run the following command: + +```console +kubectl get svc scalardb-cluster-envoy +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +scalardb-cluster-envoy LoadBalancer 10.105.121.51 localhost 60053:30641/TCP 16h +``` + +In this case, the `EXTERNAL-IP` address is `localhost`. + +Then, create `database.properties`, and add the following to the file: + +```properties +scalar.db.transaction_manager=cluster +scalar.db.contact_points=indirect:localhost +``` + +To connect to ScalarDB Cluster, you need to specify `cluster` for the `scalar.db.transaction_manager` property. +In addition, you will use the `indirect` client mode and connect to the service resource of Envoy in this tutorial. +For details about the client modes, see [Developer Guide for ScalarDB Cluster with the Java API](developer-guide-for-scalardb-cluster-with-java-api.mdx). + +## Step 3. Load a schema + +To load a schema via ScalarDB Cluster, you need to use the dedicated Schema Loader for ScalarDB Cluster (Schema Loader for Cluster). +Using the Schema Loader for Cluster is basically the same as using the [Schema Loader for ScalarDB](../schema-loader.mdx) except the name of the JAR file is different. +You can download the Schema Loader for Cluster from [ScalarDB Releases](https://github.com/scalar-labs/scalardb/releases/tag/v3.16.0). +After downloading the JAR file, you can run the Schema Loader for Cluster with the following command: + +```console +java -jar scalardb-cluster-schema-loader-3.16.0-all.jar --config database.properties -f schema.json --coordinator +``` + +## Step 4. Run operations from GraphiQL + +In ScalarDB Cluster, if the `scalar.db.graphql.graphiql` property is set to `true` (`true` is the default value), the GraphiQL IDE will be available. + +To get the `EXTERNAL-IP` address of the service resource of ScalarDB Cluster GraphQL (`scalardb-cluster-graphql`), run the following command: + +```console +kubectl get svc scalardb-cluster-graphql +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +scalardb-cluster-graphql LoadBalancer 10.105.74.214 localhost 8080:30514/TCP 16h +``` + +In this case, the `EXTERNAL-IP` address is `localhost`, and the endpoint URL of GraphiQL IDE is `http://localhost:8080/graphql`. +Opening that URL with your web browser will take you to the GraphiQL screen. + +Let's insert the first record. +In the left pane, paste the following mutation, then push the triangle-shaped `Execute Query` button at the top of the window. + +```graphql +mutation PutUser1 { + account_put(put: {key: {id: "user1"}, values: {balance: 1000}}) +} +``` + +ScalarDB GraphQL always runs queries with transactions. +The above query starts a new transaction, executes a ScalarDB Put command, and commits the transaction at the end of the execution. + +The following response from the GraphQL server will appear in the right pane: + +```json +{ + "data": { + "account_put": true + } +} +``` + +The `"data"` field contains the result of the execution. +This response shows the `account_put` field of the mutation was successful. +The result type of mutations is `Boolean!`, which indicates whether the operation succeeded or not. + +Next, let's get the record you just inserted. +Paste the following query next to the previous mutation in the left pane, and click the `Execute Query` button. +Since you don't delete the `mutation PutUser1` above, a pull-down menu will appear below the button, and you can choose which operation should be executed. Choose `GetUser1`, as shown below: + +```graphql +query GetUser1 { + account_get(get: {key: {id: "user1"}}) { + account { + id + balance + } + } +} +``` + +You should get the following result in the right pane: + +```json +{ + "data": { + "account_get": { + "account": { + "id": "user1", + "balance": 1000 + } + } + } +} +``` + +### Mappings between GraphQL API and ScalarDB Java API + +The automatically generated GraphQL schema defines queries, mutations, and object types for input/output to allow you to run CRUD operations for all the tables in the target namespaces. These operations are designed to match the ScalarDB APIs defined in the interface. + +Assuming you have an `account` table in a namespace, the following queries and mutations will be generated: + +| ScalarDB API | GraphQL root type | GraphQL field | +|--------------------------------------------------------|-------------------|------------------------------------------------------------------------------------| +| `get(Get get)` | `Query` | `account_get(get: account_GetInput!): account_GetPayload` | +| `scan(Scan scan)` | `Query` | `account_scan(scan: account_ScanInput!): account_ScanPayload` | +| `put(Put put)` | `Mutation` | `account_put(put: account_PutInput!): Boolean!` | +| `put(java.util.List puts)` | `Mutation` | `account_bulkPut(put: [account_PutInput!]!): Boolean!` | +| `delete(Delete delete)` | `Mutation` | `account_delete(delete: account_DeleteInput!): Boolean!` | +| `delete(java.util.List deletes)` | `Mutation` | `account_bulkDelete(delete: [account_DeleteInput!]!): Boolean!` | +| `mutate(java.util.List mutations)` | `Mutation` | `account_mutate(put: [account_PutInput!]delete: [account_DeleteInput!]): Boolean!` | + +Note that the `scan` field is not generated for a table with no clustering key. +This is the reason why the `account_scan` field is not available in this electronic money sample application. + +You can see all generated GraphQL types in GraphiQL's Documentation Explorer (the `< Docs` link at the top-left corner). + +## Step 5. Run a transaction across multiple requests from GraphiQL + +Let's run a transaction that spans multiple GraphQL requests. + +The generated schema provides the `@transaction` directive that allows you to identify transactions. +You can use this directive with both queries and mutations. + +Before starting a transaction, you need to insert the necessary record with the following mutation: + +```graphql +mutation PutUser2 { + account_put(put: {key: {id: "user2"}, values: {balance: 1000}}) +} +``` + +### Start a transaction before running an operation + +Running the following to add a `@transaction` directive with no arguments to a query or mutation directs the execution to start a new transaction: + +```graphql +query GetAccounts @transaction { + user1: account_get(get: {key: {id: "user1"}}) { + account { balance } + } + user2: account_get(get: {key: {id: "user2"}}) { + account { balance } + } +} +``` + +After running the above command, you will get a result with a transaction ID in the `extensions` field. +The `id` value in the extensions is the transaction ID in which the operation in the request was run. +In this case, the following is the new ID of the transaction just started by the request: + +```json +{ + "data": { + "user1": { + "account": { + "balance": 1000 + } + }, + "user2": { + "account": { + "balance": 1000 + } + } + }, + "extensions": { + "transaction": { + "id": "c88da8a6-a13f-4857-82fe-45f1ab4150f9" + } + } +} +``` + +### Run an operation in a continued transaction + +To run the next queries or mutations in the transaction you started, specify the transaction ID as the `id` argument of the `@transaction`. +The following example updates two accounts you retrieved in the previous example by transferring a balance from user1's account to user2's account in the same transaction: + +```graphql +mutation Transfer @transaction(id: "c88da8a6-a13f-4857-82fe-45f1ab4150f9") { + user1: account_put(put: {key: {id: "user1"}, values: {balance: 750}}) + user2: account_put(put: {key: {id: "user2"}, values: {balance: 1250}}) +} +``` + +Note that a transaction started with GraphQL has a timeout of 1 minute (by default) and will be aborted automatically if it exceeds the timeout. + +### Commit a transaction + +To commit the continued transaction, specify both the `id` and the `commit: true` flag as arguments of the `@transaction` directive: + +```graphql +query GetAndCommit @transaction(id: "c88da8a6-a13f-4857-82fe-45f1ab4150f9", commit: true) { + user1: account_get(get: {key: {id: "user1"}}) { + account { balance } + } + user2: account_get(get: {key: {id: "user2"}}) { + account { balance } + } +} +``` + +**Note:** If you specify a `commit: true` flag without an `id` argument like `@transaction(commit: true)`, a new transaction will start and be committed just for one operation. +This behavior is exactly the same as not specifying the `@transaction` directive, as seen in the above examples using GraphiQL. +In other words, you can omit the directive itself when `@transaction(commit: true)` is specified. + +### Abort or roll back a transaction + +If you need to abort or roll back a transaction explicitly, you can use the `abort` or `rollback` mutation fields interchangeably (both have the same effect and usage). +Note that you cannot mix these fields with any other operations, so you must specify only the `abort` or `rollback` mutation field as follows: + +```graphql +mutation AbortTx @transaction(id: "c88da8a6-a13f-4857-82fe-45f1ab4150f9") { + abort +} +``` + +Or: + +```graphql +mutation RollbackTx @transaction(id: "c88da8a6-a13f-4857-82fe-45f1ab4150f9") { + rollback +} +``` + +## See also + +For other ScalarDB Cluster tutorials, see the following: + +- [Getting Started with ScalarDB Cluster](getting-started-with-scalardb-cluster.mdx) +- [Getting Started with ScalarDB Cluster SQL via JDBC](getting-started-with-scalardb-cluster-sql-jdbc.mdx) +- [Getting Started with ScalarDB Cluster SQL via Spring Data JDBC for ScalarDB](getting-started-with-scalardb-cluster-sql-spring-data-jdbc.mdx) +- [Getting Started with Using Go for ScalarDB Cluster](getting-started-with-using-go-for-scalardb-cluster.mdx) +- [Getting Started with Using Python for ScalarDB Cluster](getting-started-with-using-python-for-scalardb-cluster.mdx) + +For details about developing applications that use ScalarDB Cluster with the Java API, refer to the following: + +- [Developer Guide for ScalarDB Cluster with the Java API](developer-guide-for-scalardb-cluster-with-java-api.mdx) + +For details about the ScalarDB Cluster gRPC API, refer to the following: + +- [ScalarDB Cluster gRPC API Guide](scalardb-cluster-grpc-api-guide.mdx) +- [ScalarDB Cluster SQL gRPC API Guide](scalardb-cluster-sql-grpc-api-guide.mdx) diff --git a/versioned_docs/version-3.X/scalardb-cluster/getting-started-with-scalardb-cluster-sql-dotnet.mdx b/versioned_docs/version-3.X/scalardb-cluster/getting-started-with-scalardb-cluster-sql-dotnet.mdx new file mode 100644 index 00000000..8e40f7b3 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster/getting-started-with-scalardb-cluster-sql-dotnet.mdx @@ -0,0 +1,439 @@ +--- +tags: + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Getting Started with ScalarDB Cluster SQL via .NET + +This tutorial describes how to create a sample application that uses [ScalarDB Cluster](./index.mdx) SQL through the .NET API. + +## Overview + +This tutorial illustrates the process of creating a sample e-commerce application, where items can be ordered and paid for with a line of credit by using ScalarDB. + +:::note + +Since the focus of the sample application is to demonstrate using ScalarDB, application-specific error handling, authentication processing, and similar functions are not included in the sample application. For details about exception handling, see [Exception Handling in the ScalarDB Cluster .NET Client SDK](../scalardb-cluster-dotnet-client-sdk/exception-handling.mdx). + +::: + +The following diagram shows the system architecture of the sample application: + +```mermaid +stateDiagram-v2 + state "Sample application using the .NET API" as SA + state "Kubernetes Cluster" as KC + state "Service (Envoy)" as SE + state "Pod" as P1 + state "Pod" as P2 + state "Pod" as P3 + state "Envoy" as E1 + state "Envoy" as E2 + state "Envoy" as E3 + state "Service (ScalarDB Cluster)" as SSC + state "ScalarDB Cluster" as SC1 + state "ScalarDB Cluster" as SC2 + state "ScalarDB Cluster" as SC3 + state "PostgreSQL" as PSQL + SA --> SE + state KC { + SE --> E1 + SE --> E2 + SE --> E3 + state P1 { + E1 --> SSC + E2 --> SSC + E3 --> SSC + } + SSC --> SC1 + SSC --> SC2 + SSC --> SC3 + state P2 { + SC1 --> PSQL + SC1 --> SC2 + SC1 --> SC3 + SC2 --> PSQL + SC2 --> SC1 + SC2 --> SC3 + SC3 --> PSQL + SC3 --> SC1 + SC3 --> SC2 + } + state P3 { + PSQL + } + } +``` + +### What you can do in this sample application + +The sample application supports the following types of transactions: + +- Get customer information. +- Place an order by using a line of credit. + - Checks if the cost of the order is below the customer's credit limit. + - If the check passes, records the order history and updates the amount the customer has spent. +- Get order information by order ID. +- Get order information by customer ID. +- Make a payment. + - Reduces the amount the customer has spent. + +## Prerequisites for this sample application + +- [.NET SDK 8.0](https://dotnet.microsoft.com/en-us/download/dotnet/8.0) +- ScalarDB Cluster running on a Kubernetes cluster + - We assume that you have a ScalarDB Cluster running on a Kubernetes cluster that you deployed by following the instructions in [How to Deploy ScalarDB Cluster Locally](setup-scalardb-cluster-on-kubernetes-by-using-helm-chart.mdx). + +:::note + +.NET SDK 8.0 is the version used to create the sample application. For information about all supported versions, see [Requirements](../requirements.mdx#net) + +::: + +## Set up ScalarDB Cluster + +The following sections describe how to set up the sample e-commerce application. + +### Clone the ScalarDB samples repository + +Open **Terminal**, then clone the ScalarDB samples repository by running the following command: + +```console +git clone https://github.com/scalar-labs/scalardb-samples +``` + +Then, go to the directory that contains the sample application by running the following command: + +```console +cd scalardb-samples/scalardb-dotnet-samples/scalardb-cluster-sql-sample +``` + +### Update the referenced version of the ScalarDB.Client package + +To use ScalarDB Cluster, open `ScalarDbClusterSample.csproj` in your preferred text editor. Then, update the version of the referenced `ScalarDB.Client` package, replacing `.` with the version of the deployed ScalarDB Cluster: + +```xml + +``` + +### Modify `scalardb-options.json` + +You need to modify `scalardb-options.json` to connect to ScalarDB Cluster as well. But before doing so, you need to get the `EXTERNAL-IP` address of the Envoy service resource (`scalardb-cluster-envoy`). To get the service resource, run the following command: + +```console +kubectl get svc scalardb-cluster-envoy +``` + +You should see a similar output as below, with different values for `CLUSTER-IP`, `PORT(S)`, and `AGE`: + +```console +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +scalardb-cluster-envoy LoadBalancer 10.105.121.51 localhost 60053:30641/TCP 16h +``` + +In this case, the `EXTERNAL-IP` address is `localhost`. + +Open `scalardb-options.json` by running the following command: + +```console +vim scalardb-options.json +``` + +Then, modify `scalardb-options.json` as follows: + +```json +{ + "ScalarDbOptions": { + "Address": "http://localhost:60053" + } +} +``` + +### Load the initial data + +Before running the sample application, you need to load the initial data by running the following command: + +```console +dotnet run LoadInitialData +``` + +#### Schema details + +Running the command above will also apply the schema. All the tables are created in the `sample` namespace. + +- `sample.customers`: a table that manages customer information + - `credit_limit`: the maximum amount of money that the lender will allow the customer to spend from their line of credit + - `credit_total`: the amount of money that the customer has spent from their line of credit +- `sample.orders`: a table that manages order information +- `sample.statements`: a table that manages order statement information +- `sample.items`: a table that manages information for items to be ordered + +The Entity Relationship Diagram for the schema is as follows: + +![ERD](images/getting-started-ERD.png) + +#### Initial data + +After the initial data has loaded, the following records should be stored in the tables. + +**`sample.customers` table** + +| customer_id | name | credit_limit | credit_total | +|-------------|---------------|--------------|--------------| +| 1 | Yamada Taro | 10000 | 0 | +| 2 | Yamada Hanako | 10000 | 0 | +| 3 | Suzuki Ichiro | 10000 | 0 | + +**`sample.items` table** + +| item_id | name | price | +|---------|--------|-------| +| 1 | Apple | 1000 | +| 2 | Orange | 2000 | +| 3 | Grape | 2500 | +| 4 | Mango | 5000 | +| 5 | Melon | 3000 | + +## Execute transactions and retrieve data in the sample application + +The following sections describe how to execute transactions and retrieve data in the sample e-commerce application. + +### Get customer information + +Start with getting information about the customer whose ID is `1` by running the following command: + +```console +dotnet run GetCustomerInfo 1 +``` + +You should see the following output: + +```console +{ + "id": 1, + "name": "Yamada Taro", + "credit_limit": 10000, + "credit_total": 0 +} +``` + +### Place an order + +Then, have customer ID `1` place an order for three apples and two oranges by running the following command: + +:::note + +The order format in this command is `dotnet run PlaceOrder :,:,..."`. + +::: + +```console +dotnet run PlaceOrder 1 1:3,2:2 +``` + +You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: + +```console +{ + "order_id": "5a22150b-1944-403f-b02c-77183e705d1b" +} +``` + +### Check order details + +Check details about the order by running the following command, replacing `` with the UUID for the `order_id` that was shown after running the previous command: + +```console +dotnet run GetOrder +``` + +You should see a similar output as below, with different UUIDs for `order_id` and `timestamp`: + +```console +{ + "order": { + "order_id": "5a22150b-1944-403f-b02c-77183e705d1b", + "timestamp": 1743143358216, + "customer_id": 1, + "customer_name": "Yamada Taro", + "statements": [ + { + "item_id": 1, + "item_name": "Apple", + "price": 1000, + "count": 3, + "total": 3000 + }, + { + "item_id": 2, + "item_name": "Orange", + "price": 2000, + "count": 2, + "total": 4000 + } + ], + "total": 7000 + } +} +``` + +### Place another order + +Place an order for one melon that uses the remaining amount in `credit_total` for customer ID `1` by running the following command: + +```console +dotnet run PlaceOrder 1 5:1 +``` + +You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: + +```console +{ + "order_id": "79fcd778-94ba-4e8b-b993-cdb88a6186a8" +} +``` + +### Check order history + +Get the history of all orders for customer ID `1` by running the following command: + +```console +dotnet run GetOrders 1 +``` + +You should see a similar output as below, with different UUIDs for `order_id` and `timestamp`, which shows the history of all orders for customer ID `1`: + +```console +{ + "orders": [ + { + "order_id": "5a22150b-1944-403f-b02c-77183e705d1b", + "timestamp": 1743143358216, + "customer_id": 1, + "customer_name": "Yamada Taro", + "statements": [ + { + "item_id": 1, + "item_name": "Apple", + "price": 1000, + "count": 3, + "total": 3000 + }, + { + "item_id": 2, + "item_name": "Orange", + "price": 2000, + "count": 2, + "total": 4000 + } + ], + "total": 7000 + }, + { + "order_id": "79fcd778-94ba-4e8b-b993-cdb88a6186a8", + "timestamp": 1743143505436, + "customer_id": 1, + "customer_name": "Yamada Taro", + "statements": [ + { + "item_id": 5, + "item_name": "Melon", + "price": 3000, + "count": 1, + "total": 3000 + } + ], + "total": 3000 + } + ] +} +``` + +### Check credit total + +Get the credit total for customer ID `1` by running the following command: + +```console +dotnet run GetCustomerInfo 1 +``` + +You should see the following output, which shows that customer ID `1` has reached their `credit_limit` in `credit_total` and cannot place anymore orders: + +```console +{ + "id": 1, + "name": "Yamada Taro", + "credit_limit": 10000, + "credit_total": 10000 +} +``` + +Try to place an order for one grape and one mango by running the following command: + +```console +dotnet run PlaceOrder 1 3:1,4:1 +``` + +You should see a similar output as below, which shows that the order failed because the `credit_total` amount would exceed the `credit_limit` amount. + +```console +Unhandled exception: System.Exception: Credit limit exceeded (17500 > 10000) + at ScalarDbClusterSqlSample.Sample.PlaceOrder(Int32 customerId, IReadOnlyDictionary`2 itemCounts) in /scalar-labs/scalardb-samples/scalardb-dotnet-samples/scalardb-cluster-sql-sample/Sample.cs:line 237 + at ScalarDbClusterSqlSample.Commands.PlaceOrderCommand.<>c.<b__6_0>d.MoveNext() in /scalar-labs/scalardb-samples/scalardb-dotnet-samples/scalardb-cluster-sql-sample/Commands/PlaceOrderCommand.cs:line 47 +--- End of stack trace from previous location --- +... +``` + +### Make a payment + +To continue making orders, customer ID `1` must make a payment to reduce the `credit_total` amount. + +Make a payment by running the following command: + +```console +dotnet run Repayment 1 8000 +``` + +Then, check the `credit_total` amount for customer ID `1` by running the following command: + +```console +dotnet run GetCustomerInfo 1 +``` + +You should see the following output, which shows that a payment was applied to customer ID `1`, reducing the `credit_total` amount: + +```console +{ + "id": 1, + "name": "Yamada Taro", + "credit_limit": 10000, + "credit_total": 2000 +} +``` + +Now that customer ID `1` has made a payment, place an order for one grape and one melon by running the following command: + +```console +dotnet run PlaceOrder 1 3:1,4:1 +``` + +You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: + +```console +{ + "order_id": "ecd68f46-e248-4f2e-b581-620e9019bf5b" +} +``` + +## See also + +For details about developing applications that use ScalarDB Cluster with the .NET API, refer to the following: + +- [ScalarDB Cluster .NET Client SDK Overview](../scalardb-cluster-dotnet-client-sdk/index.mdx) +- [Getting Started with Distributed SQL Transactions in the ScalarDB Cluster .NET Client SDK](../scalardb-cluster-dotnet-client-sdk/getting-started-with-distributed-sql-transactions.mdx) + +For details about the ScalarDB Cluster gRPC API, refer to the following: + +- [ScalarDB Cluster gRPC API Guide](scalardb-cluster-grpc-api-guide.mdx) +- [ScalarDB Cluster SQL gRPC API Guide](scalardb-cluster-sql-grpc-api-guide.mdx) diff --git a/versioned_docs/version-3.X/scalardb-cluster/getting-started-with-scalardb-cluster-sql-jdbc.mdx b/versioned_docs/version-3.X/scalardb-cluster/getting-started-with-scalardb-cluster-sql-jdbc.mdx new file mode 100644 index 00000000..5ed6d41c --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster/getting-started-with-scalardb-cluster-sql-jdbc.mdx @@ -0,0 +1,232 @@ +--- +tags: + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Getting Started with ScalarDB Cluster SQL via JDBC + +This tutorial describes how to create a sample application by using ScalarDB Cluster SQL via JDBC. + +## Prerequisites for this sample application + +- OpenJDK LTS version (8, 11, 17, or 21) from [Eclipse Temurin](https://adoptium.net/temurin/releases/) +- ScalarDB Cluster running on a Kubernetes cluster + - We assume that you have a ScalarDB Cluster running on a Kubernetes cluster that you deployed by following the instructions in [Set Up ScalarDB Cluster on Kubernetes by Using a Helm Chart](setup-scalardb-cluster-on-kubernetes-by-using-helm-chart.mdx). + +:::note + +This sample application has been tested with OpenJDK from Eclipse Temurin. ScalarDB itself, however, has been tested with JDK distributions from various vendors. For details about the requirements for ScalarDB, including compatible JDK distributions, please see [Requirements](../requirements.mdx). + +::: + +## Sample application + +This tutorial illustrates the process of creating a sample e-commerce application, where items can be ordered and paid for with a line of credit by using ScalarDB JDBC. + +The following diagram shows the system architecture of the sample application: + +``` + +------------------------------------------------------------------------------------------------------------------------------+ + | [Kubernetes Cluster] | + | | + | [Pod] [Pod] [Pod] | + +------------------------+ | | + | SQL CLI | | +-------+ +-----------------------+ | + | (indirect client mode) | --+ | +---> | Envoy | ---+ +---> | ScalarDB Cluster Node | ---+ | + +------------------------+ | | | +-------+ | | +-----------------------+ | | + | | | | | | | + | | +---------+ | +-------+ | +--------------------+ | +-----------------------+ | +------------+ | + +--+-> | Service | ---+---> | Envoy | ---+---> | Service | ---+---> | ScalarDB Cluster Node | ---+---> | PostgreSQL | | + | | | (Envoy) | | +-------+ | | (ScalarDB Cluster) | | +-----------------------+ | +------------+ | + +------------------------+ | | +---------+ | | +--------------------+ | | | + | Sample application | | | | +-------+ | | +-----------------------+ | | + | with ScalarDB JDBC | --+ | +---> | Envoy | ---+ +---> | ScalarDB Cluster Node | ---+ | + | (indirect client mode) | | +-------+ +-----------------------+ | + +------------------------+ | | + +------------------------------------------------------------------------------------------------------------------------------+ +``` + +## Step 1. Clone the ScalarDB Samples repository + +```console +git clone https://github.com/scalar-labs/scalardb-samples.git +cd scalardb-samples/scalardb-sql-jdbc-sample +``` + +## Step 2. Modify `scalardb-sql.properties` + +You need to modify `scalardb-sql.properties` to connect to ScalarDB Cluster as well. +But before doing so, you need to get the `EXTERNAL-IP` address of the service resource of Envoy (`scalardb-cluster-envoy`) as follows: + +```console +kubectl get svc scalardb-cluster-envoy +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +scalardb-cluster-envoy LoadBalancer 10.105.121.51 localhost 60053:30641/TCP 16h +``` + +In this case, the `EXTERNAL-IP` address is `localhost`. + +Next, open `scalardb-sql.properties`: + +```console +vim scalardb-sql.properties +``` + +Then, modify `scalardb-sql.properties` as follows: + +```properties +scalar.db.sql.connection_mode=cluster +scalar.db.sql.cluster_mode.contact_points=indirect:localhost +``` + +To connect to ScalarDB Cluster, you need to specify `cluster` for the `scalar.db.sql.connection_mode` property. +In addition, you will use the `indirect` client mode and connect to the service resource of Envoy in this tutorial. +For details about the client modes, see [Developer Guide for ScalarDB Cluster with the Java API](developer-guide-for-scalardb-cluster-with-java-api.mdx). + +## Step 3. Load a schema + +To load a schema, you need to use [the SQL CLI](developer-guide-for-scalardb-cluster-with-java-api.mdx#sql-cli). You can download the SQL CLI from [ScalarDB Releases](https://github.com/scalar-labs/scalardb/releases/tag/v3.16.0). After downloading the JAR file, you can use SQL CLI for Cluster by running the following command: + +```console +java -jar scalardb-cluster-sql-cli-3.16.0-all.jar --config scalardb-sql.properties --file schema.sql +``` + +## Step 4. Load the initial data + +Before running the sample application, you need to load the initial data by running the following command: + +```console +./gradlew run --args="LoadInitialData" +``` + +After the initial data has loaded, the following records should be stored in the tables: + +- For the `sample.customers` table: + +| customer_id | name | credit_limit | credit_total | +|-------------|---------------|--------------|--------------| +| 1 | Yamada Taro | 10000 | 0 | +| 2 | Yamada Hanako | 10000 | 0 | +| 3 | Suzuki Ichiro | 10000 | 0 | + +- For the `sample.items` table: + +| item_id | name | price | +|---------|--------|-------| +| 1 | Apple | 1000 | +| 2 | Orange | 2000 | +| 3 | Grape | 2500 | +| 4 | Mango | 5000 | +| 5 | Melon | 3000 | + +## Step 5. Run the sample application + +Let's start with getting information about the customer whose ID is `1`: + +```console +./gradlew run --args="GetCustomerInfo 1" +... +{"id": 1, "name": "Yamada Taro", "credit_limit": 10000, "credit_total": 0} +... +``` + +Then, place an order for three apples and two oranges by using customer ID `1`. +Note that the order format is `:,:,...`: + +```console +./gradlew run --args="PlaceOrder 1 1:3,2:2" +... +{"order_id": "454f9c97-f456-44fd-96da-f527187fe39b"} +... +``` + +You can see that running this command shows the order ID. + +Let's check the details of the order by using the order ID: + +```console +./gradlew run --args="GetOrder 454f9c97-f456-44fd-96da-f527187fe39b" +... +{"order": {"order_id": "454f9c97-f456-44fd-96da-f527187fe39b","timestamp": 1685602722821,"customer_id": 1,"customer_name": "Yamada Taro","statement": [{"item_id": 1, "name": "Apple", "price": 1000, "count": 3},{"item_id": 2, "name": "Orange", "price": 2000, "count": 2}],"total": 7000}} +... +``` + +Then, let's place another order and get the order history of customer ID `1`: + +```console +./gradlew run --args="PlaceOrder 1 5:1" +... +{"order_id": "3f40c718-59ec-48aa-a6fe-2fdaf12ad094"} +... +./gradlew run --args="GetOrders 1" +... +{"order": [{"order_id": "454f9c97-f456-44fd-96da-f527187fe39b","timestamp": 1685602722821,"customer_id": 1,"customer_name": "Yamada Taro","statement": [{"item_id": 1, "name": "Apple", "price": 1000, "count": 3},{"item_id": 2, "name": "Orange", "price": 2000, "count": 2}],"total": 7000},{"order_id": "3f40c718-59ec-48aa-a6fe-2fdaf12ad094","timestamp": 1685602811718,"customer_id": 1,"customer_name": "Yamada Taro","statement": [{"item_id": 5, "name": "Melon", "price": 3000, "count": 1}],"total": 3000}]} +... +``` + +This order history is shown in descending order by timestamp. + +The customer's current `credit_total` is `10000`. +Since the customer has now reached their `credit_limit`, which was shown when retrieving their information, they cannot place anymore orders. + +```console +./gradlew run --args="GetCustomerInfo 1" +... +{"id": 1, "name": "Yamada Taro", "credit_limit": 10000, "credit_total": 10000} +... +./gradlew run --args="PlaceOrder 1 3:1,4:1" +... +java.lang.RuntimeException: Credit limit exceeded + at sample.Sample.placeOrder(Sample.java:184) + at sample.command.PlaceOrderCommand.call(PlaceOrderCommand.java:32) + at sample.command.PlaceOrderCommand.call(PlaceOrderCommand.java:8) + at picocli.CommandLine.executeUserObject(CommandLine.java:2041) + at picocli.CommandLine.access$1500(CommandLine.java:148) + at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461) + at picocli.CommandLine$RunLast.handle(CommandLine.java:2453) + at picocli.CommandLine$RunLast.handle(CommandLine.java:2415) + at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273) + at picocli.CommandLine$RunLast.execute(CommandLine.java:2417) + at picocli.CommandLine.execute(CommandLine.java:2170) + at sample.command.SampleCommand.main(SampleCommand.java:35) +... +``` + +After making a payment, the customer will be able to place orders again. + +```console +./gradlew run --args="Repayment 1 8000" +... +./gradlew run --args="GetCustomerInfo 1" +... +{"id": 1, "name": "Yamada Taro", "credit_limit": 10000, "credit_total": 2000} +... +./gradlew run --args="PlaceOrder 1 3:1,4:1" +... +{"order_id": "fb71279d-88ea-4974-a102-0ec4e7d65e25"} +... +``` + +## Source code of the sample application + +To learn more about ScalarDB Cluster SQL JDBC, you can check the [source code of the sample application](https://github.com/scalar-labs/scalardb-samples/tree/main/scalardb-sql-jdbc-sample/src/main/java/sample). + +## See also + +For other ScalarDB Cluster tutorials, see the following: + +- [Getting Started with ScalarDB Cluster](getting-started-with-scalardb-cluster.mdx) +- [Getting Started with ScalarDB Cluster GraphQL](getting-started-with-scalardb-cluster-graphql.mdx) +- [Getting Started with ScalarDB Cluster SQL via Spring Data JDBC for ScalarDB](getting-started-with-scalardb-cluster-sql-spring-data-jdbc.mdx) +- [Getting Started with Using Go for ScalarDB Cluster](getting-started-with-using-go-for-scalardb-cluster.mdx) +- [Getting Started with Using Python for ScalarDB Cluster](getting-started-with-using-python-for-scalardb-cluster.mdx) + +For details about developing applications that use ScalarDB Cluster with the Java API, refer to the following: + +- [Developer Guide for ScalarDB Cluster with the Java API](developer-guide-for-scalardb-cluster-with-java-api.mdx) + +For details about the ScalarDB Cluster gRPC API, refer to the following: + +- [ScalarDB Cluster gRPC API Guide](scalardb-cluster-grpc-api-guide.mdx) +- [ScalarDB Cluster SQL gRPC API Guide](scalardb-cluster-sql-grpc-api-guide.mdx) diff --git a/versioned_docs/version-3.X/scalardb-cluster/getting-started-with-scalardb-cluster-sql-linq.mdx b/versioned_docs/version-3.X/scalardb-cluster/getting-started-with-scalardb-cluster-sql-linq.mdx new file mode 100644 index 00000000..40e243d0 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster/getting-started-with-scalardb-cluster-sql-linq.mdx @@ -0,0 +1,439 @@ +--- +tags: + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Getting Started with ScalarDB Cluster SQL via .NET and LINQ + +This tutorial describes how to create a sample application that uses [ScalarDB Cluster](./index.mdx) SQL through LINQ. + +## Overview + +This tutorial illustrates the process of creating a sample e-commerce application, where items can be ordered and paid for with a line of credit by using ScalarDB. + +:::note + +Since the focus of the sample application is to demonstrate using ScalarDB, application-specific error handling, authentication processing, and similar functions are not included in the sample application. For details about exception handling, see [Exception Handling in the ScalarDB Cluster .NET Client SDK](../scalardb-cluster-dotnet-client-sdk/exception-handling.mdx). + +::: + +The following diagram shows the system architecture of the sample application: + +```mermaid +stateDiagram-v2 + state "Sample application using the .NET API" as SA + state "Kubernetes Cluster" as KC + state "Service (Envoy)" as SE + state "Pod" as P1 + state "Pod" as P2 + state "Pod" as P3 + state "Envoy" as E1 + state "Envoy" as E2 + state "Envoy" as E3 + state "Service (ScalarDB Cluster)" as SSC + state "ScalarDB Cluster" as SC1 + state "ScalarDB Cluster" as SC2 + state "ScalarDB Cluster" as SC3 + state "PostgreSQL" as PSQL + SA --> SE + state KC { + SE --> E1 + SE --> E2 + SE --> E3 + state P1 { + E1 --> SSC + E2 --> SSC + E3 --> SSC + } + SSC --> SC1 + SSC --> SC2 + SSC --> SC3 + state P2 { + SC1 --> PSQL + SC1 --> SC2 + SC1 --> SC3 + SC2 --> PSQL + SC2 --> SC1 + SC2 --> SC3 + SC3 --> PSQL + SC3 --> SC1 + SC3 --> SC2 + } + state P3 { + PSQL + } + } +``` + +### What you can do in this sample application + +The sample application supports the following types of transactions: + +- Get customer information. +- Place an order by using a line of credit. + - Checks if the cost of the order is below the customer's credit limit. + - If the check passes, records the order history and updates the amount the customer has spent. +- Get order information by order ID. +- Get order information by customer ID. +- Make a payment. + - Reduces the amount the customer has spent. + +## Prerequisites for this sample application + +- [.NET SDK 8.0](https://dotnet.microsoft.com/en-us/download/dotnet/8.0) +- ScalarDB Cluster running on a Kubernetes cluster + - We assume that you have a ScalarDB Cluster running on a Kubernetes cluster that you deployed by following the instructions in [How to Deploy ScalarDB Cluster Locally](setup-scalardb-cluster-on-kubernetes-by-using-helm-chart.mdx). + +:::note + +.NET SDK 8.0 is the version used to create the sample application. For information about all supported versions, see [Requirements](../requirements.mdx#net) + +::: + +## Set up ScalarDB Cluster + +The following sections describe how to set up the sample e-commerce application. + +### Clone the ScalarDB samples repository + +Open **Terminal**, then clone the ScalarDB samples repository by running the following command: + +```console +git clone https://github.com/scalar-labs/scalardb-samples +``` + +Then, go to the directory that contains the sample application by running the following command: + +```console +cd scalardb-samples/scalardb-dotnet-samples/scalardb-cluster-linq-sample +``` + +### Update the referenced version of the ScalarDB.Client package + +To use ScalarDB Cluster, open `ScalarDbClusterSample.csproj` in your preferred text editor. Then, update the version of the referenced `ScalarDB.Client` package, replacing `.` with the version of the deployed ScalarDB Cluster: + +```xml + +``` + +### Modify `scalardb-options.json` + +You need to modify `scalardb-options.json` to connect to ScalarDB Cluster as well. But before doing so, you need to get the `EXTERNAL-IP` address of the Envoy service resource (`scalardb-cluster-envoy`). To get the service resource, run the following command: + +```console +kubectl get svc scalardb-cluster-envoy +``` + +You should see a similar output as below, with different values for `CLUSTER-IP`, `PORT(S)`, and `AGE`: + +```console +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +scalardb-cluster-envoy LoadBalancer 10.105.121.51 localhost 60053:30641/TCP 16h +``` + +In this case, the `EXTERNAL-IP` address is `localhost`. + +Open `scalardb-options.json` by running the following command: + +```console +vim scalardb-options.json +``` + +Then, modify `scalardb-options.json` as follows: + +```json +{ + "ScalarDbOptions": { + "Address": "http://localhost:60053" + } +} +``` + +### Load the initial data + +Before running the sample application, you need to load the initial data by running the following command: + +```console +dotnet run LoadInitialData +``` + +#### Schema details + +Running the command above will also apply the schema. All the tables are created in the `sample` namespace. + +- `sample.customers`: a table that manages customer information + - `credit_limit`: the maximum amount of money that the lender will allow the customer to spend from their line of credit + - `credit_total`: the amount of money that the customer has spent from their line of credit +- `sample.orders`: a table that manages order information +- `sample.statements`: a table that manages order statement information +- `sample.items`: a table that manages information for items to be ordered + +The Entity Relationship Diagram for the schema is as follows: + +![ERD](images/getting-started-ERD.png) + +#### Initial data + +After the initial data has loaded, the following records should be stored in the tables. + +**`sample.customers` table** + +| customer_id | name | credit_limit | credit_total | +|-------------|---------------|--------------|--------------| +| 1 | Yamada Taro | 10000 | 0 | +| 2 | Yamada Hanako | 10000 | 0 | +| 3 | Suzuki Ichiro | 10000 | 0 | + +**`sample.items` table** + +| item_id | name | price | +|---------|--------|-------| +| 1 | Apple | 1000 | +| 2 | Orange | 2000 | +| 3 | Grape | 2500 | +| 4 | Mango | 5000 | +| 5 | Melon | 3000 | + +## Execute transactions and retrieve data in the sample application + +The following sections describe how to execute transactions and retrieve data in the sample e-commerce application. + +### Get customer information + +Start with getting information about the customer whose ID is `1` by running the following command: + +```console +dotnet run GetCustomerInfo 1 +``` + +You should see the following output: + +```console +{ + "id": 1, + "name": "Yamada Taro", + "credit_limit": 10000, + "credit_total": 0 +} +``` + +### Place an order + +Then, have customer ID `1` place an order for three apples and two oranges by running the following command: + +:::note + +The order format in this command is `dotnet run PlaceOrder :,:,..."`. + +::: + +```console +dotnet run PlaceOrder 1 1:3,2:2 +``` + +You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: + +```console +{ + "order_id": "5a22150b-1944-403f-b02c-77183e705d1b" +} +``` + +### Check order details + +Check details about the order by running the following command, replacing `` with the UUID for the `order_id` that was shown after running the previous command: + +```console +dotnet run GetOrder +``` + +You should see a similar output as below, with different UUIDs for `order_id` and `timestamp`: + +```console +{ + "order": { + "order_id": "5a22150b-1944-403f-b02c-77183e705d1b", + "timestamp": 1743143358216, + "customer_id": 1, + "customer_name": "Yamada Taro", + "statements": [ + { + "item_id": 1, + "item_name": "Apple", + "price": 1000, + "count": 3, + "total": 3000 + }, + { + "item_id": 2, + "item_name": "Orange", + "price": 2000, + "count": 2, + "total": 4000 + } + ], + "total": 7000 + } +} +``` + +### Place another order + +Place an order for one melon that uses the remaining amount in `credit_total` for customer ID `1` by running the following command: + +```console +dotnet run PlaceOrder 1 5:1 +``` + +You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: + +```console +{ + "order_id": "79fcd778-94ba-4e8b-b993-cdb88a6186a8" +} +``` + +### Check order history + +Get the history of all orders for customer ID `1` by running the following command: + +```console +dotnet run GetOrders 1 +``` + +You should see a similar output as below, with different UUIDs for `order_id` and `timestamp`, which shows the history of all orders for customer ID `1`: + +```console +{ + "orders": [ + { + "order_id": "5a22150b-1944-403f-b02c-77183e705d1b", + "timestamp": 1743143358216, + "customer_id": 1, + "customer_name": "Yamada Taro", + "statements": [ + { + "item_id": 1, + "item_name": "Apple", + "price": 1000, + "count": 3, + "total": 3000 + }, + { + "item_id": 2, + "item_name": "Orange", + "price": 2000, + "count": 2, + "total": 4000 + } + ], + "total": 7000 + }, + { + "order_id": "79fcd778-94ba-4e8b-b993-cdb88a6186a8", + "timestamp": 1743143505436, + "customer_id": 1, + "customer_name": "Yamada Taro", + "statements": [ + { + "item_id": 5, + "item_name": "Melon", + "price": 3000, + "count": 1, + "total": 3000 + } + ], + "total": 3000 + } + ] +} +``` + +### Check credit total + +Get the credit total for customer ID `1` by running the following command: + +```console +dotnet run GetCustomerInfo 1 +``` + +You should see the following output, which shows that customer ID `1` has reached their `credit_limit` in `credit_total` and cannot place anymore orders: + +```console +{ + "id": 1, + "name": "Yamada Taro", + "credit_limit": 10000, + "credit_total": 10000 +} +``` + +Try to place an order for one grape and one mango by running the following command: + +```console +dotnet run PlaceOrder 1 3:1,4:1 +``` + +You should see a similar output as below, which shows that the order failed because the `credit_total` amount would exceed the `credit_limit` amount. + +```console +Unhandled exception: System.Exception: Credit limit exceeded (17500 > 10000) + at ScalarDbClusterLinqSample.Sample.PlaceOrder(Int32 customerId, IReadOnlyDictionary`2 itemCounts) in /scalar-labs/scalardb-samples/scalardb-dotnet-samples/scalardb-cluster-linq-sample/Sample.cs:line 145 + at ScalarDbClusterLinqSample.Commands.PlaceOrderCommand.<>c.<b__6_0>d.MoveNext() in /scalar-labs/scalardb-samples/scalardb-dotnet-samples/scalardb-cluster-linq-sample/Commands/PlaceOrderCommand.cs:line 47 +--- End of stack trace from previous location --- +... +``` + +### Make a payment + +To continue making orders, customer ID `1` must make a payment to reduce the `credit_total` amount. + +Make a payment by running the following command: + +```console +dotnet run Repayment 1 8000 +``` + +Then, check the `credit_total` amount for customer ID `1` by running the following command: + +```console +dotnet run GetCustomerInfo 1 +``` + +You should see the following output, which shows that a payment was applied to customer ID `1`, reducing the `credit_total` amount: + +```console +{ + "id": 1, + "name": "Yamada Taro", + "credit_limit": 10000, + "credit_total": 2000 +} +``` + +Now that customer ID `1` has made a payment, place an order for one grape and one melon by running the following command: + +```console +dotnet run PlaceOrder 1 3:1,4:1 +``` + +You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: + +```console +{ + "order_id": "ecd68f46-e248-4f2e-b581-620e9019bf5b" +} +``` + +## See also + +For details about developing applications that use ScalarDB Cluster with the .NET API, refer to the following: + +- [ScalarDB Cluster .NET Client SDK Overview](../scalardb-cluster-dotnet-client-sdk/index.mdx) +- [Getting Started with LINQ in the ScalarDB Cluster .NET Client SDK](../scalardb-cluster-dotnet-client-sdk/getting-started-with-linq.mdx) + +For details about the ScalarDB Cluster gRPC API, refer to the following: + +- [ScalarDB Cluster gRPC API Guide](scalardb-cluster-grpc-api-guide.mdx) +- [ScalarDB Cluster SQL gRPC API Guide](scalardb-cluster-sql-grpc-api-guide.mdx) diff --git a/versioned_docs/version-3.X/scalardb-cluster/getting-started-with-scalardb-cluster-sql-spring-data-jdbc.mdx b/versioned_docs/version-3.X/scalardb-cluster/getting-started-with-scalardb-cluster-sql-spring-data-jdbc.mdx new file mode 100644 index 00000000..076b66c0 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster/getting-started-with-scalardb-cluster-sql-spring-data-jdbc.mdx @@ -0,0 +1,268 @@ +--- +tags: + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Getting Started with ScalarDB Cluster SQL via Spring Data JDBC for ScalarDB + +This tutorial describes how to create a sample application by using ScalarDB Cluster SQL via Spring Data JDBC for ScalarDB. + +## Prerequisites for this sample application + +- OpenJDK LTS version (8, 11, 17, or 21) from [Eclipse Temurin](https://adoptium.net/temurin/releases/) +- ScalarDB Cluster running on a Kubernetes cluster + - We assume that you have a ScalarDB Cluster running on a Kubernetes cluster that you deployed by following the instructions in [Set Up ScalarDB Cluster on Kubernetes by Using a Helm Chart](setup-scalardb-cluster-on-kubernetes-by-using-helm-chart.mdx). + +:::note + +This sample application has been tested with OpenJDK from Eclipse Temurin. ScalarDB itself, however, has been tested with JDK distributions from various vendors. For details about the requirements for ScalarDB, including compatible JDK distributions, please see [Requirements](../requirements.mdx). + +::: + +## Sample application + +This tutorial illustrates the process of creating a sample e-commerce application, where items can be ordered and paid for with a line of credit by using Spring Data JDBC for ScalarDB. + +The following diagram shows the system architecture of the sample application: + +``` + +------------------------------------------------------------------------------------------------------------------------------+ + | [Kubernetes Cluster] | + | | + | [Pod] [Pod] [Pod] | + +------------------------+ | | + | SQL CLI | | +-------+ +-----------------------+ | + | (indirect client mode) | --+ | +---> | Envoy | ---+ +---> | ScalarDB Cluster Node | ---+ | + +------------------------+ | | | +-------+ | | +-----------------------+ | | + | | | | | | | + | | +---------+ | +-------+ | +--------------------+ | +-----------------------+ | +------------+ | + +--+-> | Service | ---+---> | Envoy | ---+---> | Service | ---+---> | ScalarDB Cluster Node | ---+---> | PostgreSQL | | + +------------------------+ | | | (Envoy) | | +-------+ | | (ScalarDB Cluster) | | +-----------------------+ | +------------+ | + | Sample application | | | +---------+ | | +--------------------+ | | | + | with Spring Data JDBC | | | | +-------+ | | +-----------------------+ | | + | for ScalarDB | --+ | +---> | Envoy | ---+ +---> | ScalarDB Cluster Node | ---+ | + | (indirect client mode) | | +-------+ +-----------------------+ | + +------------------------+ | | + +------------------------------------------------------------------------------------------------------------------------------+ +``` + +## Step 1. Clone the ScalarDB Samples repository + +```console +git clone https://github.com/scalar-labs/scalardb-samples.git +cd scalardb-samples/spring-data-sample +``` + +## Step 2. Modify `scalardb-sql.properties` + +You need to modify `scalardb-sql.properties` to connect to ScalarDB Cluster as well. +But before doing so, you need to get the `EXTERNAL-IP` address of the service resource of Envoy (`scalardb-cluster-envoy`) as follows: + +```console +kubectl get svc scalardb-cluster-envoy +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +scalardb-cluster-envoy LoadBalancer 10.105.121.51 localhost 60053:30641/TCP 16h +``` + +In this case, the `EXTERNAL-IP` address is `localhost`. + +Next, open `scalardb-sql.properties`: + +```console +vim scalardb-sql.properties +``` + +Then, modify `scalardb-sql.properties` as follows: + +```properties +scalar.db.sql.connection_mode=cluster +scalar.db.sql.cluster_mode.contact_points=indirect:localhost +``` + +To connect to ScalarDB Cluster, you need to specify `cluster` for the `scalar.db.sql.connection_mode` property. +In addition, you will use the `indirect` client mode and connect to the service resource of Envoy in this tutorial. +For details about the client modes, see [Developer Guide for ScalarDB Cluster with the Java API](developer-guide-for-scalardb-cluster-with-java-api.mdx). + +## Step 3. Load a schema + +To load a schema, you need to use [the SQL CLI](developer-guide-for-scalardb-cluster-with-java-api.mdx#sql-cli). You can download the SQL CLI from [ScalarDB Releases](https://github.com/scalar-labs/scalardb/releases/tag/v3.16.0). After downloading the JAR file, you can use SQL CLI for Cluster by running the following command: + +```console +java -jar scalardb-cluster-sql-cli-3.16.0-all.jar --config scalardb-sql.properties --file schema.sql +``` + +## Step 4. Modify `application.properties` + +Then, you need to modify `application.properties` to connect to ScalarDB Cluster as well: + +```console +vim src/main/resources/application.properties +``` + +Similar to `scalardb-sql.properties`, you need to specify `cluster` for the `scalar.db.sql.connection_mode` property and use the `indirect` client mode. +To do so, modify `application.properties` as follows: + +```properties +spring.datasource.driver-class-name=com.scalar.db.sql.jdbc.SqlJdbcDriver +spring.datasource.url=jdbc:scalardb:\ +?scalar.db.sql.connection_mode=cluster\ +&scalar.db.sql.cluster_mode.contact_points=indirect:localhost\ +&scalar.db.consensus_commit.isolation_level=SERIALIZABLE\ +&scalar.db.sql.default_namespace_name=sample +``` + +## Step 5. Load the initial data + +Before running the sample application, you need to load the initial data by running the following command: + +```console +./gradlew run --args="LoadInitialData" +``` + +After the initial data has loaded, the following records should be stored in the tables: + +- For the `sample.customers` table: + +| customer_id | name | credit_limit | credit_total | +|-------------|---------------|--------------|--------------| +| 1 | Yamada Taro | 10000 | 0 | +| 2 | Yamada Hanako | 10000 | 0 | +| 3 | Suzuki Ichiro | 10000 | 0 | + +- For the `sample.items` table: + +| item_id | name | price | +|---------|--------|-------| +| 1 | Apple | 1000 | +| 2 | Orange | 2000 | +| 3 | Grape | 2500 | +| 4 | Mango | 5000 | +| 5 | Melon | 3000 | + +## Step 6. Run the sample application + +Let's start with getting information about the customer whose ID is `1`: + +```console +./gradlew run --args="GetCustomerInfo 1" +... +{"customer_id":1,"name":"Yamada Taro","credit_limit":10000,"credit_total":0} +... +``` + +Then, place an order for three apples and two oranges by using customer ID `1`. Note that the order format is `:,:,...`: + +```console +./gradlew run --args="PlaceOrder 1 1:3,2:2" +... +{"order_id":"2358ab35-5819-4f8f-acb1-12e73d97d34e","customer_id":1,"timestamp":1677478005400} +... +``` + +You can see that running this command shows the order ID. + +Let's check the details of the order by using the order ID: + +```console +./gradlew run --args="GetOrder 2358ab35-5819-4f8f-acb1-12e73d97d34e" +... +{"order_id":"2358ab35-5819-4f8f-acb1-12e73d97d34e","timestamp":1677478005400,"customer_id":1,"customer_name":"Yamada Taro","statements":[{"item_id":1,"item_name":"Apple","price":1000,"count":3,"total":3000},{"item_id":2,"item_name":"Orange","price":2000,"count":2,"total":4000}],"total":7000} +... +``` + +Then, let's place another order and get the order history of customer ID `1`: + +```console +./gradlew run --args="PlaceOrder 1 5:1" +... +{"order_id":"46062b16-b71b-46f9-a9ff-dc6b0991259b","customer_id":1,"timestamp":1677478201428} +... +./gradlew run --args="GetOrders 1" +... +[{"order_id":"46062b16-b71b-46f9-a9ff-dc6b0991259b","timestamp":1677478201428,"customer_id":1,"customer_name":"Yamada Taro","statements":[{"item_id":5,"item_name":"Melon","price":3000,"count":1,"total":3000}],"total":3000},{"order_id":"2358ab35-5819-4f8f-acb1-12e73d97d34e","timestamp":1677478005400,"customer_id":1,"customer_name":"Yamada Taro","statements":[{"item_id":1,"item_name":"Apple","price":1000,"count":3,"total":3000},{"item_id":2,"item_name":"Orange","price":2000,"count":2,"total":4000}],"total":7000}] +... +``` + +This order history is shown in descending order by timestamp. + +The customer's current `credit_total` is `10000`. Since the customer has now reached their `credit_limit`, which was shown when retrieving their information, they cannot place anymore orders. + +```console +./gradlew run --args="GetCustomerInfo 1" +... +{"id": 1, "name": "Yamada Taro", "credit_limit": 10000, "credit_total": 10000} +... +./gradlew run --args="PlaceOrder 1 3:1,4:1" +... +java.lang.RuntimeException: Credit limit exceeded. limit:10000, total:17500 + at sample.SampleService.placeOrder(SampleService.java:102) + at sample.SampleService$$FastClassBySpringCGLIB$$1123c447.invoke() + at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) + at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793) + at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) + at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) + at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) + at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) + at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) + at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) + at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) + at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708) + at sample.SampleService$$EnhancerBySpringCGLIB$$a94e1d9.placeOrder() + at sample.command.PlaceOrderCommand.call(PlaceOrderCommand.java:37) + at sample.command.PlaceOrderCommand.call(PlaceOrderCommand.java:13) + at picocli.CommandLine.executeUserObject(CommandLine.java:2041) + at picocli.CommandLine.access$1500(CommandLine.java:148) + at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461) + at picocli.CommandLine$RunLast.handle(CommandLine.java:2453) + at picocli.CommandLine$RunLast.handle(CommandLine.java:2415) + at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273) + at picocli.CommandLine$RunLast.execute(CommandLine.java:2417) + at picocli.CommandLine.execute(CommandLine.java:2170) + at sample.SampleApp.run(SampleApp.java:26) + at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:768) + at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:752) + at org.springframework.boot.SpringApplication.run(SpringApplication.java:314) + at org.springframework.boot.SpringApplication.run(SpringApplication.java:1303) + at org.springframework.boot.SpringApplication.run(SpringApplication.java:1292) + at sample.SampleApp.main(SampleApp.java:35) +... +``` + +After making a payment, the customer will be able to place orders again. + +```console +./gradlew run --args="Repayment 1 8000" +... +./gradlew run --args="GetCustomerInfo 1" +... +{"customer_id":1,"name":"Yamada Taro","credit_limit":10000,"credit_total":2000} +... +./gradlew run --args="PlaceOrder 1 3:1,4:1" +... +{"order_id":"0350947a-9003-46f2-870e-6aa4b2df0f1f","customer_id":1,"timestamp":1677478728134} +... +``` + +## Source code of the sample application + +To learn more about Spring Data JDBC for ScalarDB, you can check the [source code of the sample application](https://github.com/scalar-labs/scalardb-samples/tree/main/spring-data-sample/src/main). + +## See also + +For other ScalarDB Cluster tutorials, see the following: + +- [Getting Started with ScalarDB Cluster](getting-started-with-scalardb-cluster.mdx) +- [Getting Started with ScalarDB Cluster GraphQL](getting-started-with-scalardb-cluster-graphql.mdx) +- [Getting Started with ScalarDB Cluster SQL via JDBC](getting-started-with-scalardb-cluster-sql-jdbc.mdx) +- [Getting Started with Using Go for ScalarDB Cluster](getting-started-with-using-go-for-scalardb-cluster.mdx) +- [Getting Started with Using Python for ScalarDB Cluster](getting-started-with-using-python-for-scalardb-cluster.mdx) + +For details about developing applications that use ScalarDB Cluster with the Java API, refer to the following: + +- [Developer Guide for ScalarDB Cluster with the Java API](developer-guide-for-scalardb-cluster-with-java-api.mdx) + +For details about the ScalarDB Cluster gRPC API, refer to the following: + +- [ScalarDB Cluster gRPC API Guide](scalardb-cluster-grpc-api-guide.mdx) +- [ScalarDB Cluster SQL gRPC API Guide](scalardb-cluster-sql-grpc-api-guide.mdx) diff --git a/versioned_docs/version-3.X/scalardb-cluster/getting-started-with-scalardb-cluster.mdx b/versioned_docs/version-3.X/scalardb-cluster/getting-started-with-scalardb-cluster.mdx new file mode 100644 index 00000000..d8f1db4f --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster/getting-started-with-scalardb-cluster.mdx @@ -0,0 +1,405 @@ +--- +tags: + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Getting Started with ScalarDB Cluster + +This tutorial describes how to create a sample application that uses [ScalarDB Cluster](./index.mdx) through the Java API. + +## Overview + +This tutorial illustrates the process of creating a sample e-commerce application, where items can be ordered and paid for with a line of credit by using ScalarDB. + +:::note + +Since the focus of the sample application is to demonstrate using ScalarDB, application-specific error handling, authentication processing, and similar functions are not included in the sample application. For details about exception handling in ScalarDB, see [Handle exceptions](../api-guide.mdx#how-to-handle-exceptions). + +::: + +The following diagram shows the system architecture of the sample application: + +```mermaid +stateDiagram-v2 + state "Schema Loader
(indirect client mode)" as SL + state "Sample application using the Java API
(indirect client mode)" as SA + state "Kubernetes Cluster" as KC + state "Service (Envoy)" as SE + state "Pod" as P1 + state "Pod" as P2 + state "Pod" as P3 + state "Envoy" as E1 + state "Envoy" as E2 + state "Envoy" as E3 + state "Service (ScalarDB Cluster)" as SSC + state "ScalarDB Cluster" as SC1 + state "ScalarDB Cluster" as SC2 + state "ScalarDB Cluster" as SC3 + state "PostgreSQL" as PSQL + SL --> SE + SA --> SE + state KC { + SE --> E1 + SE --> E2 + SE --> E3 + state P1 { + E1 --> SSC + E2 --> SSC + E3 --> SSC + } + SSC --> SC1 + SSC --> SC2 + SSC --> SC3 + state P2 { + SC1 --> PSQL + SC1 --> SC2 + SC1 --> SC3 + SC2 --> PSQL + SC2 --> SC1 + SC2 --> SC3 + SC3 --> PSQL + SC3 --> SC1 + SC3 --> SC2 + } + state P3 { + PSQL + } + } +``` + +### What you can do in this sample application + +The sample application supports the following types of transactions: + +- Get customer information. +- Place an order by using a line of credit. + - Checks if the cost of the order is below the customer's credit limit. + - If the check passes, records the order history and updates the amount the customer has spent. +- Get order information by order ID. +- Get order information by customer ID. +- Make a payment. + - Reduces the amount the customer has spent. + +## Prerequisites for this sample application + +- OpenJDK LTS version (8, 11, 17, or 21) from [Eclipse Temurin](https://adoptium.net/temurin/releases/) +- ScalarDB Cluster running on a Kubernetes cluster + - We assume that you have a ScalarDB Cluster running on a Kubernetes cluster that you deployed by following the instructions in [Set Up ScalarDB Cluster on Kubernetes by Using a Helm Chart](setup-scalardb-cluster-on-kubernetes-by-using-helm-chart.mdx). + +:::note + +This sample application has been tested with OpenJDK from Eclipse Temurin. ScalarDB itself, however, has been tested with JDK distributions from various vendors. For details about the requirements for ScalarDB, including compatible JDK distributions, please see [Requirements](../requirements.mdx). + +::: + +## Set up ScalarDB Cluster + +The following sections describe how to set up the sample e-commerce application. + +### Clone the ScalarDB samples repository + +Open **Terminal**, then clone the ScalarDB samples repository by running the following command: + +```console +git clone https://github.com/scalar-labs/scalardb-samples +``` + +Then, go to the directory that contains the sample application by running the following command: + +```console +cd scalardb-samples/scalardb-sample +``` + +### Modify `build.gradle` + +To use ScalarDB Cluster, open `build.gradle` in your preferred text editor. Then, delete the existing dependency for `com.scalar-labs:scalardb` from the `dependencies` section, and add the following dependency to the `dependencies` section: + +```gradle +dependencies { + ... + + implementation 'com.scalar-labs:scalardb-cluster-java-client-sdk:3.16.0' +} +``` + +### Modify `database.properties` + +You need to modify `database.properties` to connect to ScalarDB Cluster as well. But before doing so, you need to get the `EXTERNAL-IP` address of the Envoy service resource (`scalardb-cluster-envoy`). To get the service resource, run the following command: + +```console +kubectl get svc scalardb-cluster-envoy +``` + +You should see a similar output as below, with different values for `CLUSTER-IP`, `PORT(S)`, and `AGE`: + +```console +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +scalardb-cluster-envoy LoadBalancer 10.105.121.51 localhost 60053:30641/TCP 16h +``` + +In this case, the `EXTERNAL-IP` address is `localhost`. + +In `database.properties`, you need to specify `cluster` for the `scalar.db.transaction_manager` property and use `indirect` as the client mode for `scalar.db.contact_points` to connect to the Envoy service resource. + +Open `database.properties` by running the following command: + +```console +vim database.properties +``` + +Then, modify `database.properties` as follows: + +```properties +scalar.db.transaction_manager=cluster +scalar.db.contact_points=indirect:localhost +``` + +:::note + +For details about the client modes, see [Developer Guide for ScalarDB Cluster with the Java API](developer-guide-for-scalardb-cluster-with-java-api.mdx). + +::: + +### Load the schema + +The database schema (the method in which the data will be organized) for the sample application has already been defined in [`schema.json`](https://github.com/scalar-labs/scalardb-samples/tree/main/scalardb-sample/schema.json). + +To apply the schema, go to [ScalarDB Releases](https://github.com/scalar-labs/scalardb/releases/tag/v3.16.0) and download the ScalarDB Cluster Schema Loader to the `scalardb-samples/scalardb-sample` folder. + +Then, run the following command: + +```console +java -jar scalardb-cluster-schema-loader-3.16.0-all.jar --config database.properties -f schema.json --coordinator +``` + +#### Schema details + +As shown in [`schema.json`](https://github.com/scalar-labs/scalardb-samples/tree/main/scalardb-sample/schema.json) for the sample application, all the tables are created in the `sample` namespace. + +- `sample.customers`: a table that manages customer information + - `credit_limit`: the maximum amount of money that the lender will allow the customer to spend from their line of credit + - `credit_total`: the amount of money that the customer has spent from their line of credit +- `sample.orders`: a table that manages order information +- `sample.statements`: a table that manages order statement information +- `sample.items`: a table that manages information for items to be ordered + +The Entity Relationship Diagram for the schema is as follows: + +![ERD](images/getting-started-ERD.png) + +### Load the initial data + +Before running the sample application, you need to load the initial data by running the following command: + +```console +./gradlew run --args="LoadInitialData" +``` + +After the initial data has loaded, the following records should be stored in the tables. + +**`sample.customers` table** + +| customer_id | name | credit_limit | credit_total | +|-------------|---------------|--------------|--------------| +| 1 | Yamada Taro | 10000 | 0 | +| 2 | Yamada Hanako | 10000 | 0 | +| 3 | Suzuki Ichiro | 10000 | 0 | + +**`sample.items` table** + +| item_id | name | price | +|---------|--------|-------| +| 1 | Apple | 1000 | +| 2 | Orange | 2000 | +| 3 | Grape | 2500 | +| 4 | Mango | 5000 | +| 5 | Melon | 3000 | + +## Execute transactions and retrieve data in the sample application + +The following sections describe how to execute transactions and retrieve data in the sample e-commerce application. + +### Get customer information + +Start with getting information about the customer whose ID is `1` by running the following command: + +```console +./gradlew run --args="GetCustomerInfo 1" +``` + +You should see the following output: + +```console +... +{"id": 1, "name": "Yamada Taro", "credit_limit": 10000, "credit_total": 0} +... +``` + +### Place an order + +Then, have customer ID `1` place an order for three apples and two oranges by running the following command: + +:::note + +The order format in this command is `./gradlew run --args="PlaceOrder :,:,..."`. + +::: + +```console +./gradlew run --args="PlaceOrder 1 1:3,2:2" +``` + +You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: + +```console +... +{"order_id": "dea4964a-ff50-4ecf-9201-027981a1566e"} +... +``` + +### Check order details + +Check details about the order by running the following command, replacing `` with the UUID for the `order_id` that was shown after running the previous command: + +```console +./gradlew run --args="GetOrder " +``` + +You should see a similar output as below, with different UUIDs for `order_id` and `timestamp`: + +```console +... +{"order": {"order_id": "dea4964a-ff50-4ecf-9201-027981a1566e","timestamp": 1650948340914,"customer_id": 1,"customer_name": "Yamada Taro","statement": [{"item_id": 1,"item_name": "Apple","price": 1000,"count": 3,"total": 3000},{"item_id": 2,"item_name": "Orange","price": 2000,"count": 2,"total": 4000}],"total": 7000}} +... +``` + +### Place another order + +Place an order for one melon that uses the remaining amount in `credit_total` for customer ID `1` by running the following command: + +```console +./gradlew run --args="PlaceOrder 1 5:1" +``` + +You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: + +```console +... +{"order_id": "bcc34150-91fa-4bea-83db-d2dbe6f0f30d"} +... +``` + +### Check order history + +Get the history of all orders for customer ID `1` by running the following command: + +```console +./gradlew run --args="GetOrders 1" +``` + +You should see a similar output as below, with different UUIDs for `order_id` and `timestamp`, which shows the history of all orders for customer ID `1` in descending order by timestamp: + +```console +... +{"order": [{"order_id": "dea4964a-ff50-4ecf-9201-027981a1566e","timestamp": 1650948340914,"customer_id": 1,"customer_name": "Yamada Taro","statement": [{"item_id": 1,"item_name": "Apple","price": 1000,"count": 3,"total": 3000},{"item_id": 2,"item_name": "Orange","price": 2000,"count": 2,"total": 4000}],"total": 7000},{"order_id": "bcc34150-91fa-4bea-83db-d2dbe6f0f30d","timestamp": 1650948412766,"customer_id": 1,"customer_name": "Yamada Taro","statement": [{"item_id": 5,"item_name": "Melon","price": 3000,"count": 1,"total": 3000}],"total": 3000}]} +... +``` + +### Check credit total + +Get the credit total for customer ID `1` by running the following command: + +```console +./gradlew run --args="GetCustomerInfo 1" +``` + +You should see the following output, which shows that customer ID `1` has reached their `credit_limit` in `credit_total` and cannot place anymore orders: + +```console +... +{"id": 1, "name": "Yamada Taro", "credit_limit": 10000, "credit_total": 10000} +... +``` + +Try to place an order for one grape and one mango by running the following command: + +```console +./gradlew run --args="PlaceOrder 1 3:1,4:1" +``` + +You should see the following output, which shows that the order failed because the `credit_total` amount would exceed the `credit_limit` amount. + +```console +... +java.lang.RuntimeException: Credit limit exceeded + at sample.Sample.placeOrder(Sample.java:205) + at sample.command.PlaceOrderCommand.call(PlaceOrderCommand.java:33) + at sample.command.PlaceOrderCommand.call(PlaceOrderCommand.java:8) + at picocli.CommandLine.executeUserObject(CommandLine.java:1783) + at picocli.CommandLine.access$900(CommandLine.java:145) + at picocli.CommandLine$RunLast.handle(CommandLine.java:2141) + at picocli.CommandLine$RunLast.handle(CommandLine.java:2108) + at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:1975) + at picocli.CommandLine.execute(CommandLine.java:1904) + at sample.command.SampleCommand.main(SampleCommand.java:35) +... +``` + +### Make a payment + +To continue making orders, customer ID `1` must make a payment to reduce the `credit_total` amount. + +Make a payment by running the following command: + +```console +./gradlew run --args="Repayment 1 8000" +``` + +Then, check the `credit_total` amount for customer ID `1` by running the following command: + +```console +./gradlew run --args="GetCustomerInfo 1" +``` + +You should see the following output, which shows that a payment was applied to customer ID `1`, reducing the `credit_total` amount: + +```console +... +{"id": 1, "name": "Yamada Taro", "credit_limit": 10000, "credit_total": 2000} +... +``` + +Now that customer ID `1` has made a payment, place an order for one grape and one melon by running the following command: + +```console +./gradlew run --args="PlaceOrder 1 3:1,4:1" +``` + +You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: + +```console +... +{"order_id": "8911cab3-1c2b-4322-9386-adb1c024e078"} +... +``` + +## See also + +For other ScalarDB Cluster tutorials, see the following: + +- [Getting Started with ScalarDB Cluster GraphQL](getting-started-with-scalardb-cluster-graphql.mdx) +- [Getting Started with ScalarDB Cluster SQL via JDBC](getting-started-with-scalardb-cluster-sql-jdbc.mdx) +- [Getting Started with ScalarDB Cluster SQL via Spring Data JDBC for ScalarDB](getting-started-with-scalardb-cluster-sql-spring-data-jdbc.mdx) +- [Getting Started with Using Go for ScalarDB Cluster](getting-started-with-using-go-for-scalardb-cluster.mdx) +- [Getting Started with Using Python for ScalarDB Cluster](getting-started-with-using-python-for-scalardb-cluster.mdx) + +For details about developing applications that use ScalarDB Cluster with the Java API, refer to the following: + +- [Developer Guide for ScalarDB Cluster with the Java API](developer-guide-for-scalardb-cluster-with-java-api.mdx) + +For details about the ScalarDB Cluster gRPC API, refer to the following: + +- [ScalarDB Cluster gRPC API Guide](scalardb-cluster-grpc-api-guide.mdx) +- [ScalarDB Cluster SQL gRPC API Guide](scalardb-cluster-sql-grpc-api-guide.mdx) diff --git a/versioned_docs/version-3.X/scalardb-cluster/getting-started-with-using-go-for-scalardb-cluster.mdx b/versioned_docs/version-3.X/scalardb-cluster/getting-started-with-using-go-for-scalardb-cluster.mdx new file mode 100644 index 00000000..780a30c0 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster/getting-started-with-using-go-for-scalardb-cluster.mdx @@ -0,0 +1,444 @@ +--- +tags: + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Getting Started with Using Go for ScalarDB Cluster + +import WarningLicenseKeyContact from '/src/components/en-us/_warning-license-key-contact.mdx'; + +This document explains how to write gRPC client code for ScalarDB Cluster by using Go. + +## Prerequisites + +- [Go](https://go.dev/dl/) (any one of the three latest major releases) +- ScalarDB Cluster running on a Kubernetes cluster + - We assume that you have a ScalarDB Cluster running on a Kubernetes cluster that you deployed by following the instructions in [Set Up ScalarDB Cluster on Kubernetes by Using a Helm Chart](setup-scalardb-cluster-on-kubernetes-by-using-helm-chart.mdx). + + + +## Sample application + +This tutorial illustrates the process of creating an electronic money application, where money can be transferred between accounts. + +## Step 1. Create `schema.json` + +The following is a simple example schema. + +Create `schema.json`, and add the following to the file: + +```json +{ + "emoney.account": { + "transaction": true, + "partition-key": [ + "id" + ], + "clustering-key": [], + "columns": { + "id": "TEXT", + "balance": "INT" + } + } +} +``` + +## Step 2. Create `database.properties` + +You need to create `database.properties` for the Schema Loader for ScalarDB Cluster. +But first, you need to get the `EXTERNAL-IP` address of the service resource of the `LoadBalancer` service (`scalardb-cluster-envoy`). + +To see the `EXTERNAL-IP` address, run the following command: + +```console +kubectl get svc scalardb-cluster-envoy +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +scalardb-cluster-envoy LoadBalancer 10.105.121.51 localhost 60053:30641/TCP 16h +``` + +In this case, the `EXTERNAL-IP` address is `localhost`. + +Then, create `database.properties`, and add the following to the file: + +```properties +scalar.db.transaction_manager=cluster +scalar.db.contact_points=indirect:localhost +``` + +To connect to ScalarDB Cluster, you need to specify `cluster` for the `scalar.db.transaction_manager` property. +In addition, you will use the `indirect` client mode and connect to the service resource of Envoy in this tutorial. +For details about the client modes, see [Developer Guide for ScalarDB Cluster with the Java API](developer-guide-for-scalardb-cluster-with-java-api.mdx). + +## Step 3. Load a schema + +To load a schema via ScalarDB Cluster, you need to use the dedicated Schema Loader for ScalarDB Cluster (Schema Loader for Cluster). Using the Schema Loader for Cluster is basically the same as using the [Schema Loader for ScalarDB](../schema-loader.mdx) except the name of the JAR file is different. You can download the Schema Loader for Cluster from [ScalarDB Releases](https://github.com/scalar-labs/scalardb/releases/tag/v3.16.0). After downloading the JAR file, you can run the Schema Loader for Cluster with the following command: + +```console +java -jar scalardb-cluster-schema-loader-3.16.0-all.jar --config database.properties -f schema.json --coordinator +``` + +## Step 4. Set up a Go environment + +Follow the [Prerequisites](https://grpc.io/docs/languages/go/quickstart/#prerequisites) section in the gRPC quick-start document to install the following components: + +- Go +- Protocol buffer compiler, `protoc`, version 3.15 or later +- Go plugins for the protocol compiler + +## Step 5. Generate the stub code for ScalarDB Cluster gRPC + +To communicate with the gRPC server for ScalarDB Cluster, you will need to generate the stub code from the proto file. + +First, in a new working directory, create a directory named `scalardb-cluster`, which you will use to generate the gRPC code from, by running the following command: + +```console +mkdir scalardb-cluster +``` + +Then, download the `scalardb-cluster.proto` file and save it in the directory that you created. For ScalarDB Cluster users who have a commercial license, please [contact support](https://www.scalar-labs.com/support) if you need the `scalardb-cluster.proto` file. + +Generate the gRPC code by running the following command: + +```console +protoc --go_out=. --go_opt=paths=source_relative \ + --go_opt=Mscalardb-cluster/scalardb-cluster.proto=example.com/scalardb-cluster \ + --go-grpc_out=. --go-grpc_opt=paths=source_relative \ + --go-grpc_opt=Mscalardb-cluster/scalardb-cluster.proto=example.com/scalardb-cluster \ + scalardb-cluster/scalardb-cluster.proto +``` + +After running the command, you should see two files in the `scalardb-cluster` subdirectory: `scalardb-cluster.pb.go` and `scalardb-cluster_grpc.pb.go`. + +## Step 6. Write a sample application + +The following is the program that uses the gRPC code. Save it as `main.go` in the working directory. This program does the same thing as the `ElectronicMoney.java` program in [Getting Started with ScalarDB](https://scalardb.scalar-labs.com/docs/latest/getting-started-with-scalardb/). Note that you have to update the value of `SERVER_ADDRESS` based on the `EXTERNAL-IP` value of the ScalarDB Cluster `LoadBalancer` service in your environment. + +```go +package main + +import ( + "context" + "errors" + "flag" + "fmt" + "log" + "os" + "time" + + pb "emoney/scalardb-cluster" + + "google.golang.org/grpc" + "google.golang.org/grpc/credentials/insecure" +) + +const ( + SERVER_ADDRESS = "localhost:60053" + NAMESPACE = "emoney" + TABLENAME = "account" + ID = "id" + BALANCE = "balance" +) + +var requestHeader = pb.RequestHeader{HopLimit: 10} + +type TxFn func(ctx context.Context, client pb.DistributedTransactionClient, transactionId string) error + +func withTransaction(fn TxFn) error { + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + + // Set up a connection to the server. + conn, err := grpc.Dial(SERVER_ADDRESS, grpc.WithTransportCredentials(insecure.NewCredentials())) + if err != nil { + return err + } + defer conn.Close() + + client := pb.NewDistributedTransactionClient(conn) + + // Begin a transaction + beginResponse, err := client.Begin(ctx, &pb.BeginRequest{RequestHeader: &requestHeader}) + if err != nil { + return err + } + transactionId := beginResponse.TransactionId + + // Execute the function + err = fn(ctx, client, transactionId) + if err != nil { + // Rollback the transaction if there is an error + client.Rollback(ctx, &pb.RollbackRequest{TransactionId: transactionId}) + return err + } + + // Commit the transaction + _, err = client.Commit(ctx, &pb.CommitRequest{RequestHeader: &requestHeader, TransactionId: transactionId}) + return err +} + +func charge(ctx context.Context, client pb.DistributedTransactionClient, transactionId string, id string, amount int) error { + partitionKey := pb.Key{Columns: []*pb.Column{{Name: ID, Value: &pb.Column_TextValue_{TextValue: &pb.Column_TextValue{Value: &id}}}}} + + // Retrieve the current balance for id + get := pb.Get{ + NamespaceName: NAMESPACE, TableName: TABLENAME, + PartitionKey: &partitionKey, ClusteringKey: nil, + GetType: pb.Get_GET_TYPE_GET, + } + getResponse, err := client.Get(ctx, &pb.GetRequest{RequestHeader: &requestHeader, TransactionId: transactionId, Get: &get}) + if err != nil { + return err + } + + // Calculate the balance + balance := int32(amount) + if result := getResponse.GetResult(); result != nil { + for _, column := range result.GetColumns() { + if column.Name == BALANCE { + balance += column.GetIntValue().GetValue() + break + } + } + } + + // Update the balance + put := pb.Put{ + NamespaceName: NAMESPACE, TableName: TABLENAME, + PartitionKey: &partitionKey, ClusteringKey: nil, + Columns: []*pb.Column{ + {Name: BALANCE, Value: &pb.Column_IntValue_{IntValue: &pb.Column_IntValue{Value: &balance}}}, + }, + } + _, err = client.Put(ctx, &pb.PutRequest{RequestHeader: &requestHeader, TransactionId: transactionId, Puts: []*pb.Put{&put}}) + return err +} + +func pay(ctx context.Context, client pb.DistributedTransactionClient, transactionId string, fromId string, toId string, amount int) error { + fromPartitionKey := pb.Key{Columns: []*pb.Column{{Name: ID, Value: &pb.Column_TextValue_{TextValue: &pb.Column_TextValue{Value: &fromId}}}}} + toPartitionKey := pb.Key{Columns: []*pb.Column{{Name: ID, Value: &pb.Column_TextValue_{TextValue: &pb.Column_TextValue{Value: &toId}}}}} + + // Retrieve the current balances for ids + fromGet := pb.Get{ + NamespaceName: NAMESPACE, TableName: TABLENAME, + PartitionKey: &fromPartitionKey, ClusteringKey: nil, + GetType: pb.Get_GET_TYPE_GET, + } + fromGetResponse, err := client.Get(ctx, &pb.GetRequest{RequestHeader: &requestHeader, TransactionId: transactionId, Get: &fromGet}) + if err != nil { + return err + } + toGet := pb.Get{ + NamespaceName: NAMESPACE, TableName: TABLENAME, + PartitionKey: &toPartitionKey, ClusteringKey: nil, + GetType: pb.Get_GET_TYPE_GET, + } + toGetResponse, err := client.Get(ctx, &pb.GetRequest{RequestHeader: &requestHeader, TransactionId: transactionId, Get: &toGet}) + if err != nil { + return err + } + + // Calculate the balances (it assumes that both accounts exist) + var ( + fromBalance int32 + toBalance int32 + ) + for _, column := range fromGetResponse.GetResult().GetColumns() { + if column.Name == BALANCE { + fromBalance = column.GetIntValue().GetValue() + break + } + } + for _, column := range toGetResponse.GetResult().GetColumns() { + if column.Name == BALANCE { + toBalance = column.GetIntValue().GetValue() + break + } + } + newFromBalance := fromBalance - int32(amount) + newToBalance := toBalance + int32(amount) + + if newFromBalance < 0 { + return errors.New(fromId + " doesn't have enough balance.") + } + + // Update the balances + fromPut := pb.Put{ + NamespaceName: NAMESPACE, TableName: TABLENAME, + PartitionKey: &fromPartitionKey, ClusteringKey: nil, + Columns: []*pb.Column{ + {Name: BALANCE, Value: &pb.Column_IntValue_{IntValue: &pb.Column_IntValue{Value: &newFromBalance}}}, + }, + } + toPut := pb.Put{ + NamespaceName: NAMESPACE, TableName: TABLENAME, + PartitionKey: &toPartitionKey, ClusteringKey: nil, + Columns: []*pb.Column{ + {Name: BALANCE, Value: &pb.Column_IntValue_{IntValue: &pb.Column_IntValue{Value: &newToBalance}}}, + }, + } + _, err = client.Put(ctx, &pb.PutRequest{RequestHeader: &requestHeader, TransactionId: transactionId, Puts: []*pb.Put{&fromPut, &toPut}}) + return err +} + +func getBalance(ctx context.Context, client pb.DistributedTransactionClient, transactionId string, id string) (int, error) { + // Retrieve the current balance for id + get := pb.Get{ + NamespaceName: NAMESPACE, TableName: TABLENAME, + PartitionKey: &pb.Key{Columns: []*pb.Column{{Name: ID, Value: &pb.Column_TextValue_{TextValue: &pb.Column_TextValue{Value: &id}}}}}, + ClusteringKey: nil, + GetType: pb.Get_GET_TYPE_GET, + } + getResponse, err := client.Get(ctx, &pb.GetRequest{RequestHeader: &requestHeader, TransactionId: transactionId, Get: &get}) + if err != nil { + return 0, err + } + if getResponse.GetResult() == nil || len(getResponse.GetResult().GetColumns()) == 0 { + return 0, errors.New("Account " + id + " doesn't exist.") + } + + var balance int + for _, column := range getResponse.GetResult().GetColumns() { + if column.Name == BALANCE { + balance = int(column.GetIntValue().GetValue()) + break + } + } + return balance, nil +} + +func main() { + var ( + action = flag.String("action", "", "Action to perform: charge / pay / getBalance") + fromId = flag.String("from", "", "From account (needed for pay)") + toId = flag.String("to", "", "To account (needed for charge and pay)") + id = flag.String("id", "", "Account id (needed for getBalance)") + ) + var amount int + flag.IntVar(&amount, "amount", 0, "Amount to transfer (needed for charge and pay)") + flag.Parse() + + if *action == "charge" { + if *toId == "" || amount < 0 { + printUsageAndExit() + } + err := withTransaction(func(ctx context.Context, client pb.DistributedTransactionClient, txId string) error { + return charge(ctx, client, txId, *toId, amount) + }) + if err != nil { + log.Fatalf("error: %v", err) + } + } else if *action == "pay" { + if *toId == "" || *fromId == "" || amount < 0 { + printUsageAndExit() + } + err := withTransaction(func(ctx context.Context, client pb.DistributedTransactionClient, txId string) error { + return pay(ctx, client, txId, *fromId, *toId, amount) + }) + if err != nil { + log.Fatalf("error: %v", err) + } + } else if *action == "getBalance" { + if *id == "" { + printUsageAndExit() + } + var balance int + err := withTransaction(func(ctx context.Context, client pb.DistributedTransactionClient, txId string) error { + var err error + balance, err = getBalance(ctx, client, txId, *id) + return err + }) + if err != nil { + log.Fatalf("error: %v", err) + } + fmt.Println(balance) + } else { + fmt.Fprintln(os.Stderr, "Unknown action "+*action) + printUsageAndExit() + } +} + +func printUsageAndExit() { + flag.Usage() + os.Exit(1) +} +``` + +After creating the `main.go` file, you need to create the `go.mod` file by running the following commands: + +```console +go mod init emoney +go mod tidy +``` + +Now, the directory structure should be as follows: + +```text +. +├── go.mod +├── go.sum +├── main.go +└── scalardb-cluster + ├── scalardb-cluster.pb.go + ├── scalardb-cluster.proto + └── scalardb-cluster_grpc.pb.go +``` + +You can then run the program as follows: + +- Charge `1000` to `user1`: + + ```console + go run main.go -action charge -amount 1000 -to user1 + ``` + +- Charge `0` to `merchant1` (Just create an account for `merchant1`): + + ```console + go run main.go -action charge -amount 0 -to merchant1 + ``` + +- Pay `100` from `user1` to `merchant1`: + + ```console + go run main.go -action pay -amount 100 -from user1 -to merchant1 + ``` + +- Get the balance of `user1`: + + ```console + go run main.go -action getBalance -id user1 + ``` + +- Get the balance of `merchant1`: + + ```console + go run main.go -action getBalance -id merchant1 + ``` + +Note that you can also use `go build` to get the binary and then run it: + +```console +go build +./emoney -action getBalance -id user1 +``` + +## See also + +For other ScalarDB Cluster tutorials, see the following: + +- [Getting Started with ScalarDB Cluster](getting-started-with-scalardb-cluster.mdx) +- [Getting Started with ScalarDB Cluster GraphQL](getting-started-with-scalardb-cluster-graphql.mdx) +- [Getting Started with ScalarDB Cluster SQL via JDBC](getting-started-with-scalardb-cluster-sql-jdbc.mdx) +- [Getting Started with ScalarDB Cluster SQL via Spring Data JDBC for ScalarDB](getting-started-with-scalardb-cluster-sql-spring-data-jdbc.mdx) +- [Getting Started with Using Python for ScalarDB Cluster](getting-started-with-using-python-for-scalardb-cluster.mdx) + +For details about developing applications that use ScalarDB Cluster with the Java API, refer to the following: + +- [Developer Guide for ScalarDB Cluster with the Java API](developer-guide-for-scalardb-cluster-with-java-api.mdx) + +For details about the ScalarDB Cluster gRPC API, refer to the following: + +- [ScalarDB Cluster gRPC API Guide](scalardb-cluster-grpc-api-guide.mdx) +- [ScalarDB Cluster SQL gRPC API Guide](scalardb-cluster-sql-grpc-api-guide.mdx) diff --git a/versioned_docs/version-3.X/scalardb-cluster/getting-started-with-using-python-for-scalardb-cluster.mdx b/versioned_docs/version-3.X/scalardb-cluster/getting-started-with-using-python-for-scalardb-cluster.mdx new file mode 100644 index 00000000..868b3c9d --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster/getting-started-with-using-python-for-scalardb-cluster.mdx @@ -0,0 +1,487 @@ +--- +tags: + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Getting Started with Using Python for ScalarDB Cluster + +import WarningLicenseKeyContact from '/src/components/en-us/_warning-license-key-contact.mdx'; + +This document explains how to write gRPC client code for ScalarDB Cluster by using Python. + +## Prerequisites + +- [Python](https://www.python.org/downloads) 3.7 or later +- ScalarDB Cluster running on a Kubernetes cluster + - We assume that you have a ScalarDB Cluster running on a Kubernetes cluster that you deployed by following the instructions in [Set Up ScalarDB Cluster on Kubernetes by Using a Helm Chart](setup-scalardb-cluster-on-kubernetes-by-using-helm-chart.mdx). + + + +## Sample application + +This tutorial illustrates the process of creating an electronic money application, where money can be transferred between accounts. + +## Step 1. Create `schema.json` + +The following is a simple example schema. + +Create `schema.json`, and add the following to the file: + +```json +{ + "emoney.account": { + "transaction": true, + "partition-key": [ + "id" + ], + "clustering-key": [], + "columns": { + "id": "TEXT", + "balance": "INT" + } + } +} +``` + +## Step 2. Create `database.properties` + +You need to create `database.properties` for the Schema Loader for ScalarDB Cluster. +But first, you need to get the `EXTERNAL-IP` address of the service resource of the `LoadBalancer` service (`scalardb-cluster-envoy`). + +To see the `EXTERNAL-IP` address, run the following command: + +```console +kubectl get svc scalardb-cluster-envoy +NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE +scalardb-cluster-envoy LoadBalancer 10.105.121.51 localhost 60053:30641/TCP 16h +``` + +In this case, the `EXTERNAL-IP` address is `localhost`. + +Then, create `database.properties`, and add the following to the file: + +```properties +scalar.db.transaction_manager=cluster +scalar.db.contact_points=indirect:localhost +``` + +To connect to ScalarDB Cluster, you need to specify `cluster` for the `scalar.db.transaction_manager` property. +In addition, you will use the `indirect` client mode and connect to the service resource of Envoy in this tutorial. +For details about the client modes, see [Developer Guide for ScalarDB Cluster with the Java API](developer-guide-for-scalardb-cluster-with-java-api.mdx). + +## Step 3. Load a schema + +To load a schema via ScalarDB Cluster, you need to use the dedicated Schema Loader for ScalarDB Cluster (Schema Loader for Cluster). Using the Schema Loader for Cluster is basically the same as using the [Schema Loader for ScalarDB](../schema-loader.mdx) except the name of the JAR file is different. You can download the Schema Loader for Cluster from [ScalarDB Releases](https://github.com/scalar-labs/scalardb/releases/tag/v3.16.0). After downloading the JAR file, you can run the Schema Loader for Cluster with the following command: + +```console +java -jar scalardb-cluster-schema-loader-3.16.0-all.jar --config database.properties -f schema.json --coordinator +``` + +## Step 4. Set up a Python environment + +You can choose any way you like to manage your Python environment. For the purpose of this guide, we assume that your Python application is running in an environment by using `venv`. + +Create a working directory anywhere, and go there. Then, run the following command to activate `venv` by running the following command: + +```console +python3 -m venv venv +source venv/bin/activate +``` + +Let's install the gRPC packages with the `pip` command: + +```console +pip install grpcio grpcio-tools +``` + +## Step 5. Generate the stub code for ScalarDB Cluster gRPC + +To communicate with the gRPC server for ScalarDB Cluster, you will need to generate the stub code from the proto file. + +First, download the `scalardb-cluster.proto` file, then save it in the working directory. For ScalarDB Cluster users who have a commercial license, please [contact support](https://www.scalar-labs.com/support) if you need the `scalardb-cluster.proto` file. + +You can generate the stub code by running the following command: + +```console +python -m grpc_tools.protoc -I . --python_out=. --pyi_out=. --grpc_python_out=. scalardb-cluster.proto +``` + +The following files will be generated: + +- `scalardb_cluster_pb2.py` +- `scalardb_cluster_pb2.pyi` +- `scalardb_cluster_pb2_grpc.py` + +## Step 6. Write a sample application + +The following is the sample Python application (`electronic_money.py`) that uses the stub code. This program does the same thing as the `ElectronicMoney.java` program in [Getting Started with ScalarDB](https://scalardb.scalar-labs.com/docs/latest/getting-started-with-scalardb/). Note that you have to update the value of `SERVER_ADDRESS` based on the `EXTERNAL-IP` value of the ScalarDB Cluster `LoadBalancer` service in your environment. + +```python +import argparse +from typing import Optional + +import grpc + +import scalardb_cluster_pb2_grpc +from scalardb_cluster_pb2 import ( + BeginRequest, + BeginResponse, + Column, + CommitRequest, + Get, + GetRequest, + GetResponse, + Key, + Put, + PutRequest, + RequestHeader, + RollbackRequest, +) + +SERVER_ADDRESS = "localhost:60053" +NAMESPACE = "emoney" +TABLENAME = "account" +ID = "id" +BALANCE = "balance" + +request_header = RequestHeader(hop_limit=10) + + +def charge(id: str, amount: int) -> None: + with grpc.insecure_channel(SERVER_ADDRESS) as channel: + stub = scalardb_cluster_pb2_grpc.DistributedTransactionStub(channel) + + begin_response: BeginResponse = stub.Begin( + BeginRequest(request_header=request_header) + ) + + transaction_id = begin_response.transaction_id + + try: + pkey = Key( + columns=[ + Column( + name=ID, + text_value=Column.TextValue(value=id), + ) + ] + ) + + # Retrieve the current balance for id + get = Get( + namespace_name=NAMESPACE, + table_name=TABLENAME, + get_type=Get.GetType.GET_TYPE_GET, + partition_key=pkey, + ) + get_response: GetResponse = stub.Get( + GetRequest( + request_header=request_header, + transaction_id=transaction_id, + get=get, + ) + ) + + # Calculate the balance + balance = amount + if get_response.result.columns: + balance_column = next( + c for c in get_response.result.columns if c.name == BALANCE + ) + current = balance_column.int_value.value + balance += current + + # Update the balance + put = Put( + namespace_name=NAMESPACE, + table_name=TABLENAME, + partition_key=pkey, + columns=[ + Column(name=BALANCE, int_value=Column.IntValue(value=balance)) + ], + ) + stub.Put( + PutRequest( + request_header=request_header, + transaction_id=transaction_id, + puts=[put], + ) + ) + + # Commit the transaction + stub.Commit( + CommitRequest( + request_header=request_header, + transaction_id=transaction_id, + ) + ) + except Exception as e: + # Rollback the transaction + stub.Rollback( + RollbackRequest( + request_header=request_header, + transaction_id=transaction_id, + ) + ) + raise e + + +def pay(from_id: str, to_id: str, amount: int) -> None: + with grpc.insecure_channel(SERVER_ADDRESS) as channel: + stub = scalardb_cluster_pb2_grpc.DistributedTransactionStub(channel) + + begin_response: BeginResponse = stub.Begin( + BeginRequest(request_header=request_header) + ) + + transaction_id = begin_response.transaction_id + + try: + from_pkey = Key( + columns=[ + Column( + name=ID, + text_value=Column.TextValue(value=from_id), + ) + ] + ) + to_pkey = Key( + columns=[ + Column( + name=ID, + text_value=Column.TextValue(value=to_id), + ) + ] + ) + + # Retrieve the current balances for ids + from_get = Get( + namespace_name=NAMESPACE, + table_name=TABLENAME, + get_type=Get.GetType.GET_TYPE_GET, + partition_key=from_pkey, + ) + from_get_response: GetResponse = stub.Get( + GetRequest( + request_header=request_header, + transaction_id=transaction_id, + get=from_get, + ) + ) + to_get = Get( + namespace_name=NAMESPACE, + table_name=TABLENAME, + get_type=Get.GetType.GET_TYPE_GET, + partition_key=to_pkey, + ) + to_get_response: GetResponse = stub.Get( + GetRequest( + request_header=request_header, + transaction_id=transaction_id, + get=to_get, + ) + ) + + # Calculate the balances (it assumes that both accounts exist) + new_from_balance = ( + next( + c for c in from_get_response.result.columns if c.name == BALANCE + ).int_value.value + - amount + ) + new_to_balance = ( + next( + c for c in to_get_response.result.columns if c.name == BALANCE + ).int_value.value + + amount + ) + + if new_from_balance < 0: + raise RuntimeError(from_id + " doesn't have enough balance.") + + # Update the balances + from_put = Put( + namespace_name=NAMESPACE, + table_name=TABLENAME, + partition_key=from_pkey, + columns=[ + Column( + name=BALANCE, int_value=Column.IntValue(value=new_from_balance) + ) + ], + ) + to_put = Put( + namespace_name=NAMESPACE, + table_name=TABLENAME, + partition_key=to_pkey, + columns=[ + Column( + name=BALANCE, int_value=Column.IntValue(value=new_to_balance) + ) + ], + ) + stub.Put( + PutRequest( + request_header=request_header, + transaction_id=transaction_id, + puts=[from_put, to_put], + ) + ) + + # Commit the transaction (records are automatically recovered in case of failure) + stub.Commit( + CommitRequest( + request_header=request_header, + transaction_id=transaction_id, + ) + ) + except Exception as e: + # Rollback the transaction + stub.Rollback( + RollbackRequest( + request_header=request_header, + transaction_id=transaction_id, + ) + ) + raise e + + +def get_balance(id: str) -> Optional[int]: + with grpc.insecure_channel(SERVER_ADDRESS) as channel: + stub = scalardb_cluster_pb2_grpc.DistributedTransactionStub(channel) + + begin_response: BeginResponse = stub.Begin( + BeginRequest(request_header=request_header) + ) + + transaction_id = begin_response.transaction_id + + try: + # Retrieve the current balance for id + get = Get( + namespace_name=NAMESPACE, + table_name=TABLENAME, + get_type=Get.GetType.GET_TYPE_GET, + partition_key=Key( + columns=[ + Column( + name=ID, + text_value=Column.TextValue(value=id), + ) + ] + ), + ) + get_response: GetResponse = stub.Get( + GetRequest( + request_header=request_header, + transaction_id=transaction_id, + get=get, + ) + ) + + balance = None + if get_response.result.columns: + balance_column = next( + c for c in get_response.result.columns if c.name == BALANCE + ) + balance = balance_column.int_value.value + + # Commit the transaction + stub.Commit( + CommitRequest( + request_header=request_header, + transaction_id=transaction_id, + ) + ) + + return balance + + except Exception as e: + # Rollback the transaction + stub.Rollback( + RollbackRequest( + request_header=request_header, + transaction_id=transaction_id, + ) + ) + raise e + + +if __name__ == "__main__": + parser = argparse.ArgumentParser() + subparsers = parser.add_subparsers(required=True) + + parser_charge = subparsers.add_parser("charge") + parser_charge.add_argument("-amount", type=int, required=True) + parser_charge.add_argument("-to", type=str, required=True, dest="to_id") + parser_charge.set_defaults(func=lambda args: charge(args.to_id, args.amount)) + + parser_pay = subparsers.add_parser("pay") + parser_pay.add_argument("-amount", type=int, required=True) + parser_pay.add_argument("-from", type=str, required=True, dest="from_id") + parser_pay.add_argument("-to", type=str, required=True, dest="to_id") + parser_pay.set_defaults( + func=lambda args: pay(args.from_id, args.to_id, args.amount) + ) + + parser_get_balance = subparsers.add_parser("get-balance") + parser_get_balance.add_argument("-id", type=str, required=True) + parser_get_balance.set_defaults(func=lambda args: print(get_balance(args.id))) + + args = parser.parse_args() + args.func(args) + +``` + +You can then run the program as follows: + +- Charge `1000` to `user1`: + + ```console + python electronic_money.py charge -amount 1000 -to user1 + ``` + +- Charge `0` to `merchant1` (Just create an account for `merchant1`): + + ```console + python electronic_money.py charge -amount 0 -to merchant1 + ``` + +- Pay `100` from `user1` to `merchant1`: + + ```console + python electronic_money.py pay -amount 100 -from user1 -to merchant1 + ``` + +- Get the balance of `user1`: + + ```console + python electronic_money.py get-balance -id user1 + ``` + +- Get the balance of `merchant1`: + + ```console + python electronic_money.py get-balance -id merchant1 + ``` + +## See also + +For other ScalarDB Cluster tutorials, see the following: + +- [Getting Started with ScalarDB Cluster](getting-started-with-scalardb-cluster.mdx) +- [Getting Started with ScalarDB Cluster GraphQL](getting-started-with-scalardb-cluster-graphql.mdx) +- [Getting Started with ScalarDB Cluster SQL via JDBC](getting-started-with-scalardb-cluster-sql-jdbc.mdx) +- [Getting Started with ScalarDB Cluster SQL via Spring Data JDBC for ScalarDB](getting-started-with-scalardb-cluster-sql-spring-data-jdbc.mdx) +- [Getting Started with Using Go for ScalarDB Cluster](getting-started-with-using-go-for-scalardb-cluster.mdx) + +For details about developing applications that use ScalarDB Cluster with the Java API, refer to the following: + +- [Developer Guide for ScalarDB Cluster with the Java API](developer-guide-for-scalardb-cluster-with-java-api.mdx) + +For details about the ScalarDB Cluster gRPC API, refer to the following: + +- [ScalarDB Cluster gRPC API Guide](scalardb-cluster-grpc-api-guide.mdx) +- [ScalarDB Cluster SQL gRPC API Guide](scalardb-cluster-sql-grpc-api-guide.mdx) diff --git a/versioned_docs/version-3.X/scalardb-cluster/getting-started-with-vector-search.mdx b/versioned_docs/version-3.X/scalardb-cluster/getting-started-with-vector-search.mdx new file mode 100644 index 00000000..b30b51e1 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster/getting-started-with-vector-search.mdx @@ -0,0 +1,463 @@ +--- +tags: + - Enterprise Premium + - Private Preview +displayed_sidebar: docsEnglish +--- + +# Getting Started with ScalarDB Cluster for Vector Search + +import WarningLicenseKeyContact from '/src/components/en-us/_warning-license-key-contact.mdx'; +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +ScalarDB Cluster provides a vector store abstraction to help applications interact with vector stores (embedding stores) in a unified way. This getting-started tutorial explains how to run vector search in ScalarDB Cluster. + +## What is the vector store abstraction? + +ScalarDB Cluster provides an abstraction for various vector stores, similar to how it abstracts different types of databases, including relational databases, NoSQL databases, and NewSQL databases. With this vector store abstraction, you can develop applications that interact with vector stores in a unified manner, making your applications independent of specific vector store implementations and ensuring their portability. Additionally, since the integration of vector stores is built into ScalarDB Cluster, your applications can take advantage of its scalability. + +The current implementation of the vector store abstraction leverages [LangChain4j](https://docs.langchain4j.dev/) and supports the following vector stores and embedding models. + +Vector stores: +- In-memory +- OpenSearch +- Azure Cosmos DB NoSQL +- Azure AI Search +- pgvector + +Embedding models: +- In-process +- Amazon Bedrock +- Azure OpenAI +- Google Vertex AI +- OpenAI + +## Why use the vector store abstraction? + +In the era of generative AI, one of the challenges organizations face when deploying large language models (LLMs) is enabling these models to understand their enterprise data. Retrieval-augmented generation (RAG) is a key technique used to enhance LLMs with specific enterprise knowledge. For example, to ensure that chatbots powered by LLMs provide accurate and relevant responses, companies use RAG to integrate domain-specific information from user manuals and support documents. + +RAG relies on vector stores, which are typically created by extracting data from databases, converting that data into vectors, and then loading those vectors. By using vector store and database abstraction in ScalarDB Cluster, you can implement the entire process seamlessly. This approach significantly simplifies the workflow and code, eliminating the need to write complex applications that depend on specific vector stores and databases. + +## Tutorial + +This tutorial explains how to run vector search in ScalarDB Cluster. + +### Prerequisites + +- OpenJDK LTS version (8, 11, 17, or 21) from [Eclipse Temurin](https://adoptium.net/temurin/releases/) +- [Docker](https://www.docker.com/get-started/) 20.10 or later with [Docker Compose](https://docs.docker.com/compose/install/) V2 or later + +:::note + +This tutorial has been tested with OpenJDK from Eclipse Temurin. ScalarDB itself, however, has been tested with JDK distributions from various vendors. For details about the requirements for ScalarDB, including compatible JDK distributions, please see [Requirements](../requirements.mdx). + +::: + + + +### 1. Create the ScalarDB Cluster configuration file + +Create the following configuration file as `scalardb-cluster-node.properties`, replacing `` and `` with your ScalarDB license key and license check certificate values. For more information about the license key and certificate, see [How to Configure a Product License Key](../scalar-licensing/index.mdx). + +```yaml +scalar.db.transaction.enabled=false + +# Enable the standalone mode +scalar.db.cluster.node.standalone_mode.enabled=true + +# Enable the embedding feature +scalar.db.embedding.enabled=true + +# License key configurations +scalar.db.cluster.node.licensing.license_key= +scalar.db.cluster.node.licensing.license_check_cert_pem= +``` + +Additionally, you need to add the properties for the embedding store and the embedding model to the configuration file, depending on the embedding store and the embedding model you want to use. + +Select the embedding store that you want to use, and follow the instructions to configure it. + + + + The in-memory embedding store is a basic in-memory implementation. This embedding store is useful for fast prototyping and simple use cases. + + To use the in-memory embedding store, add the following property to the configuration file: + + ```properties + scalar.db.embedding.store.type=in-memory + ``` + + + The OpenSearch embedding store is an embedding store that uses OpenSearch as the backend. + + Select whether your OpenSearch implementation is running locally or running on AWS, and follow the instructions to configure it. + + + + For OpenSearch clusters that are running locally and are reachable on the network, add the following properties to the configuration file: + + ```properties + scalar.db.embedding.store.type=opensearch + + # OpenSearch Server URL. + scalar.db.embedding.store.opensearch.server_url= + + # OpenSearch API key (optional). + scalar.db.embedding.store.opensearch.api_key= + + # OpenSearch username (optional). + scalar.db.embedding.store.opensearch.user_name= + + # OpenSearch password (optional). + scalar.db.embedding.store.opensearch.password= + + # OpenSearch index name. + scalar.db.embedding.store.opensearch.index_name= + ``` + + + For OpenSearch clusters that are running as a fully managed service on AWS, add the following properties to the configuration file: + + ```properties + scalar.db.embedding.store.type=opensearch + + # OpenSearch Server URL. + scalar.db.embedding.store.opensearch.server_url= + + # The AWS signing service name, one of `es` (Amazon OpenSearch) or `aoss` (Amazon OpenSearch Serverless). + scalar.db.embedding.store.opensearch.service_name= + + # The AWS region for which requests will be signed. This should typically match the region in `server_url`. + scalar.db.embedding.store.opensearch.region= + + # The AWS access key ID. + scalar.db.embedding.store.opensearch.access_key_id= + + # The AWS secret access key. + scalar.db.embedding.store.opensearch.secret_access_key= + + # OpenSearch index name. + scalar.db.embedding.store.opensearch.index_name= + ``` + + + + + The Azure Cosmos DB for NoSQL embedding store is an embedding store that uses Azure Cosmos DB as the backend. + + To use the Azure Cosmos DB for NoSQL embedding store, add the following properties to the configuration file: + + ```properties + scalar.db.embedding.store.type=azure-cosmos-nosql + + # The Azure Cosmos DB endpoint that the SDK will connect to. + scalar.db.embedding.store.azure-cosmos-nosql.endpoint= + + # A master key used to perform authentication for accessing resources. A read-only key can also be used only for read-only operations. + scalar.db.embedding.store.azure-cosmos-nosql.key= + + # The database name to be used. + scalar.db.embedding.store.azure-cosmos-nosql.database_name= + + # The container name to be used. + scalar.db.embedding.store.azure-cosmos-nosql.container_name= + ``` + + + The Azure AI Search embedding store is an embedding store that uses Azure AI Search as the backend. + + To use the Azure AI Search embedding store, add the following properties to the configuration file: + + ```properties + scalar.db.embedding.store.type=azure-ai-search + # The Azure AI Search endpoint. + scalar.db.embedding.store.azure-ai-search.endpoint= + + # The Azure AI Search API key. + scalar.db.embedding.store.azure-ai-search.api_key= + + # The name of the index to be used. If no index is provided, the name of the default index to be used. + scalar.db.embedding.store.azure-ai-search.index_name= + ``` + + + The pgvector embedding store is an embedding store that uses pgvector, which is a Postgres extension for vector similarity search, as the backend. + + To use the pgvector embedding store, add the following properties to the configuration file: + + ```properties + scalar.db.embedding.store.type=pgvector + + # The database host. + scalar.db.embedding.store.pgvector.host= + + # The database port. + scalar.db.embedding.store.pgvector.port= + + # The database user. + scalar.db.embedding.store.pgvector.user= + + # The database password. + scalar.db.embedding.store.pgvector.password= + + # The database name. + scalar.db.embedding.store.pgvector.database= + + # The table name. + scalar.db.embedding.store.pgvector.table= + ``` + + + +Select the embedding model that you want to use, and follow the instructions to configure it. + + + + The in-process embedding model is a local embedding model powered by [ONNX runtime](https://onnxruntime.ai/docs/get-started/with-java.html) and is running in the ScalarDB Cluster process. This embedding model is useful for fast prototyping and simple use cases. + + To use the in-process embedding model, add the following property to the configuration file: + + ```properties + scalar.db.embedding.model.type=in-process + ``` + + + The Amazon Bedrock embedding model is an embedding model that uses Amazon Bedrock as the backend. + + To use the Amazon Bedrock embedding model, add the following properties to the configuration file: + + ```properties + scalar.db.embedding.model.type=bedrock-titan + + # The AWS region for which requests will be signed. + scalar.db.embedding.model.bedrock-titan.region= + + # The AWS access key ID. + scalar.db.embedding.model.bedrock-titan.access_key_id= + + # The AWS secret access key. + scalar.db.embedding.model.bedrock-titan.secret_access_key= + + # The model. Either `amazon.titan-embed-text-v1` or `amazon.titan-embed-text-v2:0`. + scalar.db.embedding.model.bedrock-titan.model= + + # The dimensions. + scalar.db.embedding.model.bedrock-titan.dimensions= + ``` + + + The Azure OpenAI embedding model is an embedding model that uses Azure OpenAI as the backend. + + To use the Azure OpenAI embedding model, add the following properties to the configuration file: + + ```properties + scalar.db.embedding.model.type=azure-open-ai + + # The Azure OpenAI endpoint. + scalar.db.embedding.model.azure-open-ai.endpoint= + + # The Azure OpenAI API key. + scalar.db.embedding.model.azure-open-ai.api_key= + + # The deployment name in Azure OpenAI. + scalar.db.embedding.model.azure-open-ai.deployment_name= + + # The dimensions. + scalar.db.embedding.model.azure-open-ai.dimensions= + ``` + + + The Google Vertex AI embedding model is an embedding model that uses Google Vertex AI as the backend. + + To use the Google Vertex AI embedding model, add the following properties to the configuration file: + + ```properties + scalar.db.embedding.model.type=vertex-ai + + # The Google Cloud project. + scalar.db.embedding.model.vertex-ai.project= + + # The Google Cloud location. + scalar.db.embedding.model.vertex-ai.location= + + # The endpoint. + scalar.db.embedding.model.vertex-ai.endpoint= + + # The publisher. + scalar.db.embedding.model.vertex-ai.publisher= + + # The model name. + scalar.db.embedding.model.vertex-ai.model_name= + + # The output dimensionality. + scalar.db.embedding.model.vertex-ai.output_dimensionality= + ``` + + + The OpenAI embedding model is an embedding model that uses OpenAI as the backend. + + To use the OpenAI embedding model, add the following properties to the configuration file: + + ```properties + scalar.db.embedding.model.type=open-ai + + # The OpenAI API key. + scalar.db.embedding.model.open-ai.api_key= + + # The model name. + scalar.db.embedding.model.open-ai.model_name= + + # The base URL. + scalar.db.embedding.model.open-ai.base_url= + + # The organization ID. + scalar.db.embedding.model.open-ai.organization_id= + + # The dimensions. + scalar.db.embedding.model.open-ai.dimensions= + + # The user. + scalar.db.embedding.model.open-ai.user= + ``` + + + +### 2. Create the Docker Compose file + +Create the following configuration file as `docker-compose.yaml`. + +```yaml +services: + scalardb-cluster-standalone: + container_name: "scalardb-cluster-node" + image: "ghcr.io/scalar-labs/scalardb-cluster-node-byol-premium:3.16.0" + ports: + - 60053:60053 + - 9080:9080 + volumes: + - ./scalardb-cluster-node.properties:/scalardb-cluster/node/scalardb-cluster-node.properties +``` + +### 3. Start ScalarDB Cluster + +Run the following command to start ScalarDB Cluster in standalone mode. + +```console +docker compose up -d +``` + +It may take a few minutes for ScalarDB Cluster to fully start. + +### 4. Add the Java Client SDK for the embedding store abstraction to your project + +The ScalarDB Cluster Embedding Java Client SDK library is available on the [Maven Central Repository](https://mvnrepository.com/artifact/com.scalar-labs/scalardb-cluster-embedding-java-client-sdk). You can add the library as a build dependency to your application by using Gradle or Maven. + +Select your build tool, and follow the instructions to add the build dependency for the ScalarDB Cluster Embedding Java Client SDK to your application. + + + + To add the build dependency for the ScalarDB Cluster Embedding Java Client SDK by using Gradle, add the following to `build.gradle` in your application: + + ```gradle + dependencies { + implementation 'com.scalar-labs:scalardb-cluster-embedding-java-client-sdk:3.16.0' + } + ``` + + + To add the build dependency for the ScalarDB Cluster Embedding Java Client SDK by using Maven, add the following to `pom.xml` in your application: + + ```xml + + com.scalar-labs + scalardb-cluster-embedding-java-client-sdk + 3.16.0 + + ``` + + + +### 5. Run the sample code + +Create a new Java class and add the following code to run the sample code. + +```java +try (ScalarDbEmbeddingClientFactory scalarDbEmbeddingClientFactory = + ScalarDbEmbeddingClientFactory.builder() + .withProperty("scalar.db.embedding.client.contact_points", "indirect:localhost") + .build()) { + // Create an embedding store and an embedding model. + EmbeddingStore scalarDbEmbeddingStore = + scalarDbEmbeddingClientFactory.createEmbeddingStore(); + EmbeddingModel scalarDbEmbeddingModel = scalarDbEmbeddingClientFactory.createEmbeddingModel(); + + // Add embeddings to the embedding store. + TextSegment segment1 = TextSegment.from("I like football."); + Embedding embedding1 = scalarDbEmbeddingModel.embed(segment1).content(); + scalarDbEmbeddingStore.add(embedding1, segment1); + + TextSegment segment2 = TextSegment.from("The weather is good today."); + Embedding embedding2 = scalarDbEmbeddingModel.embed(segment2).content(); + scalarDbEmbeddingStore.add(embedding2, segment2); + + // Search for embeddings. + Embedding queryEmbedding = + scalarDbEmbeddingModel.embed("What is your favourite sport?").content(); + EmbeddingSearchResult result = + scalarDbEmbeddingStore.search( + EmbeddingSearchRequest.builder() + .queryEmbedding(queryEmbedding) + .maxResults(1) + .build()); + + // Print the search result. + List> matches = result.matches(); + EmbeddingMatch embeddingMatch = matches.get(0); + System.out.println(embeddingMatch.embedded().text()); + System.out.println(embeddingMatch.score()); +} +``` + +This sample code demonstrates how to create an embedding store and an embedding model, add embeddings to the embedding store, and search for embeddings. + +Except for the part of the code that creates an embedding store and an embedding model, the usage is the same as LangChain4j. For more information about LangChain4j, see the following: +- [LangChain4j](https://docs.langchain4j.dev/) +- [Embedding Model](https://docs.langchain4j.dev/tutorials/rag#embedding-model) +- [Embedding Store](https://docs.langchain4j.dev/tutorials/rag#embedding-store) + +#### About `ScalarDbEmbeddingClientFactory` + +As shown in the code snippet, the `ScalarDbEmbeddingClientFactory` class provides a builder to create an instance of the factory. The builder allows you to set properties for the factory. In this example, the `withProperty()` method is used to set the contact points for the factory as follows: + +```java +ScalarDbEmbeddingClientFactory scalarDbEmbeddingClientFactory = ScalarDbEmbeddingClientFactory.builder() + .withProperty("scalar.db.embedding.client.contact_points", "indirect:localhost") + .build(); +``` + +You can also set a properties file by using the `withPropertiesFile()` method. + +Then, you can create an embedding store and an embedding model by using the factory as follows: + +```java +EmbeddingStore scalarDbEmbeddingStore = + scalarDbEmbeddingClientFactory.createEmbeddingStore(); +EmbeddingModel scalarDbEmbeddingModel = scalarDbEmbeddingClientFactory.createEmbeddingModel(); +``` + +Their methods internally connect to ScalarDB Cluster, which interacts with the embedding store by using the embedding model, both of which are specified in the configuration. + +You should reuse the `scalarDbEmbeddingStore` and `scalarDbEmbeddingModel` instances to interact with vector stores in an application. + +:::note + +The `ScalarDbEmbeddingClientFactory` instance should be closed after use to release the connection to ScalarDB Cluster. + +::: + +## Additional details + +The vector search feature is currently in Private Preview. For more details, please [contact us](https://www.scalar-labs.com/contact) or wait for this feature to become publicly available in a future version. + +- [Javadoc](https://javadoc.io/doc/com.scalar-labs/scalardb-cluster-embedding-java-client-sdk/3.16.0/index.html) diff --git a/versioned_docs/version-3.X/scalardb-cluster/images/direct-kubernetes-client-mode.png b/versioned_docs/version-3.X/scalardb-cluster/images/direct-kubernetes-client-mode.png new file mode 100644 index 00000000..13df52e6 Binary files /dev/null and b/versioned_docs/version-3.X/scalardb-cluster/images/direct-kubernetes-client-mode.png differ diff --git a/versioned_docs/version-3.X/scalardb-cluster/images/getting-started-ERD.png b/versioned_docs/version-3.X/scalardb-cluster/images/getting-started-ERD.png new file mode 100644 index 00000000..1a6d13c5 Binary files /dev/null and b/versioned_docs/version-3.X/scalardb-cluster/images/getting-started-ERD.png differ diff --git a/versioned_docs/version-3.X/scalardb-cluster/images/indirect-client-mode.png b/versioned_docs/version-3.X/scalardb-cluster/images/indirect-client-mode.png new file mode 100644 index 00000000..4e96108f Binary files /dev/null and b/versioned_docs/version-3.X/scalardb-cluster/images/indirect-client-mode.png differ diff --git a/versioned_docs/version-3.X/scalardb-cluster/images/scalardb-cluster-architecture.png b/versioned_docs/version-3.X/scalardb-cluster/images/scalardb-cluster-architecture.png new file mode 100644 index 00000000..24a0a50d Binary files /dev/null and b/versioned_docs/version-3.X/scalardb-cluster/images/scalardb-cluster-architecture.png differ diff --git a/versioned_docs/version-3.X/scalardb-cluster/images/scalardb-deployment-patterns.png b/versioned_docs/version-3.X/scalardb-cluster/images/scalardb-deployment-patterns.png new file mode 100644 index 00000000..6098c910 Binary files /dev/null and b/versioned_docs/version-3.X/scalardb-cluster/images/scalardb-deployment-patterns.png differ diff --git a/versioned_docs/version-3.X/scalardb-cluster/index.mdx b/versioned_docs/version-3.X/scalardb-cluster/index.mdx new file mode 100644 index 00000000..31f56f19 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster/index.mdx @@ -0,0 +1,71 @@ +--- +tags: + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# ScalarDB Cluster + +ScalarDB Cluster is a clustering solution for [ScalarDB](../overview.mdx) that consists of a set of cluster nodes, each of which provides ScalarDB functionality. Each cluster node has a routing mechanism that directs transaction requests to the appropriate cluster node within the cluster. + +## Why ScalarDB Cluster? + +When executing a transaction that spans multiple client requests, such as in microservice transactions, all requests for the transaction must be processed on the same server due to the stateful nature of transaction processing. However, in a distributed environment, routing requests to the same server isn't straightforward because a service typically runs on multiple servers (or hosts) for scalability and availability. In this scenario, all requests within a transaction must be routed to the same server, while different transactions should be distributed to ensure load balancing. + +To address this challenge, a routing mechanism such as session affinity (also known as sticky sessions) needs to be configured. This strategy ensures that requests within a transaction are consistently routed to the same server. Alternatively, you can leverage a bidirectional-streaming RPC by using gRPC. However, it's important to note that implementing these configurations typically requires significant time and effort. In addition, specific configuration adjustments may be required depending on the load balancer product you are using. + +For more details on this topic, see [Request routing in transactions with a two-phase commit interface](../two-phase-commit-transactions.mdx#request-routing-in-transactions-with-a-two-phase-commit-interface). + +ScalarDB Cluster addresses this issue by providing a routing mechanism capable of directing requests to the appropriate cluster node within the cluster. Thus, when a cluster node receives a request, the node can route that request to the correct cluster node in the cluster. + +## Architecture + +ScalarDB Cluster consists of a set of cluster nodes, each equipped with ScalarDB functionality. By using this solution, each cluster node can execute transactions independently. + +A notable feature of ScalarDB Cluster is the distribution of transaction requests by using a routing mechanism. When a cluster node receives a request, the node determines whether it's the appropriate cluster node to process the request. If it's not the appropriate node, the node routes the request to the appropriate cluster node within the cluster. To determine the appropriate cluster node, ScalarDB Cluster uses a consistent hashing algorithm. + +Membership management plays a critical role in ScalarDB Cluster. When a cluster node either joins or leaves the cluster, the configuration of the cluster is automatically adjusted to reflect this change. ScalarDB Cluster currently retrieves membership information by using the Kubernetes API. + +:::note + +Currently, ScalarDB Cluster supports running on Kubernetes only. + +::: + +![ScalarDB Cluster architecture](images/scalardb-cluster-architecture.png) + +## Getting started + +Before you start the tutorials, you need to set up ScalarDB Cluster. To set up ScalarDB Cluster, see [Set Up ScalarDB Cluster on Kubernetes by Using a Helm Chart](setup-scalardb-cluster-on-kubernetes-by-using-helm-chart.mdx). + +For tutorials on getting started with ScalarDB Cluster, see the following: + +* [Getting Started with ScalarDB Cluster](getting-started-with-scalardb-cluster.mdx) +* [Getting Started with ScalarDB Cluster GraphQL](getting-started-with-scalardb-cluster-graphql.mdx) +* [Getting Started with ScalarDB Cluster SQL via JDBC](getting-started-with-scalardb-cluster-sql-jdbc.mdx) +* [Getting Started with ScalarDB Cluster SQL via Spring Data JDBC for ScalarDB](getting-started-with-scalardb-cluster-sql-spring-data-jdbc.mdx) +* [Getting Started with ScalarDB Cluster via .NET](getting-started-with-scalardb-cluster-dotnet.mdx) +* [Getting Started with ScalarDB Cluster SQL via .NET](getting-started-with-scalardb-cluster-sql-dotnet.mdx) +* [Getting Started with ScalarDB Cluster SQL via .NET and LINQ](getting-started-with-scalardb-cluster-sql-linq.mdx) + +## References + +For details about the ScalarDB Cluster Helm Chart, refer to the following: + +* [ScalarDB Cluster Helm Chart](https://github.com/scalar-labs/helm-charts/tree/main/charts/scalardb-cluster) +* [Deploy Scalar products using Scalar Helm Charts](../helm-charts/how-to-deploy-scalar-products.mdx) +* [How to deploy ScalarDB Cluster](../helm-charts/how-to-deploy-scalardb-cluster.mdx) + +For details about the configurations for ScalarDB Cluster, refer to the following: + +* [ScalarDB Cluster Configurations](scalardb-cluster-configurations.mdx) + +For details about developing applications that use ScalarDB Cluster with the Java API, refer to the following: + +* [Developer Guide for ScalarDB Cluster with the Java API](developer-guide-for-scalardb-cluster-with-java-api.mdx) + +For details about the ScalarDB Cluster gRPC API, refer to the following: + +* [ScalarDB Cluster gRPC API Guide](scalardb-cluster-grpc-api-guide.mdx) +* [ScalarDB Cluster SQL gRPC API Guide](scalardb-cluster-sql-grpc-api-guide.mdx) diff --git a/versioned_docs/version-3.X/scalardb-cluster/run-non-transactional-storage-operations-through-scalardb-cluster.mdx b/versioned_docs/version-3.X/scalardb-cluster/run-non-transactional-storage-operations-through-scalardb-cluster.mdx new file mode 100644 index 00000000..3fcdeffd --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster/run-non-transactional-storage-operations-through-scalardb-cluster.mdx @@ -0,0 +1,334 @@ +--- +tags: + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Run Non-Transactional Storage Operations Through ScalarDB Cluster + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +This guide explains how to run non-transactional storage operations through ScalarDB Cluster. + +:::warning + +You need to have a license key (trial license or commercial license) for ScalarDB Cluster. If you don't have a license key, please [contact us](https://www.scalar-labs.com/contact). + +::: + +## Preparation + +For the purpose of this guide, you will set up a database and ScalarDB Cluster in standalone mode by using a sample in the ScalarDB samples repository. + +:::note + +ScalarDB Cluster in standalone mode is primarily for development and testing purposes. + +::: + +### Clone the ScalarDB samples repository + +Open **Terminal**, then clone the ScalarDB samples repository by running the following command: + +```console +git clone https://github.com/scalar-labs/scalardb-samples +``` + +Then, go to the directory that contains the necessary files by running the following command: + +```console +cd scalardb-samples/scalardb-cluster-standalone-mode +``` + +## Set up a database + +Select your database, and follow the instructions to configure it for ScalarDB Cluster. + +For a list of databases that ScalarDB supports, see [Databases](../requirements.mdx#databases). + + + +

Run MySQL locally

+ + You can run MySQL in Docker Compose by using the `docker-compose.yaml` file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory. + + To start MySQL, run the following command: + + ```console + docker compose up -d mysql + ``` + +

Configure ScalarDB Cluster

+ + The **scalardb-cluster-node.properties** file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Please uncomment the properties for MySQL in the **scalardb-cluster-node.properties** file so that the configuration looks as follows: + + ```properties + # For MySQL + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:mysql://mysql-1:3306/ + scalar.db.username=root + scalar.db.password=mysql + ``` + + +

Run PostgreSQL locally

+ + You can run PostgreSQL in Docker Compose by using the `docker-compose.yaml` file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory. + + To start PostgreSQL, run the following command: + + ```console + docker compose up -d postgres + ``` + +

Configure ScalarDB Cluster

+ + The **scalardb-cluster-node.properties** file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Please uncomment the properties for PostgreSQL in the **scalardb-cluster-node.properties** file so that the configuration looks as follows: + + ```properties + # For PostgreSQL + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:postgresql://postgres-1:5432/ + scalar.db.username=postgres + scalar.db.password=postgres + ``` + + +

Run Oracle Database locally

+ + You can run Oracle Database in Docker Compose by using the `docker-compose.yaml` file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory. + + To start Oracle Database, run the following command: + + ```console + docker compose up -d oracle + ``` + +

Configure ScalarDB Cluster

+ + The **scalardb-cluster-node.properties** file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Please uncomment the properties for Oracle Database in the **scalardb-cluster-node.properties** file so that the configuration looks as follows: + + ```properties + # For Oracle + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:oracle:thin:@//oracle-1:1521/FREEPDB1 + scalar.db.username=SYSTEM + scalar.db.password=Oracle + ``` + + +

Run SQL Server locally

+ + You can run SQL Server in Docker Compose by using the `docker-compose.yaml` file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory. + + To start SQL Server, run the following command: + + ```console + docker compose up -d sqlserver + ``` + +

Configure ScalarDB Cluster

+ + The **scalardb-cluster-node.properties** file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Please uncomment the properties for SQL Server in the **scalardb-cluster-node.properties** file so that the configuration looks as follows: + + ```properties + # For SQL Server + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:sqlserver://sqlserver-1:1433;encrypt=true;trustServerCertificate=true + scalar.db.username=sa + scalar.db.password=SqlServer22 + ``` + + +

Run Db2 locally

+ + You can run IBM Db2 in Docker Compose by using the `docker-compose.yml` file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory. + + To start IBM Db2, run the following command: + + ```console + docker compose up -d db2 + ``` + +

Configure ScalarDB Cluster

+ + The **scalardb-cluster-node.properties** file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Please uncomment the properties for Db2 in the **scalardb-cluster-node.properties** file so that the configuration looks as follows: + + ```properties + # For Db2 + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:db2://db2-1:50000/sample + scalar.db.username=db2inst1 + scalar.db.password=db2inst1 + ``` + + +

Run Amazon DynamoDB Local

+ + You can run Amazon DynamoDB Local in Docker Compose by using the `docker-compose.yaml` file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory. + + To start Amazon DynamoDB Local, run the following command: + + ```console + docker compose up -d dynamodb + ``` + +

Configure ScalarDB Cluster

+ + The **scalardb-cluster-node.properties** file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Please uncomment the properties for Amazon DynamoDB Local in the **scalardb-cluster-node.properties** file so that the configuration looks as follows: + + ```properties + # For DynamoDB Local + scalar.db.storage=dynamo + scalar.db.contact_points=sample + scalar.db.username=sample + scalar.db.password=sample + scalar.db.dynamo.endpoint_override=http://dynamodb-1:8000 + ``` + + + To use Azure Cosmos DB for NoSQL, you must have an Azure account. If you don't have an Azure account, visit [Create an Azure Cosmos DB account](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/quickstart-portal#create-account). + +

Configure Cosmos DB for NoSQL

+ + Set the **default consistency level** to **Strong** according to the official document at [Configure the default consistency level](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-manage-consistency#configure-the-default-consistency-level). + +

Configure ScalarDB Cluster

+ + The following instructions assume that you have properly installed and configured the JDK in your local environment and properly configured your Cosmos DB for NoSQL account in Azure. + + The **scalardb-cluster-node.properties** file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Be sure to change the values for `scalar.db.contact_points` and `scalar.db.password` as described. + + ```properties + # For Cosmos DB + scalar.db.storage=cosmos + scalar.db.contact_points= + scalar.db.password= + ``` + +:::note + +You can use the primary key or the secondary key in your Azure Cosmos DB account as the value for `scalar.db.password`. + +::: + + +

Run Cassandra locally

+ + You can run Apache Cassandra in Docker Compose by using the `docker-compose.yaml` file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory. + + To start Apache Cassandra, run the following command: + + ```console + docker compose up -d cassandra + ``` + +

Configure ScalarDB Cluster

+ + The **scalardb-cluster-node.properties** file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Please uncomment the properties for Cassandra in the **scalardb-cluster-node.properties** file so that the configuration looks as follows: + + ```properties + # For Cassandra + scalar.db.storage=cassandra + scalar.db.contact_points=cassandra-1 + scalar.db.username=cassandra + scalar.db.password=cassandra + ``` + + + +For a comprehensive list of configurations for ScalarDB, see [ScalarDB Configurations](../configurations.mdx). + +## Set up ScalarDB Cluster in standalone mode + +To set up ScalarDB Cluster in standalone mode, you'll need to configure ScalarDB Cluster to run non-transactional storage operations, set a license key, and then start ScalarDB Cluster. + +### Configure ScalarDB Cluster to run non-transactional storage operations + +To run non-transactional storage operations, you need to configure the `scalar.db.transaction_manager` property to `single-crud-operation` in the configuration file `scalardb-cluster-node.properties`: + +```properties +scalar.db.transaction_manager=single-crud-operation +``` + +### Set the license key + +Set the license key (trial license or commercial license) for the ScalarDB Clusters in the properties file. For details, see [How to Configure a Product License Key](../scalar-licensing/index.mdx). + +### Start ScalarDB Cluster in standalone mode + +To start ScalarDB Cluster in standalone mode, run the following command: + +:::note + +If you want to change other configurations for ScalarDB Cluster, update the `scalardb-cluster-node.properties` file before running the command below. + +::: + +```console +docker compose up -d scalardb-cluster-node +``` + +## Create or import a schema + +ScalarDB has its own data model and schema that maps to the implementation-specific data model and schema. + +- **Need to create a database schema?** See [Schema Loader for Cluster](developer-guide-for-scalardb-cluster-with-java-api.mdx#schema-loader-for-cluster). +- **Need to import an existing database?** See [Importing Existing Tables to ScalarDB by Using ScalarDB Schema Loader](../schema-loader-import.mdx). + +## Create your Java application + +This section describes how to add the ScalarDB Cluster Java Client SDK to your project and how to configure it to run non-transactional storage operations by using Java. + +### Add the ScalarDB Cluster Java Client SDK to your build + +The ScalarDB Cluster Java Client SDK is available in the [Maven Central Repository](https://mvnrepository.com/artifact/com.scalar-labs/scalardb-cluster-java-client-sdk). You can add the SDK as a build dependency to your application by using Gradle or Maven. + +Select your build tool, and follow the instructions to add the build dependency for the ScalarDB Cluster Java Client SDK to your application. + + + + To add the build dependency for the ScalarDB Cluster Java Client SDK by using Gradle, add the following to `build.gradle` in your application: + + ```gradle + dependencies { + implementation 'com.scalar-labs:scalardb-cluster-java-client-sdk:3.16.0' + } + ``` + + + To add the build dependency for the ScalarDB Cluster Java Client SDK by using Maven, add the following to `pom.xml` in your application: + + ```xml + + com.scalar-labs + scalardb-cluster-java-client-sdk + 3.16.0 + + ``` + + + +### Configure the ScalarDB Cluster Java SDK + +For details about configuring the ScalarDB Cluster Java SDK, see [Client configurations](developer-guide-for-scalardb-cluster-with-java-api.mdx#client-configurations) + +### Use the Java API + +For details about the Java API, see [ScalarDB Java API Guide](../api-guide.mdx). + +:::note + +The following limitations apply to non-transactional storage operations: + +- Beginning a transaction is not supported. For more details, see [Execute transactions without beginning or starting a transaction](../api-guide.mdx#execute-transactions-without-beginning-or-starting-a-transaction). +- Executing multiple mutations in a single transaction is not supported. + +::: + +### Learn more + +- [Javadoc](https://javadoc.io/doc/com.scalar-labs/scalardb/3.16.0/index.html) +- [Developer Guide for ScalarDB Cluster with the Java API](developer-guide-for-scalardb-cluster-with-java-api.mdx) diff --git a/versioned_docs/version-3.X/scalardb-cluster/run-non-transactional-storage-operations-through-sql-interface.mdx b/versioned_docs/version-3.X/scalardb-cluster/run-non-transactional-storage-operations-through-sql-interface.mdx new file mode 100644 index 00000000..71249c3b --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster/run-non-transactional-storage-operations-through-sql-interface.mdx @@ -0,0 +1,393 @@ +--- +tags: + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Run Non-Transactional Storage Operations Through the SQL Interface + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +This guide explains how to run non-transactional storage operations through the SQL interface for ScalarDB Cluster. + +:::warning + +You need to have a license key (trial license or commercial license) for ScalarDB Cluster. If you don't have a license key, please [contact us](https://www.scalar-labs.com/contact). + +::: + +## Preparation + +For the purpose of this guide, you will set up a database and ScalarDB Cluster in standalone mode by using a sample in the ScalarDB samples repository. + +:::note + +ScalarDB Cluster in standalone mode is primarily for development and testing purposes. + +::: + +### Clone the ScalarDB samples repository + +Open **Terminal**, then clone the ScalarDB samples repository by running the following command: + +```console +git clone https://github.com/scalar-labs/scalardb-samples +``` + +Then, go to the directory that contains the necessary files by running the following command: + +```console +cd scalardb-samples/scalardb-cluster-standalone-mode +``` + +## Set up a database + +Select your database, and follow the instructions to configure it for ScalarDB Cluster. + +For a list of databases that ScalarDB supports, see [Databases](../requirements.mdx#databases). + + + +

Run MySQL locally

+ + You can run MySQL in Docker Compose by using the `docker-compose.yaml` file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory. + + To start MySQL, run the following command: + + ```console + docker compose up -d mysql + ``` + +

Configure ScalarDB Cluster

+ + The **scalardb-cluster-node.properties** file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Please uncomment the properties for MySQL in the **scalardb-cluster-node.properties** file so that the configuration looks as follows: + + ```properties + # For MySQL + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:mysql://mysql-1:3306/ + scalar.db.username=root + scalar.db.password=mysql + ``` + + +

Run PostgreSQL locally

+ + You can run PostgreSQL in Docker Compose by using the `docker-compose.yaml` file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory. + + To start PostgreSQL, run the following command: + + ```console + docker compose up -d postgres + ``` + +

Configure ScalarDB Cluster

+ + The **scalardb-cluster-node.properties** file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Please uncomment the properties for PostgreSQL in the **scalardb-cluster-node.properties** file so that the configuration looks as follows: + + ```properties + # For PostgreSQL + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:postgresql://postgres-1:5432/ + scalar.db.username=postgres + scalar.db.password=postgres + ``` + + +

Run Oracle Database locally

+ + You can run Oracle Database in Docker Compose by using the `docker-compose.yaml` file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory. + + To start Oracle Database, run the following command: + + ```console + docker compose up -d oracle + ``` + +

Configure ScalarDB Cluster

+ + The **scalardb-cluster-node.properties** file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Please uncomment the properties for Oracle Database in the **scalardb-cluster-node.properties** file so that the configuration looks as follows: + + ```properties + # For Oracle + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:oracle:thin:@//oracle-1:1521/FREEPDB1 + scalar.db.username=SYSTEM + scalar.db.password=Oracle + ``` + + +

Run SQL Server locally

+ + You can run SQL Server in Docker Compose by using the `docker-compose.yaml` file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory. + + To start SQL Server, run the following command: + + ```console + docker compose up -d sqlserver + ``` + +

Configure ScalarDB Cluster

+ + The **scalardb-cluster-node.properties** file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Please uncomment the properties for SQL Server in the **scalardb-cluster-node.properties** file so that the configuration looks as follows: + + ```properties + # For SQL Server + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:sqlserver://sqlserver-1:1433;encrypt=true;trustServerCertificate=true + scalar.db.username=sa + scalar.db.password=SqlServer22 + ``` + + +

Run Amazon DynamoDB Local

+ + You can run Amazon DynamoDB Local in Docker Compose by using the `docker-compose.yaml` file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory. + + To start Amazon DynamoDB Local, run the following command: + + ```console + docker compose up -d dynamodb + ``` + +

Configure ScalarDB Cluster

+ + The **scalardb-cluster-node.properties** file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Please uncomment the properties for Amazon DynamoDB Local in the **scalardb-cluster-node.properties** file so that the configuration looks as follows: + + ```properties + # For DynamoDB Local + scalar.db.storage=dynamo + scalar.db.contact_points=sample + scalar.db.username=sample + scalar.db.password=sample + scalar.db.dynamo.endpoint_override=http://dynamodb-1:8000 + ``` + + + To use Azure Cosmos DB for NoSQL, you must have an Azure account. If you don't have an Azure account, visit [Create an Azure Cosmos DB account](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/quickstart-portal#create-account). + +

Configure Cosmos DB for NoSQL

+ + Set the **default consistency level** to **Strong** according to the official document at [Configure the default consistency level](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-manage-consistency#configure-the-default-consistency-level). + +

Configure ScalarDB Cluster

+ + The following instructions assume that you have properly installed and configured the JDK in your local environment and properly configured your Cosmos DB for NoSQL account in Azure. + + The **scalardb-cluster-node.properties** file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Be sure to change the values for `scalar.db.contact_points` and `scalar.db.password` as described. + + ```properties + # For Cosmos DB + scalar.db.storage=cosmos + scalar.db.contact_points= + scalar.db.password= + ``` + +:::note + +You can use the primary key or the secondary key in your Azure Cosmos DB account as the value for `scalar.db.password`. + +::: + + +

Run Cassandra locally

+ + You can run Apache Cassandra in Docker Compose by using the `docker-compose.yaml` file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory. + + To start Apache Cassandra, run the following command: + + ```console + docker compose up -d cassandra + ``` + +

Configure ScalarDB Cluster

+ + The **scalardb-cluster-node.properties** file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Please uncomment the properties for Cassandra in the **scalardb-cluster-node.properties** file so that the configuration looks as follows: + + ```properties + # For Cassandra + scalar.db.storage=cassandra + scalar.db.contact_points=cassandra-1 + scalar.db.username=cassandra + scalar.db.password=cassandra + ``` + + + +For a comprehensive list of configurations for ScalarDB, see [ScalarDB Configurations](../configurations.mdx). + +## Set up ScalarDB Cluster in standalone mode + +To set up ScalarDB Cluster in standalone mode, you'll need to configure ScalarDB Cluster to run non-transactional storage operations, set a license key, and then start ScalarDB Cluster. + +### Configure ScalarDB Cluster to run non-transactional storage operations + +To run non-transactional storage operations, you need to configure the `scalar.db.transaction_manager` property to `single-crud-operation` in the configuration file `scalardb-cluster-node.properties`: + +```properties +scalar.db.transaction_manager=single-crud-operation +``` + +### Set the license key + +Set the license key (trial license or commercial license) for the ScalarDB Clusters in the properties file. For details, see [How to Configure a Product License Key](../scalar-licensing/index.mdx). + +### Start ScalarDB Cluster in standalone mode + +To start ScalarDB Cluster in standalone mode, run the following command: + +:::note + +If you want to change other configurations for ScalarDB Cluster, update the `scalardb-cluster-node.properties` file before running the command below. + +::: + +```console +docker compose up -d scalardb-cluster-node +``` + +## Create or import a schema + +ScalarDB has its own data model and schema that maps to the implementation-specific data model and schema. + +- **Need to create a database schema?** See [SQL CLI](developer-guide-for-scalardb-cluster-with-java-api.mdx#sql-cli). +- **Need to import an existing database?** See [Importing Existing Tables to ScalarDB by Using ScalarDB Schema Loader](../schema-loader-import.mdx). + +Also, for a list of supported DDLs, see [ScalarDB SQL Grammar](../scalardb-sql/grammar.mdx). + +## Create your application + + + +

Configure your JDBC application

+ + This section describes how to add the ScalarDB JDBC driver to your project and how to configure it to run non-transactional storage operations by using Java. + +

Add the ScalarDB JDBC driver to your project

+ + You can add the ScalarDB JDBC driver as a build dependency to your application by using Gradle or Maven. + + Select your build tool, and follow the instructions to add the build dependency for the ScalarDB JDBC driver to your application. + + + + To add the build dependency for the ScalarDB JDBC driver by using Gradle, add the following to `build.gradle` in your application: + + ```gradle + dependencies { + implementation 'com.scalar-labs:scalardb-sql-jdbc:3.16.0' + implementation 'com.scalar-labs:scalardb-cluster-java-client-sdk:3.16.0' + } + ``` + + + To add the build dependency for the ScalarDB SQL API by using Maven, add the following to `pom.xml` in your application: + + ```xml + + + com.scalar-labs + scalardb-sql-jdbc + 3.16.0 + + + com.scalar-labs + scalardb-cluster-java-client-sdk + 3.16.0 + + + ``` + + + +

Configure the ScalarDB Cluster Java SDK for the SQL interface

+ + For details about configuring the ScalarDB Cluster Java SDK for the SQL interface, see [ScalarDB Cluster SQL client configurations](developer-guide-for-scalardb-cluster-with-java-api.mdx#scalardb-cluster-sql-client-configurations). + +

Use the JDBC API

+ + For details about the JDBC API, see [ScalarDB JDBC Guide](../scalardb-sql/jdbc-guide.mdx). + +:::note + +The following limitations apply to non-transactional storage operations: + +- Beginning a transaction is not supported. +- Executing multiple mutations in a single SQL statement is not supported. +- The isolation level is always `READ_COMMITTED`. + +::: + +

Learn more

+ + - [Developer Guide for ScalarDB Cluster with the Java API](developer-guide-for-scalardb-cluster-with-java-api.mdx) + - [Java JDBC API](https://docs.oracle.com/javase/8/docs/technotes/guides/jdbc/) + + +

Configure your Java application

+ + This section describes how to add the ScalarDB SQL API to your project and how to configure it to run non-transactional storage operations by using Java. + +

Add ScalarDB SQL API to your project

+ + You can add the ScalarDB SQL API as a build dependency to your application by using Gradle or Maven. + + Select your build tool, and follow the instructions to add the build dependency for the ScalarDB SQL API to your application. + + + + To add the build dependency for the ScalarDB SQL API by using Gradle, add the following to `build.gradle` in your application: + + ```gradle + dependencies { + implementation 'com.scalar-labs:scalardb-sql:3.16.0' + implementation 'com.scalar-labs:scalardb-cluster-java-client-sdk:3.16.0' + } + ``` + + + To add the build dependency for the ScalarDB SQL API by using Maven, add the following to `pom.xml` in your application: + + ```xml + + + com.scalar-labs + scalardb-sql + 3.16.0 + + + com.scalar-labs + scalardb-cluster-java-client-sdk + 3.16.0 + + + ``` + + + +

Configure the ScalarDB Cluster Java SDK for the SQL interface

+ + For details about configuring the ScalarDB Cluster Java SDK for the SQL interface, see [ScalarDB Cluster SQL client configurations](developer-guide-for-scalardb-cluster-with-java-api.mdx#scalardb-cluster-sql-client-configurations). + +

Use the Java API

+ + For details about the SQL API, see [ScalarDB SQL API Guide](../scalardb-sql/sql-api-guide.mdx). + +:::note + +The following limitations apply to non-transactional storage operations: + +- Beginning a transaction is not supported. +- Executing multiple mutations in a single SQL statement is not supported. +- The isolation level is always `READ_COMMITTED`. + +::: + + +

Learn more

+ + - [Javadoc](https://javadoc.io/doc/com.scalar-labs/scalardb-sql/3.16.0/index.html) + + + diff --git a/versioned_docs/version-3.X/scalardb-cluster/run-transactions-through-scalardb-cluster-sql.mdx b/versioned_docs/version-3.X/scalardb-cluster/run-transactions-through-scalardb-cluster-sql.mdx new file mode 100644 index 00000000..d4504ac0 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster/run-transactions-through-scalardb-cluster-sql.mdx @@ -0,0 +1,321 @@ +--- +tags: + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Run Transactions Through ScalarDB Cluster SQL + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +This guide explains how to configure your ScalarDB properties file and creating schemas to run transactions through a one-phase or a two-phase commit interface by using ScalarDB Cluster SQL. + +:::warning + +You need to have a license key (trial license or commercial license) for ScalarDB Cluster. If you don't have a license key, please [contact us](https://www.scalar-labs.com/contact). + +::: + +## Preparation + +For the purpose of this guide, you will set up a database and ScalarDB Cluster in standalone mode by using a sample in the ScalarDB samples repository. + +:::note + +ScalarDB Cluster in standalone mode is primarily for development and testing purposes. + +::: + +### Clone the ScalarDB samples repository + +Open **Terminal**, then clone the ScalarDB samples repository by running the following command: + +```console +git clone https://github.com/scalar-labs/scalardb-samples +``` + +Then, go to the directory that contains the necessary files by running the following command: + +```console +cd scalardb-samples/scalardb-cluster-standalone-mode +``` + +## Set up a database + +Select your database, and follow the instructions to configure it for ScalarDB Cluster. + +For a list of databases that ScalarDB supports, see [Databases](../requirements.mdx#databases). + + + +

Run MySQL locally

+ + You can run MySQL in Docker Compose by using the `docker-compose.yaml` file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory. + + To start MySQL, run the following command: + + ```console + docker compose up -d mysql + ``` + +

Configure ScalarDB Cluster

+ + The **scalardb-cluster-node.properties** file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Please uncomment the properties for MySQL in the **scalardb-cluster-node.properties** file so that the configuration looks as follows: + + ```properties + # For MySQL + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:mysql://mysql-1:3306/ + scalar.db.username=root + scalar.db.password=mysql + ``` + + +

Run PostgreSQL locally

+ + You can run PostgreSQL in Docker Compose by using the `docker-compose.yaml` file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory. + + To start PostgreSQL, run the following command: + + ```console + docker compose up -d postgres + ``` + +

Configure ScalarDB Cluster

+ + The **scalardb-cluster-node.properties** file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Please uncomment the properties for PostgreSQL in the **scalardb-cluster-node.properties** file so that the configuration looks as follows: + + ```properties + # For PostgreSQL + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:postgresql://postgres-1:5432/ + scalar.db.username=postgres + scalar.db.password=postgres + ``` + + +

Run Oracle Database locally

+ + You can run Oracle Database in Docker Compose by using the `docker-compose.yaml` file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory. + + To start Oracle Database, run the following command: + + ```console + docker compose up -d oracle + ``` + +

Configure ScalarDB Cluster

+ + The **scalardb-cluster-node.properties** file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Please uncomment the properties for Oracle Database in the **scalardb-cluster-node.properties** file so that the configuration looks as follows: + + ```properties + # For Oracle + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:oracle:thin:@//oracle-1:1521/FREEPDB1 + scalar.db.username=SYSTEM + scalar.db.password=Oracle + ``` + + +

Run SQL Server locally

+ + You can run SQL Server in Docker Compose by using the `docker-compose.yaml` file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory. + + To start SQL Server, run the following command: + + ```console + docker compose up -d sqlserver + ``` + +

Configure ScalarDB Cluster

+ + The **scalardb-cluster-node.properties** file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Please uncomment the properties for SQL Server in the **scalardb-cluster-node.properties** file so that the configuration looks as follows: + + ```properties + # For SQL Server + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:sqlserver://sqlserver-1:1433;encrypt=true;trustServerCertificate=true + scalar.db.username=sa + scalar.db.password=SqlServer22 + ``` + + +

Run Db2 locally

+ + You can run IBM Db2 in Docker Compose by using the `docker-compose.yml` file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory. + + To start IBM Db2, run the following command: + + ```console + docker compose up -d db2 + ``` + +

Configure ScalarDB Cluster

+ + The **scalardb-cluster-node.properties** file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Please uncomment the properties for Db2 in the **scalardb-cluster-node.properties** file so that the configuration looks as follows: + + ```properties + # For Db2 + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:db2://db2-1:50000/sample + scalar.db.username=db2inst1 + scalar.db.password=db2inst1 + ``` + + +

Run Amazon DynamoDB Local

+ + You can run Amazon DynamoDB Local in Docker Compose by using the `docker-compose.yaml` file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory. + + To start Amazon DynamoDB Local, run the following command: + + ```console + docker compose up -d dynamodb + ``` + +

Configure ScalarDB Cluster

+ + The **scalardb-cluster-node.properties** file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Please uncomment the properties for Amazon DynamoDB Local in the **scalardb-cluster-node.properties** file so that the configuration looks as follows: + + ```properties + # For DynamoDB Local + scalar.db.storage=dynamo + scalar.db.contact_points=sample + scalar.db.username=sample + scalar.db.password=sample + scalar.db.dynamo.endpoint_override=http://dynamodb-1:8000 + ``` + + + To use Azure Cosmos DB for NoSQL, you must have an Azure account. If you don't have an Azure account, visit [Create an Azure Cosmos DB account](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/quickstart-portal#create-account). + +

Configure Cosmos DB for NoSQL

+ + Set the **default consistency level** to **Strong** according to the official document at [Configure the default consistency level](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-manage-consistency#configure-the-default-consistency-level). + +

Configure ScalarDB Cluster

+ + The following instructions assume that you have properly installed and configured the JDK in your local environment and properly configured your Cosmos DB for NoSQL account in Azure. + + The **scalardb-cluster-node.properties** file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Be sure to change the values for `scalar.db.contact_points` and `scalar.db.password` as described. + + ```properties + # For Cosmos DB + scalar.db.storage=cosmos + scalar.db.contact_points= + scalar.db.password= + ``` + +:::note + +You can use the primary key or the secondary key in your Azure Cosmos DB account as the value for `scalar.db.password`. + +::: + + +

Run Cassandra locally

+ + You can run Apache Cassandra in Docker Compose by using the `docker-compose.yaml` file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory. + + To start Apache Cassandra, run the following command: + + ```console + docker compose up -d cassandra + ``` + +

Configure ScalarDB Cluster

+ + The **scalardb-cluster-node.properties** file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Please uncomment the properties for Cassandra in the **scalardb-cluster-node.properties** file so that the configuration looks as follows: + + ```properties + # For Cassandra + scalar.db.storage=cassandra + scalar.db.contact_points=cassandra-1 + scalar.db.username=cassandra + scalar.db.password=cassandra + ``` + + + +For a comprehensive list of configurations for ScalarDB Cluster SQL, see [ScalarDB Cluster SQL client configurations](developer-guide-for-scalardb-cluster-with-java-api.mdx#scalardb-cluster-sql-client-configurations). + +## Set up ScalarDB Cluster in standalone mode + +To set up ScalarDB Cluster in standalone mode, you'll need to set a license key and then start ScalarDB Cluster. + +### Set the license key + +Set the license key (trial license or commercial license) for the ScalarDB Clusters in the properties file. For details, see [How to Configure a Product License Key](../scalar-licensing/index.mdx). + +### Start ScalarDB Cluster in standalone mode + +To start ScalarDB Cluster in standalone mode, run the following command: + +:::note + +If you want to change other configurations for ScalarDB Cluster, update the `scalardb-cluster-node.properties` file before running the command below. + +::: + +```console +docker compose up -d scalardb-cluster-node +``` + +## Create or import a schema + +ScalarDB has its own data model and schema that maps to the implementation-specific data model and schema. + +- **Need to create a database schema?** See [SQL CLI](developer-guide-for-scalardb-cluster-with-java-api.mdx#sql-cli). +- **Need to import an existing database?** See [Importing Existing Tables to ScalarDB by Using ScalarDB Schema Loader](../schema-loader-import.mdx). + +## Run transactions + +You can run transactions by using a one-phase or a two-phase commit interface. Select your method for running transactions. + + + +

One-phase commit interface

+ + For details about how to run transactions by using a one-phase commit interface, see the [ScalarDB SQL JDBC Guide](../scalardb-sql/jdbc-guide.mdx). + +:::note + +Documentation for how to run transactions in a two-phase commit interface is coming soon. + +::: + + +

One-phase commit interface

+ + For details about how to run transactions by using a one-phase commit interface, see the [ScalarDB SQL API Guide](../scalardb-sql/sql-api-guide.mdx). + +:::note + +Documentation for how to run transactions in a two-phase commit interface is coming soon. + +::: + +

Learn more

+ + To learn more about running transactions by using ScalarDB Cluster SQL, see the following: + + - [ScalarDB Cluster SQL gRPC API Guide](scalardb-cluster-sql-grpc-api-guide.mdx) + + +

One-phase or two-phase commit interface

+ + For details about how to run transactions by using a one-phase or a two-phase commit interface, see the [Getting Started with LINQ in the ScalarDB Cluster .NET Client SDK](../scalardb-cluster-dotnet-client-sdk/getting-started-with-linq.mdx#manage-transactions). + + +

One-phase commit interface

+ + For details about how to run transactions by using a one-phase commit interface, see the [Getting Started with Distributed SQL Transactions in the ScalarDB Cluster .NET Client SDK](../scalardb-cluster-dotnet-client-sdk/getting-started-with-distributed-sql-transactions.mdx). + +:::note + +Documentation for how to run transactions in a two-phase commit interface is coming soon. For now, please refer to [Getting Started with Distributed Transactions with a Two-Phase Commit Interface in the ScalarDB Cluster .NET Client SDK](../scalardb-cluster-dotnet-client-sdk/getting-started-with-two-phase-commit-transactions.mdx). + +::: + + diff --git a/versioned_docs/version-3.X/scalardb-cluster/run-transactions-through-scalardb-cluster.mdx b/versioned_docs/version-3.X/scalardb-cluster/run-transactions-through-scalardb-cluster.mdx new file mode 100644 index 00000000..c4e7fb10 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster/run-transactions-through-scalardb-cluster.mdx @@ -0,0 +1,326 @@ +--- +tags: + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Run Transactions Through ScalarDB Cluster + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +This guide explains how to configure your ScalarDB properties file and create schemas to run transactions through a one-phase or a two-phase commit interface by using ScalarDB Cluster. + +:::warning + +You need to have a license key (trial license or commercial license) for ScalarDB Cluster. If you don't have a license key, please [contact us](https://www.scalar-labs.com/contact). + +::: + +## Preparation + +For the purpose of this guide, you will set up a database and ScalarDB Cluster in standalone mode by using a sample in the ScalarDB samples repository. + +:::note + +ScalarDB Cluster in standalone mode is primarily for development and testing purposes. + +::: + +### Clone the ScalarDB samples repository + +Open **Terminal**, then clone the ScalarDB samples repository by running the following command: + +```console +git clone https://github.com/scalar-labs/scalardb-samples +``` + +Then, go to the directory that contains the necessary files by running the following command: + +```console +cd scalardb-samples/scalardb-cluster-standalone-mode +``` + +## Set up a database + +Select your database, and follow the instructions to configure it for ScalarDB Cluster. + +For a list of databases that ScalarDB supports, see [Databases](../requirements.mdx#databases). + + + +

Run MySQL locally

+ + You can run MySQL in Docker Compose by using the `docker-compose.yaml` file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory. + + To start MySQL, run the following command: + + ```console + docker compose up -d mysql + ``` + +

Configure ScalarDB Cluster

+ + The `scalardb-cluster-node.properties` file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Please uncomment the properties for MySQL in the `scalardb-cluster-node.properties` file so that the configuration looks as follows: + + ```properties + # For MySQL + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:mysql://mysql-1:3306/ + scalar.db.username=root + scalar.db.password=mysql + ``` + + +

Run PostgreSQL locally

+ + You can run PostgreSQL in Docker Compose by using the `docker-compose.yaml` file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory. + + To start PostgreSQL, run the following command: + + ```console + docker compose up -d postgres + ``` + +

Configure ScalarDB Cluster

+ + The `scalardb-cluster-node.properties` file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Please uncomment the properties for PostgreSQL in the `scalardb-cluster-node.properties` file so that the configuration looks as follows: + + ```properties + # For PostgreSQL + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:postgresql://postgres-1:5432/ + scalar.db.username=postgres + scalar.db.password=postgres + ``` + + +

Run Oracle Database locally

+ + You can run Oracle Database in Docker Compose by using the `docker-compose.yaml` file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory. + + To start Oracle Database, run the following command: + + ```console + docker compose up -d oracle + ``` + +

Configure ScalarDB Cluster

+ + The `scalardb-cluster-node.properties` file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Please uncomment the properties for Oracle Database in the `scalardb-cluster-node.properties` file so that the configuration looks as follows: + + ```properties + # For Oracle + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:oracle:thin:@//oracle-1:1521/FREEPDB1 + scalar.db.username=SYSTEM + scalar.db.password=Oracle + ``` + + +

Run SQL Server locally

+ + You can run SQL Server in Docker Compose by using the `docker-compose.yaml` file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory. + + To start SQL Server, run the following command: + + ```console + docker compose up -d sqlserver + ``` + +

Configure ScalarDB Cluster

+ + The **scalardb-cluster-node.properties** file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Please uncomment the properties for SQL Server in the **scalardb-cluster-node.properties** file so that the configuration looks as follows: + + ```properties + # For SQL Server + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:sqlserver://sqlserver-1:1433;encrypt=true;trustServerCertificate=true + scalar.db.username=sa + scalar.db.password=SqlServer22 + ``` + + +

Run Db2 locally

+ + You can run IBM Db2 in Docker Compose by using the `docker-compose.yml` file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory. + + To start IBM Db2, run the following command: + + ```console + docker compose up -d db2 + ``` + +

Configure ScalarDB Cluster

+ + The **scalardb-cluster-node.properties** file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Please uncomment the properties for Db2 in the **scalardb-cluster-node.properties** file so that the configuration looks as follows: + + ```properties + # For Db2 + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:db2://db2-1:50000/sample + scalar.db.username=db2inst1 + scalar.db.password=db2inst1 + ``` + + +

Run Amazon DynamoDB Local

+ + You can run Amazon DynamoDB Local in Docker Compose by using the `docker-compose.yaml` file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory. + + To start Amazon DynamoDB Local, run the following command: + + ```console + docker compose up -d dynamodb + ``` + +

Configure ScalarDB Cluster

+ + The **scalardb-cluster-node.properties** file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Please uncomment the properties for Amazon DynamoDB Local in the **scalardb-cluster-node.properties** file so that the configuration looks as follows: + + ```properties + # For DynamoDB Local + scalar.db.storage=dynamo + scalar.db.contact_points=sample + scalar.db.username=sample + scalar.db.password=sample + scalar.db.dynamo.endpoint_override=http://dynamodb-1:8000 + ``` + + + To use Azure Cosmos DB for NoSQL, you must have an Azure account. If you don't have an Azure account, visit [Create an Azure Cosmos DB account](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/quickstart-portal#create-account). + +

Configure Cosmos DB for NoSQL

+ + Set the **default consistency level** to **Strong** according to the official document at [Configure the default consistency level](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-manage-consistency#configure-the-default-consistency-level). + +

Configure ScalarDB Cluster

+ + The following instructions assume that you have properly installed and configured the JDK in your local environment and properly configured your Cosmos DB for NoSQL account in Azure. + + The **scalardb-cluster-node.properties** file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Be sure to change the values for `scalar.db.contact_points` and `scalar.db.password` as described. + + ```properties + # For Cosmos DB + scalar.db.storage=cosmos + scalar.db.contact_points= + scalar.db.password= + ``` + +:::note + +You can use the primary key or the secondary key in your Azure Cosmos DB account as the value for `scalar.db.password`. + +::: + + +

Run Cassandra locally

+ + You can run Apache Cassandra in Docker Compose by using the `docker-compose.yaml` file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory. + + To start Apache Cassandra, run the following command: + + ```console + docker compose up -d cassandra + ``` + +

Configure ScalarDB Cluster

+ + The **scalardb-cluster-node.properties** file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Please uncomment the properties for Cassandra in the **scalardb-cluster-node.properties** file so that the configuration looks as follows: + + ```properties + # For Cassandra + scalar.db.storage=cassandra + scalar.db.contact_points=cassandra-1 + scalar.db.username=cassandra + scalar.db.password=cassandra + ``` + + + +For a comprehensive list of configurations for ScalarDB Cluster, see [ScalarDB Cluster Configurations](developer-guide-for-scalardb-cluster-with-java-api.mdx#client-configurations). + +## Set up ScalarDB Cluster in standalone mode + +To set up ScalarDB Cluster in standalone mode, you'll need to set a license key and then start ScalarDB Cluster. + +### Set the license key + +Set the license key (trial license or commercial license) for the ScalarDB Clusters in the properties file. For details, see [How to Configure a Product License Key](../scalar-licensing/index.mdx). + +### Start ScalarDB Cluster in standalone mode + +To start ScalarDB Cluster in standalone mode, run the following command: + +:::note + +If you want to change other configurations for ScalarDB Cluster, update the `scalardb-cluster-node.properties` file before running the command below. + +::: + +```console +docker compose up -d scalardb-cluster-node +``` + +## Create or import a schema + +ScalarDB has its own data model and schema that maps to the implementation-specific data model and schema. + +- **Need to create a database schema?** See [ScalarDB Schema Loader](../schema-loader.mdx). +- **Need to import an existing database?** See [Importing Existing Tables to ScalarDB by Using ScalarDB Schema Loader](../schema-loader-import.mdx). + +## Run transactions + +You can run transactions by using a one-phase or a two-phase commit interface. Select your method for running transactions. + +:::note + +If you are building a monolithic application, you should use the one-phase commit interface. However, if you are building a microservice application, see [ScalarDB Cluster Deployment Patterns for Microservices](./deployment-patterns-for-microservices.mdx) to decide which interface to use. + +::: + + + +

One-phase commit interface

+ + For details about how to run transactions by using a one-phase commit interface, see the [ScalarDB Java API Guide](../api-guide.mdx#transactional-api). + +:::note + +To try running transactions by using a one-phase commit interface, see the following sample tutorials: + +- [Create a Sample Application That Supports Multi-Storage Transactions](../scalardb-samples/multi-storage-transaction-sample/README.mdx) +- [Sample application of Spring Data JDBC for ScalarDB with Multi-storage Transactions](../scalardb-samples/spring-data-multi-storage-transaction-sample/README.mdx) + +::: + +

Two-phase commit interface

+ + For details about how to run transactions by using a two-phase commit interface, see [Transactions with a Two-Phase Commit Interface](../two-phase-commit-transactions.mdx). + +:::note + +To try running transactions by using a two-phase commit interface, see the following sample tutorials: + +- [Create a Sample Application That Supports Microservice Transactions](../scalardb-samples/microservice-transaction-sample/README.mdx) +- [Sample application of Spring Data JDBC for ScalarDB with Microservice Transactions](../scalardb-samples/spring-data-microservice-transaction-sample/README.mdx) + +::: + +

Learn more

+ + To learn more about running transactions by using ScalarDB Cluster, see the following: + + - [ScalarDB Cluster gRPC API Guide](scalardb-cluster-grpc-api-guide.mdx) + + +

One-phase commit interface

+ + For details about how to run transactions by using a one-phase commit interface, see the [Getting Started with Distributed Transactions in the ScalarDB Cluster .NET Client SDK](../scalardb-cluster-dotnet-client-sdk/getting-started-with-distributed-transactions.mdx). + +

Two-phase commit interface

+ + For details about how to run transactions by using a two-phase commit interface, see [Getting Started with Distributed Transactions with a Two-Phase Commit Interface in the ScalarDB Cluster .NET Client SDK](../scalardb-cluster-dotnet-client-sdk/getting-started-with-two-phase-commit-transactions.mdx). + + diff --git a/versioned_docs/version-3.X/scalardb-cluster/scalardb-abac-status-codes.mdx b/versioned_docs/version-3.X/scalardb-cluster/scalardb-abac-status-codes.mdx new file mode 100644 index 00000000..22aa089a --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster/scalardb-abac-status-codes.mdx @@ -0,0 +1,734 @@ +--- +tags: + - Enterprise Premium Option + - Private Preview +displayed_sidebar: docsEnglish +--- + +# Attribute-Based Access Control Error Codes + +This page provides a list of error codes related to attribute-based access control. + +## Error code classes and descriptions + +| Class | Description | +|:----------------|:----------------------------------------------| +| `DB-ABAC-1xxxx` | Errors for the user error category | +| `DB-ABAC-2xxxx` | Errors for the concurrency error category | +| `DB-ABAC-3xxxx` | Errors for the internal error category | + +## `DB-ABAC-1xxxx` status codes + +The following are status codes and messages for the user error category. + +### `DB-ABAC-10000` + +**Message** + +```markdown +The put operation is not supported. Use insert, upsert, or update instead +``` + +### `DB-ABAC-10001` + +**Message** + +```markdown +The default level must be less than or equal to the level. Default-level short name: %s; Default-level number: %d; Level short name: %s; Level number: %d +``` + +### `DB-ABAC-10002` + +**Message** + +```markdown +The row level must be less than or equal to the level. Row-level short name: %s; Row-level number: %d; Level short name: %s; Level number: %d +``` + +### `DB-ABAC-10003` + +**Message** + +```markdown +The operation does not have the target namespace or table name. Operation: %s +``` + +### `DB-ABAC-10004` + +**Message** + +```markdown +The policy does not exist. Policy: %s +``` + +### `DB-ABAC-10005` + +**Message** + +```markdown +The level does not exist. Policy: %s; Level short name: %s +``` + +### `DB-ABAC-10006` + +**Message** + +```markdown +The compartment does not exist. Policy: %s; Compartment short name: %s +``` + +### `DB-ABAC-10007` + +**Message** + +```markdown +The group does not exist. Policy: %s; Group short name: %s +``` + +### `DB-ABAC-10008` + +**Message** + +```markdown +The policy already exists. Policy: %s +``` + +### `DB-ABAC-10009` + +**Message** + +```markdown +The data tag column name is already in use in the policy %s. Policy: %s; Data tag column name: %s +``` + +### `DB-ABAC-10010` + +**Message** + +```markdown +The level already exists. Policy: %s; Level short name: %s +``` + +### `DB-ABAC-10011` + +**Message** + +```markdown +The compartment already exists. Policy: %s; Compartment short name: %s +``` + +### `DB-ABAC-10012` + +**Message** + +```markdown +The group already exists. Policy: %s; Group short name: %s +``` + +### `DB-ABAC-10013` + +**Message** + +```markdown +The parent group does not exist. Policy: %s; Group short name: %s; Parent-group short name: %s +``` + +### `DB-ABAC-10014` + +**Message** + +```markdown +The parent group is the same as the child group. Policy: %s; Group short name: %s; Parent-group short name: %s +``` + +### `DB-ABAC-10015` + +**Message** + +```markdown +The group has child groups. Policy: %s; Group short name: %s +``` + +### `DB-ABAC-10016` + +**Message** + +```markdown +The compartment cannot be a row compartment for read-only access mode. Policy: %s; User: %s; Compartment short name: %s +``` + +### `DB-ABAC-10017` + +**Message** + +```markdown +The compartment is already assigned to the user. Policy: %s; User: %s; Compartment short name: %s +``` + +### `DB-ABAC-10018` + +**Message** + +```markdown +The group cannot be a row group for read-only access mode. Policy: %s; User: %s; Group short name: %s +``` + +### `DB-ABAC-10019` + +**Message** + +```markdown +The group is already assigned to the user. Policy: %s; User: %s; Group short name: %s +``` + +### `DB-ABAC-10020` + +**Message** + +```markdown +The namespace does not exist. Namespace: %s +``` + +### `DB-ABAC-10021` + +**Message** + +```markdown +The namespace policy already exists. NamespacePolicy: %s +``` + +### `DB-ABAC-10022` + +**Message** + +```markdown +The namespace policy does not exist. NamespacePolicy: %s +``` + +### `DB-ABAC-10023` + +**Message** + +```markdown +The table does not exist. Table: %s +``` + +### `DB-ABAC-10024` + +**Message** + +```markdown +The table policy already exists. TablePolicy: %s +``` + +### `DB-ABAC-10025` + +**Message** + +```markdown +The table policy does not exist. TablePolicy: %s +``` + +### `DB-ABAC-10026` + +**Message** + +```markdown +The short name must not be empty. Short name: %s +``` + +### `DB-ABAC-10027` + +**Message** + +```markdown +The short name must be 30 characters or less. Short name: %s +``` + +### `DB-ABAC-10028` + +**Message** + +```markdown +The short name must not contain a colon. Short name: %s +``` + +### `DB-ABAC-10029` + +**Message** + +```markdown +The short name must not contain a comma. Short name: %s +``` + +### `DB-ABAC-10030` + +**Message** + +```markdown +The data tag is invalid. Data tag: %s +``` + +### `DB-ABAC-10031` + +**Message** + +```markdown +The level must be specified in the data tag. Data tag: %s +``` + +### `DB-ABAC-10032` + +**Message** + +```markdown +The user tag is invalid. User tag: %s +``` + +### `DB-ABAC-10033` + +**Message** + +```markdown +The level must be specified in the user tag. User tag: %s +``` + +### `DB-ABAC-10034` + +**Message** + +```markdown +The specified user tag is not allowed. User tag: %s +``` + +### `DB-ABAC-10035` + +**Message** + +```markdown +The data tag column type should be TEXT. Policy: %s; Data tag column: %s +``` + +### `DB-ABAC-10036` + +**Message** + +```markdown +The specified data tag is not allowed. Data tag: %s +``` + +### `DB-ABAC-10037` + +**Message** + +```markdown +The data tag column cannot be used in conditions. Operation: %s; Data tag column: %s +``` + +### `DB-ABAC-10038` + +**Message** + +```markdown +The operation is not allowed for the policy. Policy: %s; Operation: %s +``` + +### `DB-ABAC-10039` + +**Message** + +```markdown +Access denied: Invalid auth token +``` + +### `DB-ABAC-10040` + +**Message** + +```markdown +The authentication and authorization feature must be enabled to use the attribute-based access control feature +``` + +### `DB-ABAC-10041` + +**Message** + +```markdown +The single CRUD operation transaction manager does not support the attribute-based access control feature +``` + +### `DB-ABAC-10042` + +**Message** + +```markdown +The data tag column cannot be used in ordering. Operation: %s; Data tag column: %s +``` + +### `DB-ABAC-10043` + +**Message** + +```markdown +The namespace policy for the policy and namespace already exists. Policy: %s; Namespace: %s +``` + +### `DB-ABAC-10044` + +**Message** + +```markdown +The table policy for the policy and table already exists. Policy: %s; Table: %s +``` + +### `DB-ABAC-10045` + +**Message** + +```markdown +The user does not exist. Username: %s +``` + +## `DB-ABAC-2xxxx` status codes + +The following are status codes and messages for the concurrency error category. + +### `DB-ABAC-20000` + +**Message** + +```markdown +The record does not exist, so the %s condition is not satisfied +``` + +## `DB-ABAC-3xxxx` status codes + +The following are status codes and messages for the internal error category. + +### `DB-ABAC-30000` + +**Message** + +```markdown +Creating a policy failed. Policy: %s; Data tag column: %s +``` + +### `DB-ABAC-30001` + +**Message** + +```markdown +Enabling the policy failed. Policy: %s +``` + +### `DB-ABAC-30002` + +**Message** + +```markdown +Disabling the policy failed. Policy: %s +``` + +### `DB-ABAC-30003` + +**Message** + +```markdown +Getting the policy failed. Policy: %s +``` + +### `DB-ABAC-30004` + +**Message** + +```markdown +Getting the policies failed +``` + +### `DB-ABAC-30005` + +**Message** + +```markdown +Creating a level failed. Policy: %s; Level short name: %s; Level long name: %s; Level number: %d +``` + +### `DB-ABAC-30006` + +**Message** + +```markdown +Dropping the level failed. Policy: %s; Level short name: %s +``` + +### `DB-ABAC-30007` + +**Message** + +```markdown +Getting the level failed. Policy: %s; Level short name: %s +``` + +### `DB-ABAC-30008` + +**Message** + +```markdown +Getting the levels failed. Policy: %s +``` + +### `DB-ABAC-30009` + +**Message** + +```markdown +Creating a compartment failed. Policy: %s; Compartment short name: %s; Compartment long name: %s +``` + +### `DB-ABAC-30010` + +**Message** + +```markdown +Dropping the compartment failed. Policy: %s; Compartment short name: %s +``` + +### `DB-ABAC-30011` + +**Message** + +```markdown +Getting the compartment failed. Policy: %s; Compartment short name: %s +``` + +### `DB-ABAC-30012` + +**Message** + +```markdown +Getting the compartments failed. Policy: %s +``` + +### `DB-ABAC-30013` + +**Message** + +```markdown +Creating a group failed. Policy: %s; Group short name: %s; Group long name: %s; Parent-group short name: %s +``` + +### `DB-ABAC-30014` + +**Message** + +```markdown +Dropping the group failed. Policy: %s; Group short name: %s +``` + +### `DB-ABAC-30015` + +**Message** + +```markdown +Getting the group failed. Policy: %s; Group short name: %s +``` + +### `DB-ABAC-30016` + +**Message** + +```markdown +Getting groups failed. Policy: %s +``` + +### `DB-ABAC-30017` + +**Message** + +```markdown +Getting child groups failed. Policy: %s; Parent-group short name: %s +``` + +### `DB-ABAC-30018` + +**Message** + +```markdown +Setting the levels to the user failed. Policy: %s; User: %s; Level short name: %s; Default-level short name: %s; Row-level short name: %s +``` + +### `DB-ABAC-30019` + +**Message** + +```markdown +Adding the compartment to the user failed. Policy: %s; User: %s; Compartment short name: %s +``` + +### `DB-ABAC-30020` + +**Message** + +```markdown +Removing the compartment from the user failed. Policy: %s; User: %s; Compartment short name: %s +``` + +### `DB-ABAC-30021` + +**Message** + +```markdown +Adding the group to the user failed. Policy: %s; User: %s; Group short name: %s +``` + +### `DB-ABAC-30022` + +**Message** + +```markdown +Removing the group from the user failed. Policy: %s; User: %s; Group short name: %s +``` + +### `DB-ABAC-30023` + +**Message** + +```markdown +Dropping the user tag information from the user failed. Policy: %s; User: %s +``` + +### `DB-ABAC-30024` + +**Message** + +```markdown +Getting the user tag information failed. Policy: %s; User: %s +``` + +### `DB-ABAC-30025` + +**Message** + +```markdown +Creating a namespace policy failed. NamespacePolicy: %s; Policy: %s; Namespace: %s +``` + +### `DB-ABAC-30026` + +**Message** + +```markdown +Enabling the namespace policy failed. NamespacePolicy: %s +``` + +### `DB-ABAC-30027` + +**Message** + +```markdown +Disabling the namespace policy failed. NamespacePolicy: %s +``` + +### `DB-ABAC-30028` + +**Message** + +```markdown +Removing the namespace policies assigned to the namespace failed. Namespace: %s +``` + +### `DB-ABAC-30029` + +**Message** + +```markdown +Getting the namespace policies failed +``` + +### `DB-ABAC-30030` + +**Message** + +```markdown +Creating a table policy failed. TablePolicy: %s; Policy: %s; Table: %s +``` + +### `DB-ABAC-30031` + +**Message** + +```markdown +Enabling the table policy failed. TablePolicy: %s +``` + +### `DB-ABAC-30032` + +**Message** + +```markdown +Disabling the table policy failed. TablePolicy: %s +``` + +### `DB-ABAC-30033` + +**Message** + +```markdown +Removing the table policies assigned to the table failed. Table: %s +``` + +### `DB-ABAC-30034` + +**Message** + +```markdown +Getting the table policies failed +``` + +### `DB-ABAC-30035` + +**Message** + +```markdown +Getting the policies assigned to the namespace failed. Namespace: %s +``` + +### `DB-ABAC-30036` + +**Message** + +```markdown +Getting the policies assigned to the table failed. Table: %s +``` + +### `DB-ABAC-30037` + +**Message** + +```markdown +Registering the data tag failed. Policy: %s; Data tag: %s +``` + +### `DB-ABAC-30038` + +**Message** + +```markdown +Getting the data tags failed. Policy: %s +``` + +### `DB-ABAC-30039` + +**Message** + +```markdown +Getting the namespace policy failed. NamespacePolicy: %s +``` + +### `DB-ABAC-30040` + +**Message** + +```markdown +Getting the table policy failed. TablePolicy: %s +``` diff --git a/versioned_docs/version-3.X/scalardb-cluster/scalardb-auth-status-codes.mdx b/versioned_docs/version-3.X/scalardb-cluster/scalardb-auth-status-codes.mdx new file mode 100644 index 00000000..00ef8e73 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster/scalardb-auth-status-codes.mdx @@ -0,0 +1,305 @@ +--- +tags: + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Authentication and Authorization Error Codes + +This page provides a list of error codes related to authentication and authorization. + +## Error code classes and descriptions + +| Class | Description | +|:----------------|:---------------------------------------| +| `DB-AUTH-1xxxx` | Errors for the user error category | +| `DB-AUTH-3xxxx` | Errors for the internal error category | + +## `DB-AUTH-1xxxx` status codes + +The following are status codes and messages for the user error category. + +### `DB-AUTH-10000` + +**Message** + +```markdown +The user already exists. Username: %s +``` + +### `DB-AUTH-10001` + +**Message** + +```markdown +The user does not exist. Username: %s +``` + +### `DB-AUTH-10003` + +**Message** + +```markdown +The namespace does not exist. Namespace: %s +``` + +### `DB-AUTH-10004` + +**Message** + +```markdown +The table does not exist. Table: %s +``` + +### `DB-AUTH-10005` + +**Message** + +```markdown +Invalid username or password +``` + +### `DB-AUTH-10006` + +**Message** + +```markdown +Access denied: Invalid auth token +``` + +### `DB-AUTH-10007` + +**Message** + +```markdown +Access denied: You need the %s privilege on the namespace %s to execute this operation +``` + +### `DB-AUTH-10008` + +**Message** + +```markdown +Access denied: You need the %s privilege on the table %s to execute this operation +``` + +### `DB-AUTH-10009` + +**Message** + +```markdown +Access denied: You must be a superuser to execute this operation +``` + +### `DB-AUTH-10010` + +**Message** + +```markdown +Access denied: You can't access information about the user %s +``` + +### `DB-AUTH-10011` + +**Message** + +```markdown +Access denied: You can't alter the user %s +``` + +### `DB-AUTH-10012` + +**Message** + +```markdown +Access denied: You must be a superuser to change the SUPERUSER attribute +``` + +### `DB-AUTH-10013` + +**Message** + +```markdown +You can't change the SUPERUSER attribute for the current user %s +``` + +### `DB-AUTH-10014` + +**Message** + +```markdown +You can't drop the current user %s +``` + +### `DB-AUTH-10015` + +**Message** + +```markdown +Access denied: You can't grant the %s privilege because you don't have the same privilege on the table %s +``` + +### `DB-AUTH-10016` + +**Message** + +```markdown +Access denied: You can't grant the %s privilege because you don't have the same privilege on the namespace %s +``` + +### `DB-AUTH-10017` + +**Message** + +```markdown +Access denied: You can't revoke the %s privilege because you don't have the same privilege on the table %s +``` + +### `DB-AUTH-10018` + +**Message** + +```markdown +Access denied: You can't revoke the %s privilege because you don't have the same privilege on the namespace %s +``` + +### `DB-AUTH-10019` + +**Message** + +```markdown +The operation does not have the target namespace or table name. Operation: %s +``` + +## `DB-AUTH-3xxxx` status codes + +The following are status codes and messages for the internal error category. + +### `DB-AUTH-30000` + +**Message** + +```markdown +Getting auth token information failed +``` + +### `DB-AUTH-30001` + +**Message** + +```markdown +Getting the user failed. Username: %s +``` + +### `DB-AUTH-30002` + +**Message** + +```markdown +Creating a user failed. Username: %s +``` + +### `DB-AUTH-30003` + +**Message** + +```markdown +Altering the user failed. Username: %s +``` + +### `DB-AUTH-30004` + +**Message** + +```markdown +Dropping the user failed. Username: %s +``` + +### `DB-AUTH-30005` + +**Message** + +```markdown +Granting privileges failed. Username: %s; Namespace: %s; Privileges: %s +``` + +### `DB-AUTH-30006` + +**Message** + +```markdown +Granting privileges failed. Username: %s; Table: %s; Privileges: %s +``` + +### `DB-AUTH-30007` + +**Message** + +```markdown +Revoking privileges failed. Username: %s; Namespace: %s; Privileges: %s +``` + +### `DB-AUTH-30008` + +**Message** + +```markdown +Revoking privileges failed. Username: %s; Table: %s; Privileges: %s +``` + +### `DB-AUTH-30009` + +**Message** + +```markdown +Getting users failed +``` + +### `DB-AUTH-30010` + +**Message** + +```markdown +Getting privileges failed. Username: %s; Namespace: %s +``` + +### `DB-AUTH-30011` + +**Message** + +```markdown +Getting privileges failed. Username: %s; Table: %s +``` + +### `DB-AUTH-30012` + +**Message** + +```markdown +Deleting privileges failed. Namespace: %s +``` + +### `DB-AUTH-30013` + +**Message** + +```markdown +Deleting privileges failed. Table: %s +``` + +### `DB-AUTH-30014` + +**Message** + +```markdown +Logging in failed. Username: %s +``` + +### `DB-AUTH-30015` + +**Message** + +```markdown +Logging out failed +``` diff --git a/versioned_docs/version-3.X/scalardb-cluster/scalardb-auth-with-sql.mdx b/versioned_docs/version-3.X/scalardb-cluster/scalardb-auth-with-sql.mdx new file mode 100644 index 00000000..953e97a8 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster/scalardb-auth-with-sql.mdx @@ -0,0 +1,379 @@ +--- +tags: + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Authenticate and Authorize Users + +import JavadocLink from '/src/theme/JavadocLink.js'; +import WarningLicenseKeyContact from '/src/components/en-us/_warning-license-key-contact.mdx'; + +ScalarDB Cluster has a mechanism to authenticate and authorize users. + +This guide describes how to use authentication and authorization in ScalarDB SQL. + +:::tip + +You can also do authentication and authorization by using the primitive interface. For details, see , which implements . + +::: + +## Overview + +By using authentication and authorization, you can create users and grant or revoke their privileges. You can create a user by using the `CREATE USER` command, and you can grant or revoke one's privileges on a table or a namespace by using the `GRANT` or `REVOKE` command, respectively. For details about such data control language (DCL) commands, see [DCL](../scalardb-sql/grammar.mdx#dcl). + +Users can log in to ScalarDB Cluster with a username and a password and execute SQL statements if they have the required privileges. + +Authentication and authorization support two types of users: + +- **Superusers:** This type of user has all privileges. Only superusers can create or drop other users and namespaces. +- **Normal users:** This type of user initially doesn't have any privileges, so they need to be granted privileges by a superuser or another user who has the `GRANT` privilege. + +The following privileges are available when using authentication and authorization: + +- `SELECT` +- `INSERT` +- `UPDATE` +- `DELETE` +- `CREATE` +- `DROP` +- `TRUNCATE` +- `ALTER` +- `GRANT` + +For details about privileges, see [Which privileges are required for each type of operation](#which-privileges-are-required-for-each-type-of-operation). + +## Configurations + +This section describes the available configurations for authentication and authorization. + +### ScalarDB Cluster node configurations + +To enable authentication and authorization, you need to set `scalar.db.cluster.auth.enabled` to `true`. + +| Name | Description | Default | +|----------------------------------|------------------------------------|---------| +| `scalar.db.cluster.auth.enabled` | Whether authentication and authorization are enabled. | `false` | + +You can also set the following configurations: + +| Name | Description | Default | +|----------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------|--------------------| +| `scalar.db.cluster.auth.cache_expiration_time_millis` | Cache expiration time for authentication and authorization information in milliseconds. | `60000` (1 minute) | +| `scalar.db.cluster.auth.auth_token_expiration_time_minutes` | Authentication and authorization token expiration time in minutes. | `1440` (1 day) | +| `scalar.db.cluster.auth.auth_token_gc_thread_interval_minutes` | Authentication and authorization token garbage collection (GC) thread interval in minutes. | `360` (6 hours) | +| `scalar.db.cluster.auth.pepper` | A secret value added to a password before hashing. If not specified, the password is hashed without pepper. | | + +:::note + +If you enable authentication and authorization, you will also need to set `scalar.db.cross_partition_scan.enabled` to `true` for the system namespace (`scalardb` by default) because authentication and authorization perform cross-partition scans internally. + +::: + +### ScalarDB Cluster Java client SDK configurations + +To enable authentication and authorization on the client side, you need to set `scalar.db.cluster.auth.enabled` to `true`. + +| Name | Description | Default | +|----------------------------------|-----------------------------------|---------| +| `scalar.db.cluster.auth.enabled` | Whether authentication and authorization are enabled. | `false` | + +In addition to the configuration in the [ScalarDB Cluster SQL client configurations](developer-guide-for-scalardb-cluster-with-java-api.mdx#scalardb-cluster-sql-client-configurations) section, you also need to set `scalar.db.sql.cluster_mode.username` and `scalar.db.sql.cluster_mode.password` to specify the username and password of the client. + +| Name | Description | Default | +|---------------------------------------|-----------------------------|---------| +| `scalar.db.sql.cluster_mode.username` | The username of the client. | | +| `scalar.db.sql.cluster_mode.password` | The password of the client. | | + +## Initial user + +When you enable authentication and authorization, the initial user `admin` is created and the initial password of that user is `admin`. This user is a superuser and has all privileges. You can log in with this user and create other users if necessary. + +:::warning + +For security purposes, be sure to change the password of the initial user, especially before deploying to a production environment. + +::: + +## Which privileges are required for each type of operation + +The following tables show which privileges are required for each type of operation: + +### DDL + +| Command | Superuser required | Required privileges | +|-------------------------------|--------------------|---------------------| +| `CREATE NAMESPACE` | `true` | | +| `DROP NAMESPACE` | `true` | | +| `CREATE TABLE` | | `CREATE` | +| `DROP TABLE` | | `DROP` | +| `CREATE INDEX` | | `CREATE` | +| `DROP INDEX` | | `DROP` | +| `TRUNCATE TABLE` | | `TRUNCATE` | +| `ALTER TABLE` | | `ALTER` | +| `CREATE COORDINATOR TABLES` | `true` | | +| `DROP COORDINATOR TABLES` | `true` | | +| `TRUNCATE COORDINATOR TABLES` | `true` | | + +### DML + +| Command | Superuser required | Required privileges | +|----------|--------------------|-----------------------| +| `SELECT` | | `SELECT` | +| `INSERT` | | `INSERT` | +| `UPSERT` | | `INSERT` | +| `UPDATE` | | `SELECT` and `UPDATE` | +| `DELETE` | | `SELECT` and `DELETE` | + +### DCL + +| Command | Superuser required | Required privileges | +|---------------|-----------------------------------------------|----------------------------------------------------------------| +| `CREATE USER` | `true` | | +| `ALTER USER` | `true` (Users can change their own password.) | | +| `DROP USER` | `true` | | +| `GRANT` | | `GRANT` (Users can grant only the privileges that they have.) | +| `REVOKE` | | `GRANT` (Users can revoke only the privileges that they have.) | + +## Limitations + +There are some limitations to the privileges granted or revoked in authentication and authorization: + +- You must grant or revoke `INSERT` and `UPDATE` privileges together. +- To grant a user the `UPDATE` or `DELETE` privilege, the target user must have the `SELECT` privilege. +- If the target user has the `INSERT` or `UPDATE` privilege, you cannot revoke the `SELECT` privilege from them. + +## Wire encryption + +If you enable authentication and authorization, enabling wire encryption to protect the user credentials is strongly recommended, especially in production environments. For details about wire encryption, see [Encrypt Wire Communications](encrypt-wire-communications.mdx). + +## Tutorial - Authenticate and authorize users + +This tutorial explains how to use authentication and authorization. + +### Prerequisites + +- OpenJDK LTS version (8, 11, 17, or 21) from [Eclipse Temurin](https://adoptium.net/temurin/releases/) +- [Docker](https://www.docker.com/get-started/) 20.10 or later with [Docker Compose](https://docs.docker.com/compose/install/) V2 or later + +:::note + +This tutorial has been tested with OpenJDK from Eclipse Temurin. ScalarDB itself, however, has been tested with JDK distributions from various vendors. For details about the requirements for ScalarDB, including compatible JDK distributions, please see [Requirements](../requirements.mdx). + +::: + + + +### 1. Create the ScalarDB Cluster configuration file + +Create the following configuration file as `scalardb-cluster-node.properties`, replacing `` and `` with your ScalarDB license key and license check certificate values. For more information about the license key and certificate, see [How to Configure a Product License Key](../scalar-licensing/index.mdx). + +```properties +scalar.db.storage=jdbc +scalar.db.contact_points=jdbc:postgresql://postgresql:5432/postgres +scalar.db.username=postgres +scalar.db.password=postgres +scalar.db.cluster.node.standalone_mode.enabled=true +scalar.db.cross_partition_scan.enabled=true +scalar.db.sql.enabled=true + +# Enable authentication and authorization +scalar.db.cluster.auth.enabled=true + +# License key configurations +scalar.db.cluster.node.licensing.license_key= +scalar.db.cluster.node.licensing.license_check_cert_pem= +``` + +### 2. Create the Docker Compose file + +Create the following configuration file as `docker-compose.yaml`. + +```yaml +services: + postgresql: + container_name: "postgresql" + image: "postgres:15" + ports: + - 5432:5432 + environment: + - POSTGRES_PASSWORD=postgres + healthcheck: + test: ["CMD-SHELL", "pg_isready || exit 1"] + interval: 1s + timeout: 10s + retries: 60 + start_period: 30s + + scalardb-cluster-standalone: + container_name: "scalardb-cluster-node" + image: "ghcr.io/scalar-labs/scalardb-cluster-node-byol-premium:3.16.0" + ports: + - 60053:60053 + - 9080:9080 + volumes: + - ./scalardb-cluster-node.properties:/scalardb-cluster/node/scalardb-cluster-node.properties + depends_on: + postgresql: + condition: service_healthy +``` + +### 3. Start PostgreSQL and ScalarDB Cluster + +Run the following command to start PostgreSQL and ScalarDB Cluster in standalone mode. + +```console +docker compose up -d +``` + +It may take a few minutes for ScalarDB Cluster to fully start. + +### 4. Connect to ScalarDB Cluster + +To connect to ScalarDB Cluster, this tutorial uses the SQL CLI, a tool for connecting to ScalarDB Cluster and executing SQL queries. You can download the SQL CLI from the [ScalarDB releases page](https://github.com/scalar-labs/scalardb/releases). + +Create a configuration file named `scalardb-cluster-sql-cli.properties`. This file will be used to connect to ScalarDB Cluster by using the SQL CLI. + +```properties +scalar.db.sql.connection_mode=cluster +scalar.db.sql.cluster_mode.contact_points=indirect:localhost + +# Enable authentication and authorization +scalar.db.cluster.auth.enabled=true +``` + +Then, start the SQL CLI by running the following command. + +```console +java -jar scalardb-cluster-sql-cli-3.16.0-all.jar --config scalardb-cluster-sql-cli.properties +``` + +Enter the username and password as `admin` and `admin`, respectively. + +Now you're ready to use the database with authentication and authorization enabled in ScalarDB Cluster. + +### 5. Create namespaces and a table + +Create namespaces. + +```sql +CREATE NAMESPACE ns1; + +CREATE NAMESPACE ns2; +``` + +Next, create a table in the `ns1` namespaces. + +```sql +CREATE TABLE ns1.tbl ( + id INT PRIMARY KEY, + col1 TEXT, + col2 INT); +``` + +### 6. Create a user + +Create a user named `user1`. + +```sql +CREATE USER user1 WITH PASSWORD 'user1'; +``` + +To check the user, run the following command. + +```sql +SHOW USERS; +``` + +```console ++----------+-------------+ +| username | isSuperuser | ++----------+-------------+ +| user1 | false | +| admin | true | ++----------+-------------+ +``` + +You can see that the `user1` user has been created. + +### 7. Grant privileges + +Grant the `SELECT`, `INSERT`, and `UPDATE` privileges to `user1` on the `ns1.tbl` table. + +```sql +GRANT SELECT, INSERT, UPDATE ON ns1.tbl TO user1; +``` + +Then, grant the `SELECT` privilege to `user1` on the `ns2` namespace. + +```sql +GRANT SELECT ON NAMESPACE ns2 TO user1; +``` + +To check the privileges, run the following command. + +```sql +SHOW GRANTS FOR user1; +``` + +```console ++---------+-----------+-----------+ +| name | type | privilege | ++---------+-----------+-----------+ +| ns2 | NAMESPACE | SELECT | +| ns1.tbl | TABLE | SELECT | +| ns1.tbl | TABLE | INSERT | +| ns1.tbl | TABLE | UPDATE | ++---------+-----------+-----------+ +``` + +You can see that `user1` has been granted the `SELECT`, `INSERT`, and `UPDATE` privileges on the `ns.tbl` table, and the `SELECT` privilege on the `ns2` namespace. + +### 8. Log in as `user1` + +Log in as `user1` and execute SQL statements. + +```console +java -jar scalardb-cluster-sql-cli-3.16.0-all.jar --config scalardb-cluster-sql-cli.properties +``` + +Enter the username and password as `user1` and `user1`, respectively. + +Now you can execute SQL statements as `user1`. + +### 9. Execute DML statements + +Execute the following `INSERT` statement as `user1`. + +```sql +INSERT INTO ns1.tbl VALUES (1, 'a', 1); +``` + +Then, execute the following `SELECT` statement as `user1`. + +```sql +SELECT * FROM ns1.tbl; +``` + +```console ++----+------+------+ +| id | col1 | col2 | ++----+------+------+ +| 1 | a | 1 | ++----+------+------+ +``` + +You can see that `user1` can execute `INSERT` and `SELECT` statements. + +Next, try executing the following `DELETE` statement as `user1`. + +```sql +DELETE FROM ns1.tbl WHERE id = 1; +``` + +```console +Error: Authorization error (PERMISSION_DENIED: SQL-10021: Access denied: You need the DELETE privilege on the table ns1.tbl to execute this operation) (state=SDB11,code=9911) +``` + +You will see the above error message because `user1` doesn't have the `DELETE` privilege on the `ns1.tbl` table. diff --git a/versioned_docs/version-3.X/scalardb-cluster/scalardb-cluster-configurations.mdx b/versioned_docs/version-3.X/scalardb-cluster/scalardb-cluster-configurations.mdx new file mode 100644 index 00000000..4575bad3 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster/scalardb-cluster-configurations.mdx @@ -0,0 +1,326 @@ +--- +tags: + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# ScalarDB Cluster Configurations + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +This document describes the configurations for ScalarDB Cluster. ScalarDB Cluster consists of multiple cluster nodes, each of which needs to be configured. The configurations need to be specified in the properties file. + +## Cluster configurations + +This section describes the configurations for ScalarDB Cluster. + +### General configurations + +The following general configurations are available for ScalarDB Cluster. + +#### Transaction management configurations + +| Name | Description | Default. | +|-------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------------------| +| `scalar.db.transaction_manager` | Transaction manager of ScalarDB. Specify `consensus-commit` to use [Consensus Commit](../consensus-commit.mdx) or `single-crud-operation` to [run non-transactional storage operations](./run-non-transactional-storage-operations-through-scalardb-cluster.mdx). Note that the configurations under the `scalar.db.consensus_commit` prefix are ignored if you use `single-crud-operation`. | `consensus-commit` | +| `scalar.db.consensus_commit.isolation_level` | Isolation level used for Consensus Commit. Either `SNAPSHOT`, `SERIALIZABLE`, or `READ_COMMITTED` can be specified. | `SNAPSHOT` | +| `scalar.db.consensus_commit.coordinator.namespace` | Namespace name of Coordinator tables. | `coordinator` | +| `scalar.db.consensus_commit.include_metadata.enabled` | If set to `true`, `Get` and `Scan` operations results will contain transaction metadata. To see the transaction metadata columns details for a given table, you can use the `DistributedTransactionAdmin.getTableMetadata()` method, which will return the table metadata augmented with the transaction metadata columns. Using this configuration can be useful to investigate transaction-related issues. | `false` | + +#### Node configurations + +| Name | Description 　 | Default | +|--------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------| +| `scalar.db.cluster.membership.type` | Membership type. Currently, only `KUBERNETES` can be specified. | `KUBERNETES` | +| `scalar.db.cluster.membership.kubernetes.endpoint.namespace_name` | This configuration is for the `KUBERNETES` membership type. Namespace name for the [endpoint resource](https://kubernetes.io/docs/concepts/services-networking/service/#endpoints). | `default` | +| `scalar.db.cluster.membership.kubernetes.endpoint.name` | This configuration is for the `KUBERNETES` membership type. Name of the [endpoint resource](https://kubernetes.io/docs/concepts/services-networking/service/#endpoints) to get the membership info. | | +| `scalar.db.cluster.node.decommissioning_duration_secs` | Decommissioning duration in seconds. | `30` | +| `scalar.db.cluster.node.grpc.max_inbound_message_size` | Maximum message size allowed to be received. | The gRPC default value | +| `scalar.db.cluster.node.grpc.max_inbound_metadata_size` | Maximum size of metadata allowed to be received. | The gRPC default value | +| `scalar.db.cluster.node.port` | Port number of the ScalarDB Cluster node. | `60053` | +| `scalar.db.cluster.node.prometheus_exporter_port` | Port number of the Prometheus exporter. | `9080` | +| `scalar.db.cluster.grpc.deadline_duration_millis` | Deadline duration for gRPC in milliseconds. | `60000` (60 seconds) | +| `scalar.db.cluster.node.standalone_mode.enabled` | Whether standalone mode is enabled. Note that if standalone mode is enabled, the membership configurations (`scalar.db.cluster.membership.*`) will be ignored. | `false` | +| `scalar.db.metadata.cache_expiration_time_secs` | ScalarDB has a metadata cache to reduce the number of requests to the database. This setting specifies the expiration time of the cache in seconds. If you specify `-1`, the cache will never expire. | `60` | +| `scalar.db.active_transaction_management.expiration_time_millis` | ScalarDB Cluster nodes maintain in-progress transactions, which can be resumed by using a transaction ID. This process expires transactions that have been idle for an extended period to prevent resource leaks. This configuration specifies the expiration time of this transaction management feature in milliseconds. | `60000` (60 seconds) | +| `scalar.db.system_namespace_name` | The given namespace name will be used by ScalarDB internally. | `scalardb` | +| `scalar.db.transaction.enabled` | Whether the transaction feature is enabled. For example, if you use only the embedding feature, you can set this property to `false`. | `true` | +| `scalar.db.cluster.node.scanner_management.expiration_time_millis` | ScalarDB Cluster nodes maintain in-progress scanners. This process expires scanners that have been idle for an extended period to prevent resource leaks. This configuration specifies the expiration time of this scanner management feature in milliseconds. | `60000` (60 seconds) | + +### Performance-related configurations + +The following performance-related configurations are available for the Consensus Commit transaction manager: + +| Name | Description | Default | +|----------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------| +| `scalar.db.consensus_commit.parallel_executor_count` | Number of executors (threads) for parallel execution. This number refers to the total number of threads across transactions in a ScalarDB Cluster node or a ScalarDB process. | `128` | +| `scalar.db.consensus_commit.parallel_preparation.enabled` | Whether or not the preparation phase is executed in parallel. | `true` | +| `scalar.db.consensus_commit.parallel_validation.enabled` | Whether or not the validation phase (in `EXTRA_READ`) is executed in parallel. | The value of `scalar.db.consensus_commit.parallel_commit.enabled` | +| `scalar.db.consensus_commit.parallel_commit.enabled` | Whether or not the commit phase is executed in parallel. | `true` | +| `scalar.db.consensus_commit.parallel_rollback.enabled` | Whether or not the rollback phase is executed in parallel. | The value of `scalar.db.consensus_commit.parallel_commit.enabled` | +| `scalar.db.consensus_commit.async_commit.enabled` | Whether or not the commit phase is executed asynchronously. | `false` | +| `scalar.db.consensus_commit.async_rollback.enabled` | Whether or not the rollback phase is executed asynchronously. | The value of `scalar.db.consensus_commit.async_commit.enabled` | +| `scalar.db.consensus_commit.parallel_implicit_pre_read.enabled` | Whether or not implicit pre-read is executed in parallel. | `true` | +| `scalar.db.consensus_commit.one_phase_commit.enabled` | Whether or not the one-phase commit optimization is enabled. | `false` | +| `scalar.db.consensus_commit.coordinator.write_omission_on_read_only.enabled` | Whether or not the write omission optimization is enabled for read-only transactions. This optimization is useful for read-only transactions that do not modify any data, as it avoids unnecessary writes to the Coordinator tables. | `true` | +| `scalar.db.consensus_commit.coordinator.group_commit.enabled` | Whether or not committing the transaction state is executed in batch mode. This feature can't be used with a two-phase commit interface. | `false` | +| `scalar.db.consensus_commit.coordinator.group_commit.slot_capacity` | Maximum number of slots in a group for the group commit feature. A large value improves the efficiency of group commit, but may also increase latency and the likelihood of transaction conflicts.[^1] | `20` | +| `scalar.db.consensus_commit.coordinator.group_commit.group_size_fix_timeout_millis` | Timeout to fix the size of slots in a group. A large value improves the efficiency of group commit, but may also increase latency and the likelihood of transaction conflicts.[^1] | `40` | +| `scalar.db.consensus_commit.coordinator.group_commit.delayed_slot_move_timeout_millis` | Timeout to move delayed slots from a group to another isolated group to prevent the original group from being affected by delayed transactions. A large value improves the efficiency of group commit, but may also increase the latency and the likelihood of transaction conflicts.[^1] | `1200` | +| `scalar.db.consensus_commit.coordinator.group_commit.old_group_abort_timeout_millis` | Timeout to abort an old ongoing group. A small value reduces resource consumption through aggressive aborts, but may also increase the likelihood of unnecessary aborts for long-running transactions. | `60000` | +| `scalar.db.consensus_commit.coordinator.group_commit.timeout_check_interval_millis` | Interval for checking the group commit–related timeouts. | `20` | +| `scalar.db.consensus_commit.coordinator.group_commit.metrics_monitor_log_enabled` | Whether or not the metrics of the group commit are logged periodically. | `false` | + +### Storage-related configurations + +ScalarDB has a storage (database) abstraction layer that supports multiple storage implementations. You can specify the storage implementation by using the `scalar.db.storage` property. + +Select a database to see the configurations available for each storage. + + + + The following configurations are available for JDBC databases: + + | Name | Description | Default | + |-----------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------| + | `scalar.db.storage` | `jdbc` must be specified. | - | + | `scalar.db.contact_points` | JDBC connection URL. | | + | `scalar.db.username` | Username to access the database. | | + | `scalar.db.password` | Password to access the database. | | + | `scalar.db.jdbc.connection_pool.min_idle` | Minimum number of idle connections in the connection pool. | `20` | + | `scalar.db.jdbc.connection_pool.max_idle` | Maximum number of connections that can remain idle in the connection pool. | `50` | + | `scalar.db.jdbc.connection_pool.max_total` | Maximum total number of idle and borrowed connections that can be active at the same time for the connection pool. Use a negative value for no limit. | `100` | + | `scalar.db.jdbc.prepared_statements_pool.enabled` | Setting this property to `true` enables prepared-statement pooling. | `false` | + | `scalar.db.jdbc.prepared_statements_pool.max_open` | Maximum number of open statements that can be allocated from the statement pool at the same time. Use a negative value for no limit. | `-1` | + | `scalar.db.jdbc.isolation_level` | Isolation level for JDBC. `READ_UNCOMMITTED`, `READ_COMMITTED`, `REPEATABLE_READ`, or `SERIALIZABLE` can be specified. | Underlying-database specific | + | `scalar.db.jdbc.table_metadata.connection_pool.min_idle` | Minimum number of idle connections in the connection pool for the table metadata. | `5` | + | `scalar.db.jdbc.table_metadata.connection_pool.max_idle` | Maximum number of connections that can remain idle in the connection pool for the table metadata. | `10` | + | `scalar.db.jdbc.table_metadata.connection_pool.max_total` | Maximum total number of idle and borrowed connections that can be active at the same time for the connection pool for the table metadata. Use a negative value for no limit. | `25` | + | `scalar.db.jdbc.admin.connection_pool.min_idle` | Minimum number of idle connections in the connection pool for admin. | `5` | + | `scalar.db.jdbc.admin.connection_pool.max_idle` | Maximum number of connections that can remain idle in the connection pool for admin. | `10` | + | `scalar.db.jdbc.admin.connection_pool.max_total` | Maximum total number of idle and borrowed connections that can be active at the same time for the connection pool for admin. Use a negative value for no limit. | `25` | + | `scalar.db.jdbc.db2.variable_key_column_size` | Column size for TEXT and BLOB columns in IBM Db2 when they are used as a primary key or secondary key. Minimum 64 bytes. | `128` | + | `scalar.db.jdbc.db2.time_column.default_date_component` | Value of the date component used for storing `TIME` data in IBM Db2. Since the IBM Db2 TIMESTAMP type is used to store ScalarDB `TIME` type data because it provides fractional-second precision, ScalarDB stores `TIME` data with the same date component value for ease of comparison and sorting. | `1970-01-01` | +:::note + +If you're using SQLite3 as a JDBC database, you must set `scalar.db.contact_points` as follows: + +```properties +scalar.db.contact_points=jdbc:sqlite:?busy_timeout=10000 +``` + +Unlike other JDBC databases, [SQLite3 doesn't fully support concurrent access](https://www.sqlite.org/lang_transaction.html). +To avoid frequent errors caused internally by [`SQLITE_BUSY`](https://www.sqlite.org/rescode.html#busy), we recommend setting a [`busy_timeout`](https://www.sqlite.org/c3ref/busy_timeout.html) parameter. + +::: + + +The following configurations are available for DynamoDB: + + | Name | Description | Default | + |---------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------| + | `scalar.db.storage` | `dynamo` must be specified. | - | + | `scalar.db.contact_points` | AWS region with which ScalarDB should communicate (e.g., `us-east-1`). | | + | `scalar.db.username` | AWS access key used to identify the user interacting with AWS. | | + | `scalar.db.password` | AWS secret access key used to authenticate the user interacting with AWS. | | + | `scalar.db.dynamo.endpoint_override` | Amazon DynamoDB endpoint with which ScalarDB should communicate. This is primarily used for testing with a local instance instead of an AWS service. | | + | `scalar.db.dynamo.namespace.prefix` | Prefix for the user namespaces and metadata namespace names. Since AWS requires having unique tables names in a single AWS region, this is useful if you want to use multiple ScalarDB environments (development, production, etc.) in a single AWS region. | | + + + The following configurations are available for CosmosDB for NoSQL: + + | Name | Description | Default | + |--------------------------------------|----------------------------------------------------------------------------------------------------------|----------| + | `scalar.db.storage` | `cosmos` must be specified. | - | + | `scalar.db.contact_points` | Azure Cosmos DB for NoSQL endpoint with which ScalarDB should communicate. | | + | `scalar.db.password` | Either a master or read-only key used to perform authentication for accessing Azure Cosmos DB for NoSQL. | | + | `scalar.db.cosmos.consistency_level` | Consistency level used for Cosmos DB operations. `STRONG` or `BOUNDED_STALENESS` can be specified. | `STRONG` | + + + The following configurations are available for Cassandra: + + | Name | Description | Default | + |-----------------------------------------|-----------------------------------------------------------------------|------------| + | `scalar.db.storage` | `cassandra` must be specified. | - | + | `scalar.db.contact_points` | Comma-separated contact points. | | + | `scalar.db.contact_port` | Port number for all the contact points. | | + | `scalar.db.username` | Username to access the database. | | + | `scalar.db.password` | Password to access the database. | | + + + +#### Multi-storage support + +ScalarDB supports using multiple storage implementations simultaneously. You can use multiple storages by specifying `multi-storage` as the value for the `scalar.db.storage` property. + +For details about using multiple storages, see [Multi-Storage Transactions](../multi-storage-transactions.mdx). + +#### Cross-partition scan configurations + +By enabling the cross-partition scan option as described below, the `Scan` operation can retrieve all records across partitions. In addition, you can specify arbitrary conditions and orderings in the cross-partition `Scan` operation by enabling `cross_partition_scan.filtering` and `cross_partition_scan.ordering`, respectively. Currently, the cross-partition scan with ordering option is available only for JDBC databases. To enable filtering and ordering, `scalar.db.cross_partition_scan.enabled` must be set to `true`. + +For details on how to use cross-partition scan, see [Scan operation](../api-guide.mdx#scan-operation). + +:::warning + +For non-JDBC databases, we do not recommend enabling cross-partition scan with the `SERIALIAZABLE` isolation level because transactions could be executed at a lower isolation level (that is, `SNAPSHOT`). When using non-JDBC databases, use cross-partition scan at your own risk only if consistency does not matter for your transactions. + +::: + +| Name | Description | Default | +|----------------------------------------------------|-----------------------------------------------|---------| +| `scalar.db.cross_partition_scan.enabled` | Enable cross-partition scan. | `false` | +| `scalar.db.cross_partition_scan.filtering.enabled` | Enable filtering in cross-partition scan. | `false` | +| `scalar.db.cross_partition_scan.ordering.enabled` | Enable ordering in cross-partition scan. | `false` | + +#### Scan fetch size + +You can configure the fetch size for storage scan operations by using the following property: + +| Name | Description | Default | +|-----------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------| +| `scalar.db.scan_fetch_size` | Specifies the number of records to fetch in a single batch during a storage scan operation. A larger value can improve performance for a large result set by reducing round trips to the storage, but it also increases memory usage. A smaller value uses less memory but may increase latency. | `10` | + +### GraphQL-related configurations + +The configurations for ScalarDB Cluster GraphQL are as follows: + +| Name | Description | Default | +|-----------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------|----------------------| +| `scalar.db.graphql.enabled` | Whether ScalarDB Cluster GraphQL is enabled. | `false` | +| `scalar.db.graphql.port` | Port number of the GraphQL server. | `8080` | +| `scalar.db.graphql.path` | Path component of the URL of the GraphQL endpoint. | `/graphql` | +| `scalar.db.graphql.namespaces` | Comma-separated list of namespaces of tables for which the GraphQL server generates a schema. Note that at least one namespace is required. | | +| `scalar.db.graphql.graphiql` | Whether the GraphQL server serves [GraphiQL](https://github.com/graphql/graphiql) IDE. | `true` | +| `scalar.db.graphql.schema_checking_interval_millis` | Interval in milliseconds at which GraphQL server will rebuild the GraphQL schema if any change is detected in the ScalarDB schema. | `30000` (30 seconds) | + +#### Creating or modifying the ScalarDB schema when the server is running + +Since the GraphQL schema is statically built at server startup, if the ScalarDB schema is modified (for example, if a table is added, altered, or deleted), then the corresponding GraphQL schema won't reflect the changes unless it is rebuilt. To address this, the GraphQL server provides two mechanisms: a periodic check and an on-demand check. + +##### Run periodic checks + +The server periodically checks if changes in the ScalarDB schema occur and rebuilds the corresponding GraphQL schema if necessary. By default, the check occurs every 30 seconds, but the interval can be configured by using the `scalar.db.graphql.schema_checking_interval_millis` property. + +If you don't need to run periodic checks, you can disable it by setting the property value to `-1`. + +##### Run on-demand checks + +You can also request the server to check changes in the ScalarDB schema and rebuild the corresponding GraphQL schema if necessary by performing a POST request to the `/update-graphql-schema` endpoint of the HTTP API. + +For example, if the HTTP API is running on `localhost:8080` and the `scalar.db.graphql.path` property is set to `/graphql`, this endpoint can be called by running the following command: + +```console +curl -X POST http://localhost:8080/graphql/update-graphql-schema +``` + +### SQL-related configurations + +The configurations for ScalarDB Cluster SQL are as follows: + +| Name | Description | Default | +|------------------------------------------|----------------------------------------------------------------------------------------------------------|---------------| +| `scalar.db.sql.enabled` | Whether ScalarDB Cluster SQL is enabled. | `false` | +| `scalar.db.sql.statement_cache.enabled` | Enable the statement cache. | `false` | +| `scalar.db.sql.statement_cache.size` | Maximum number of cached statements. | `100` | +| `scalar.db.sql.default_transaction_mode` | Default transaction mode. `TRANSACTION` or `TWO_PHASE_COMMIT_TRANSACTION` can be set. | `TRANSACTION` | +| `scalar.db.sql.default_namespace_name` | Default namespace name. If you don't specify a namespace name in your SQL statement, this value is used. | | + +## Client configurations + +This section describes the general configurations for the ScalarDB Cluster client. + +### Configurations for the primitive interface + +The following table shows the general configurations for the ScalarDB Cluster client. + +| Name | Description | Default | +|----------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------| +| `scalar.db.transaction_manager` | `cluster` should be specified. | - | +| `scalar.db.contact_points` | Contact point of the cluster. If you use the `indirect` client mode, specify the IP address of the load balancer in front of your cluster nodes by using the format `indirect:`. If you use the `direct-kubernetes` client mode, specify the namespace name (optional) and the name of the [endpoint resource](https://kubernetes.io/docs/concepts/services-networking/service/#endpoints) to get the membership information by using the format `direct-kubernetes:/` or just `direct-kubernetes:`. If you don't specify the namespace name, the client will use the `default` namespace. | | +| `scalar.db.contact_port` | Port number for the contact point. | `60053` | +| `scalar.db.cluster.grpc.deadline_duration_millis` | Deadline duration for gRPC in millis. | `60000` (60 seconds) | +| `scalar.db.cluster.grpc.max_inbound_message_size` | Maximum message size allowed for a single gRPC frame. | The gRPC default value | +| `scalar.db.cluster.grpc.max_inbound_metadata_size` | Maximum size of metadata allowed to be received. | The gRPC default value | +| `scalar.db.cluster.client.scan_fetch_size` | The fetch size used for `Scanner` to fetch data from the cluster. This is the number of records that `Scanner` fetches at once from the cluster. A larger value can improve performance by reducing the number of round trips to the cluster, but it may also increase memory usage. | `10` | + +For example, if you use the `indirect` client mode and the load balancer IP address is `192.168.10.1`, you can configure the client as follows: + +```properties +scalar.db.transaction_manager=cluster +scalar.db.contact_points=indirect:192.168.10.1 +``` + +Or if you use the `direct-kubernetes` client mode, with the namespace of the endpoint as `ns` and the endpoint name as `scalardb-cluster`, you can configure the client as follows: + +```properties +scalar.db.transaction_manager=cluster +scalar.db.contact_points=direct-kubernetes:ns/scalardb-cluster +``` + +### Configurations for the SQL interface + +The following table shows the configurations for the ScalarDB Cluster SQL client. + +| Name | Description | Default | +|----------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------| +| `scalar.db.sql.connection_mode` | `cluster` should be specified. | - | +| `scalar.db.sql.cluster_mode.contact_points` | Contact point of the cluster. If you use the `indirect` client mode, specify the IP address of the load balancer in front of your cluster nodes by using the format `indirect:`. If you use the `direct-kubernetes` client mode, specify the namespace name (optional) and the name of the [endpoint resource](https://kubernetes.io/docs/concepts/services-networking/service/#endpoints) to get the membership information by using the format `direct-kubernetes:/` or just `direct-kubernetes:`. If you don't specify the namespace name, the client will use the `default` namespace. | | +| `scalar.db.sql.cluster_mode.contact_port` | Port number for the contact point. | `60053` | +| `scalar.db.sql.default_transaction_mode` | Default transaction mode. `TRANSACTION` or `TWO_PHASE_COMMIT_TRANSACTION` can be set. | `TRANSACTION` | +| `scalar.db.sql.default_namespace_name` | Default namespace name. If you don't specify a namespace name in your SQL statement, this value is used. | | +| `scalar.db.cluster.grpc.deadline_duration_millis` | Deadline duration for gRPC in millis. | `60000` (60 seconds) | +| `scalar.db.cluster.grpc.max_inbound_message_size` | Maximum message size allowed for a single gRPC frame. | The gRPC default value | +| `scalar.db.cluster.grpc.max_inbound_metadata_size` | Maximum size of metadata allowed to be received. | The gRPC default value | + +For example, if you use the `indirect` client mode and the load balancer IP address is `192.168.10.1`, you can configure the client as follows: + +```properties +scalar.db.sql.connection_mode=cluster +scalar.db.sql.cluster_mode.contact_points=indirect:192.168.10.1 +``` + +Or if you use the `direct-kubernetes` client mode, with the namespace of the endpoint as `ns` and the endpoint name as `scalardb-cluster`, you can configure the client as follows: + +```properties +scalar.db.sql.connection_mode=cluster +scalar.db.sql.cluster_mode.contact_points=direct-kubernetes:ns/scalardb-cluster +``` + +For details about how to configure ScalarDB JDBC, see [JDBC connection URL](../scalardb-sql/jdbc-guide.mdx#jdbc-connection-url). + +For details about how to configure Spring Data JDBC for ScalarDB, see [Configurations](../scalardb-sql/spring-data-guide.mdx#configurations). + +## Configuration example - App, ScalarDB Cluster, and database + +```mermaid +flowchart LR + app["App -
ScalarDB library with gRPC"] + cluster["ScalarDB Cluster -
(ScalarDB library with
Consensus Commit)"] + db[(Underlying storage or database)] + app --> cluster --> db +``` + +In this example configuration, the app (ScalarDB library with gRPC) connects to an underlying storage or database (in this case, Cassandra) through ScalarDB Cluster, which is a component that is available only in the ScalarDB Enterprise edition. + +:::note + +This configuration is acceptable for production use because ScalarDB Cluster implements the [Scalar Admin](https://github.com/scalar-labs/scalar-admin) interface, which enables you to take transactionally consistent backups for ScalarDB by pausing ScalarDB Cluster. + +::: + +The following is an example of the configuration for connecting the app to the underlying database through ScalarDB Cluster: + +```properties +# Transaction manager implementation. +scalar.db.transaction_manager=cluster + +# Contact point of the cluster. +scalar.db.contact_points=indirect: +``` + +For details about client configurations, see the [ScalarDB Cluster client configurations](#client-configurations). + +[^1]: It's worth benchmarking the performance with a few variations (for example, 75% and 125% of the default value) on the same underlying storage that your application uses, considering your application's access pattern, to determine the optimal configuration as it really depends on those factors. Also, it's important to benchmark combinations of these parameters (for example, first, `slot_capacity:20` and `group_size_fix_timeout_millis:40`; second, `slot_capacity:30` and `group_size_fix_timeout_millis:40`; and third, `slot_capacity:20` and `group_size_fix_timeout_millis:80`) to determine the optimal combination. diff --git a/versioned_docs/version-3.X/scalardb-cluster/scalardb-cluster-grpc-api-guide.mdx b/versioned_docs/version-3.X/scalardb-cluster/scalardb-cluster-grpc-api-guide.mdx new file mode 100644 index 00000000..ac860cd3 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster/scalardb-cluster-grpc-api-guide.mdx @@ -0,0 +1,236 @@ +--- +tags: + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# ScalarDB Cluster gRPC API Guide + +import WarningLicenseKeyContact from '/src/components/en-us/_warning-license-key-contact.mdx'; + +This document describes the ScalarDB Cluster gRPC API. + + + +ScalarDB Cluster provides a Java API that uses the gRPC API internally. +If you use Java or a JVM language, you can use the Java API instead of the ScalarDB Cluster gRPC API directly. +For details about the Java API, see [Developer Guide for ScalarDB Cluster with the Java API](developer-guide-for-scalardb-cluster-with-java-api.mdx). + +For details about the services and messages for the ScalarDB Cluster gRPC API, see the definitions in the `scalardb-cluster.proto` file. For ScalarDB Cluster users who have a commercial license, please [contact support](https://www.scalar-labs.com/support) if you need the `scalardb-cluster.proto` file. + +ScalarDB Cluster gRPC API is composed of the following services: + +- `scalardb.cluster.rpc.v1.DistributedTransaction`: Provides a distributed transaction capability for ScalarDB Cluster. +- `scalardb.cluster.rpc.v1.TwoPhaseCommitTransaction`: Provides a two-phase commit transaction capability for ScalarDB Cluster. +- `scalardb.cluster.rpc.v1.DistributedTransactionAdmin`: Provides comprehensive administrative operations. + +The following sections describe how to use each service. + +## Overview of error handling in ScalarDB Cluster gRPC API + +Before describing how to use each service, this section explains how error handling works in ScalarDB Cluster gRPC API. + +ScalarDB Cluster gRPC API employs [Richer error model](https://grpc.io/docs/guides/error/#richer-error-model) for error handling. +This model enables servers to return and enables clients to consume additional error details expressed as one or more protobuf messages. +ScalarDB Cluster gRPC API uses `google.rpc.ErrorInfo`, which is one of the [standard set of error message types](https://github.com/googleapis/googleapis/blob/master/google/rpc/error_details.proto), and puts additional error details in `ErrorInfo` fields. + +`ErrorInfo` has the following fields: + +- `reason`: A string that provides a short description of the error. The following sections describe the possible values of `reason` in each service. +- `domain`: A string that indicates the error's origin. In ScalarDB Cluster gRPC API, this string is always set to `com.scalar.db.cluster`. +- `metadata`: A map of metadata for the specific error. In ScalarDB Cluster gRPC API, a transaction ID with the `transactionId` key in the map is put if the error is related to a transaction. + +If you encounter an error, you can retrieve `ErrorInfo` from `google.rpc.Status` in the gRPC response, but the method for doing so depends on the programming language. +Please refer to the appropriate documentation to understand how to get `ErrorInfo` in your specific programming language. + +## How to use the `DistributedTransaction` service + +The `DistributedTransaction` service provides the following RPCs: + +- `Begin`: Begins a transaction. +- `Get`: Retrieves a record. +- `Scan`: Scans records. +- `Put`: Puts a record. +- `Delete`: Deletes a record. +- `Mutate` Mutates (puts and deletes) multiple records. +- `Commit`: Commits a transaction. +- `Rollback`: Rolls back a transaction. + +First, you call `Begin` to initiate a transaction. +Then, you can call `Get` and `Scan` to read records, `Put` and `Mutate` to write records, and `Delete` and `Mutate` to delete records. +To finalize the transaction, call `Commit`. +Alternatively, you can call `Rollback` at any time before the transaction is committed to cancel it. +By calling `Begin`, you receive a transaction ID in the response, which you can then use to call `Get`, `Scan`, `Put`, `Delete`, `Mutate`, `Commit`, and `Rollback`. + +When you call `Begin`, you can optionally specify a transaction ID. +If you specify a transaction ID, the user is responsible for guaranteeing the uniqueness of the ID. +If you do not specify a transaction ID, ScalarDB Cluster will generate a transaction ID for the transaction. + +You need to set `RequestHeader` for each RPC request. +`RequestHeader` contains a `hop_limit` field, which restricts the number of hops for a request. +The purpose of the `hop_limit` is to prevent infinite loops within the cluster. +Each time a request is forwarded to another cluster node, the `hop_limit` decreases by one. +If the `hop_limit` reaches zero, the request will be rejected. + +### Error handling + +The table below shows the status code and the possible values of `reason` in `ErrorInfo` in each RPC in the `DistributedTransaction` service: + +| RPC | Status code | `reason` in `ErrorInfo` | Description | +|--------------------------------|---------------------|----------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Begin | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | +| Begin | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | +| Begin | INTERNAL | HOP_LIMIT_EXCEEDED | The hop limit was exceeded. This occurs when the routing information between cluster nodes is inconsistent. The error is usually resolved in a short amount of time, so you can retry the transaction from the beginning after some time has passed since encountering this error. | +| Begin | INTERNAL | INTERNAL_ERROR | The operation has failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | +| Get, Scan, Put, Delete, Mutate | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | +| Get, Scan, Put, Delete, Mutate | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | +| Get, Scan, Put, Delete, Mutate | NOT_FOUND | TRANSACTION_NOT_FOUND | The transaction associated with the specified transaction ID was not found. This error indicates that the transaction has expired or the routing information has been updated due to cluster topology changes. In this case, please retry the transaction from the beginning. | +| Get, Scan, Put, Delete, Mutate | INTERNAL | HOP_LIMIT_EXCEEDED | The hop limit was exceeded. This occurs when the routing information between cluster nodes is inconsistent. The error is usually resolved in a short amount of time, so you can retry the transaction from the beginning after some time has passed since encountering this error. | +| Get, Scan, Put, Delete, Mutate | FAILED_PRECONDITION | TRANSACTION_CONFLICT | A transaction conflict occurred. If you encounter this error, please retry the transaction from the beginning. | +| Get, Scan, Put, Delete, Mutate | INTERNAL | INTERNAL_ERROR | The operation has failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | +| Put, Delete, Mutate | FAILED_PRECONDITION | UNSATISFIED_CONDITION | The mutation condition is not satisfied. | +| Commit | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | +| Commit | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | +| Commit | NOT_FOUND | TRANSACTION_NOT_FOUND | The transaction associated with the specified transaction ID was not found. This error indicates that the transaction has expired or the routing information has been updated due to cluster topology changes. In this case, please retry the transaction from the beginning. | +| Commit | INTERNAL | HOP_LIMIT_EXCEEDED | The hop limit was exceeded. This occurs when the routing information between cluster nodes is inconsistent. The error is usually resolved in a short amount of time, so you can retry the transaction from the beginning after some time has passed since encountering this error. | +| Commit | FAILED_PRECONDITION | TRANSACTION_CONFLICT | A transaction conflict occurred. If you encounter this error, please retry the transaction from the beginning. | +| Commit | INTERNAL | UNKNOWN_TRANSACTION_STATUS | The status of the transaction is unknown (it is uncertain whether the transaction was successfully committed or not). In this situation, you need to check whether the transaction was successfully committed, and if not, to retry it. The responsibility for determining the transaction status rests with the users. It may be beneficial to create a transaction status table and update it in conjunction with other application data so that you can determine the status of a transaction from the table itself. | +| Commit | INTERNAL | INTERNAL_ERROR | The operation has failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | +| Rollback | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | +| Rollback | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | +| Rollback | NOT_FOUND | TRANSACTION_NOT_FOUND | The transaction associated with the specified transaction ID was not found. In case of a rollback, you do not need to retry the transaction because the transaction will expire automatically. | +| Rollback | INTERNAL | HOP_LIMIT_EXCEEDED | The hop limit was exceeded. In case of a rollback, you do not need to retry the transaction because the transaction will expire automatically. | +| Rollback | INTERNAL | INTERNAL_ERROR | The operation has failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | + +If you encounter an error, you should roll back the transaction, except in the case of `Begin`. +Then, you can retry the transaction from the beginning for the errors that can be resolved by retrying. + +Besides the errors listed above, you may encounter errors returned by the gRPC library. +In these cases, the response will not contain `ErrorInfo`. +For details, refer to the [gRPC documentation](https://grpc.io/docs/guides/error/#error-status-codes). + +You can set a deadline for each RPC in gRPC. +If the deadline is exceeded, you will receive a `DEADLINE_EXCEEDED` error. +In general, you should roll back the transaction in this situation, unless the RPC is `Begin` or `Commit`. +In the case of `Commit`, the situation is equivalent to `UNKNOWN_TRANSACTION_STATUS` (it is uncertain whether the transaction was successfully committed or not), and you must handle the error in the same way. + +## How to use the `TwoPhaseCommitTransaction` service + +The `TwoPhaseCommitTransaction` service provides the following RPCs: + +- `Begin`: Begins a transaction. +- `Join`: Joins a transaction. +- `Get`: Retrieves a record. +- `Scan`: Scans records. +- `Put`: Puts a record. +- `Delete`: Deletes a record. +- `Mutate`: Mutates (puts and deletes) multiple records. +- `Prepare`: Prepares a transaction. +- `Validate`: Validates a transaction. +- `Commit`: Commits a transaction. +- `Rollback`: Rolls back a transaction. + +First, you call `Begin` to initiate a transaction if you are the coordinator process. +Alternatively, if you are a participant process, you can call `Join` to take part in a transaction that the coordinator has already begun. +Then, you can call `Get` and `Scan` to read records, `Put` and `Mutate` to write records, and `Delete` and `Mutate` to delete records. +To finalize the transaction, call `Prepare`, `Validate`, and then `Commit` in order. +Alternatively, you can call `Rollback` at any time before the transaction is committed to cancel it. +By calling `Begin` or `Join`, you receive a transaction ID in the response, which you can then use to call `Get`, `Scan`, `Put`, `Delete`, `Mutate`, `Prepare`, `Validate`, `Commit`, and `Rollback`. + +When you call `Begin`, you can optionally specify a transaction ID. +If you specify a transaction ID, the user is responsible for guaranteeing the uniqueness of the ID. +If you do not specify a transaction ID, ScalarDB Cluster will generate a transaction ID for the transaction. + +You need to set `RequestHeader` for each RPC request. +`RequestHeader` contains a `hop_limit` field, which restricts the number of hops for a request. +The purpose of the `hop_limit` is to prevent infinite loops within the cluster. +Each time a request is forwarded to another cluster node, the `hop_limit` decreases by one. +If the `hop_limit` reaches zero, the request will be rejected. + +### Error handling + +The table below shows the status code and the possible values of `reason` in `ErrorInfo` in each RPC in the `TwoPhaseCommitTransaction` service: + +| RPC | Status code | `reason` in `ErrorInfo` | Description | +|--------------------------------|---------------------|----------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Begin, Join | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | +| Begin, Join | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | +| Begin, Join | INTERNAL | HOP_LIMIT_EXCEEDED | The hop limit was exceeded. This occurs when the routing information between cluster nodes is inconsistent. The error is usually resolved in a short amount of time, so you can retry the transaction from the beginning after some time has passed since encountering this error. | +| Begin, Join | INTERNAL | INTERNAL_ERROR | The operation has failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | +| Get, Scan, Put, Delete, Mutate | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | +| Get, Scan, Put, Delete, Mutate | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | +| Get, Scan, Put, Delete, Mutate | NOT_FOUND | TRANSACTION_NOT_FOUND | The transaction associated with the specified transaction ID was not found. This indicates that the transaction has expired or the routing information has been updated due to cluster topology changes. In this case, please retry the transaction from the beginning. | +| Get, Scan, Put, Delete, Mutate | INTERNAL | HOP_LIMIT_EXCEEDED | The hop limit was exceeded. This occurs when the routing information between cluster nodes is inconsistent. The error is usually resolved in a short amount of time, so you can retry the transaction from the beginning after some time has passed since encountering this error. | +| Get, Scan, Put, Delete, Mutate | FAILED_PRECONDITION | TRANSACTION_CONFLICT | A transaction conflict occurred. If you encounter this error, please retry the transaction from the beginning. | +| Get, Scan, Put, Delete, Mutate | INTERNAL | INTERNAL_ERROR | The operation has failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | +| Put, Delete, Mutate | FAILED_PRECONDITION | UNSATISFIED_CONDITION | The mutation condition is not satisfied. | +| Prepare, Validate | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | +| Prepare, Validate | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | +| Prepare, Validate | NOT_FOUND | TRANSACTION_NOT_FOUND | The transaction associated with the specified transaction ID was not found. This error indicates that the transaction has expired or the routing information has been updated due to cluster topology changes. In this case, please retry the transaction from the beginning. | +| Prepare, Validate | INTERNAL | HOP_LIMIT_EXCEEDED | The hop limit was exceeded. This occurs when the routing information between cluster nodes is inconsistent. The error is usually resolved in a short amount of time, so you can retry the transaction from the beginning after some time has passed since encountering this error. | +| Prepare, Validate | FAILED_PRECONDITION | TRANSACTION_CONFLICT | A transaction conflict occurred. If you encounter this error, please retry the transaction from the beginning. | +| Prepare, Validate | INTERNAL | INTERNAL_ERROR | The operation has failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | +| Commit | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | +| Commit | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | +| Commit | NOT_FOUND | TRANSACTION_NOT_FOUND | The transaction associated with the specified transaction ID was not found. This error indicates that the transaction has expired or the routing information has been updated due to cluster topology changes. In this case, please retry the transaction from the beginning. | +| Commit | INTERNAL | HOP_LIMIT_EXCEEDED | The hop limit was exceeded. This occurs when the routing information between cluster nodes is inconsistent. The error is usually resolved in a short amount of time, so you can retry the transaction from the beginning after some time has passed since encountering this error. | +| Commit | FAILED_PRECONDITION | TRANSACTION_CONFLICT | A transaction conflict occurred. If you encounter this error, please retry the transaction from the beginning. | +| Commit | INTERNAL | UNKNOWN_TRANSACTION_STATUS | The status of the transaction is unknown (it is uncertain whether the transaction was successfully committed or not). In this situation, you need to check whether the transaction was successfully committed, and if not, to retry it. The responsibility for determining the transaction status rests with the users. It may be beneficial to create a transaction status table and update it in conjunction with other application data so that you can determine the status of a transaction from the table itself. | +| Commit | INTERNAL | INTERNAL_ERROR | The operation has failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | +| Rollback | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | +| Rollback | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | +| Rollback | NOT_FOUND | TRANSACTION_NOT_FOUND | The transaction associated with the specified transaction ID was not found. In case of a rollback, you do not need to retry the transaction because the transaction will expire automatically. | +| Rollback | INTERNAL | HOP_LIMIT_EXCEEDED | The hop limit was exceeded. In case of a rollback, you do not need to retry the transaction because the transaction will expire automatically. | +| Rollback | INTERNAL | INTERNAL_ERROR | The operation has failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | + +If you encounter an error, you should roll back the transaction, except in the case of `Begin` or `Join`. +Then, you can retry the transaction from the beginning for the errors that can be resolved by retrying. + +Besides the errors listed above, you may encounter errors returned by the gRPC library. +In these cases, the response will not contain `ErrorInfo`. +For details, refer to the [gRPC documentation](https://grpc.io/docs/guides/error/#error-status-codes). + +You can set a deadline for each RPC in gRPC. +If the deadline is exceeded, you will receive a `DEADLINE_EXCEEDED` error. +In general, you should roll back the transaction in this situation, unless the RPC is `Begin`, `Join`, or `Commit`. +In the case of `Commit`, the situation is equivalent to `UNKNOWN_TRANSACTION_STATUS` (it is uncertain whether the transaction was successfully committed or not), and you must handle the error in the same way. + +## How to use the `DistributedTransactionAdmin` service + +The `DistributedTransactionAdmin` service provides the following RPCs: + +- `CreateNamespace`: Creates a namespace. +- `DropNamespace`: Drops a namespace. +- `NamespaceExists`: Returns whether the specified namespace exists or not. +- `CreateTable`: Creates a table. +- `DropTable`: Drops a table. +- `TruncateTable`: Truncates a table. +- `TableExists`: Returns whether the specified table exists or not. +- `CreateIndex`: Creates an index. +- `DropIndex`: Drops an index. +- `IndexExists`: Returns whether the specified index exists or not. +- `RepairTable`: Repairs a namespace that may be in an unknown state. +- `AddNewColumnToTable`: Adds a new column to a table. +- `CreateCoordinatorTables`: Creates the Coordinator tables. +- `DropCoordinatorTables`: Drops the Coordinator tables. +- `TruncateCoordinatorTables`: Truncates the Coordinator tables. +- `CoordinatorTablesExist`: Returns whether the Coordinator tables exist or not. +- `RepairCoordinatorTables`: Repairs the Coordinator tables. +- `GetTableMetadata`: Returns table metadata of the specified table. +- `GetNamespaceTableNames`: Returns tables in the specified namespace. +- `ImportTable`: Imports an existing table that is not managed by ScalarDB. + +### Error handling + +The table below shows the status code and the possible values of `reason` in `ErrorInfo` for all RPCs in the `DistributedTransactionAdmin` service: + +| Status code | `reason` in `ErrorInfo` | Description | +|---------------------|----------------------------|-------------------------------------------------| +| INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | +| FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | +| INTERNAL | INTERNAL_ERROR | The operation has failed. | + +Besides the errors listed above, you may encounter errors returned by the gRPC library. +In these cases, the response will not contain `ErrorInfo`. +For details, refer to the [gRPC documentation](https://grpc.io/docs/guides/error/#error-status-codes). diff --git a/versioned_docs/version-3.X/scalardb-cluster/scalardb-cluster-sql-grpc-api-guide.mdx b/versioned_docs/version-3.X/scalardb-cluster/scalardb-cluster-sql-grpc-api-guide.mdx new file mode 100644 index 00000000..4901626b --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster/scalardb-cluster-sql-grpc-api-guide.mdx @@ -0,0 +1,219 @@ +--- +tags: + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# ScalarDB Cluster SQL gRPC API Guide + +import WarningLicenseKeyContact from '/src/components/en-us/_warning-license-key-contact.mdx'; + +This document describes the ScalarDB Cluster SQL gRPC API. + + + +ScalarDB Cluster SQL provides a Java API that uses the gRPC API internally. +If you use Java or a JVM language, you can use the Java API instead of the ScalarDB Cluster SQL gRPC API directly. +For details about the Java API, see [Developer Guide for ScalarDB Cluster with the Java API](developer-guide-for-scalardb-cluster-with-java-api.mdx). + +For details about the services and messages for the ScalarDB Cluster SQL gRPC API, see the definitions in the `scalardb-cluster-sql.proto` file. For ScalarDB Cluster users who have a commercial license, please [contact support](https://www.scalar-labs.com/support) if you need the `scalardb-cluster-sql.proto` file. + +ScalarDB Cluster SQL gRPC API is composed of the following services: + +- `scalardb.cluster.rpc.v1.sql.SqlTransaction`: Provides a transaction capability for ScalarDB Cluster SQL. +- `scalardb.cluster.rpc.v1.sql.SqlTwoPhaseCommitTransaction`: Provides a two-phase commit transaction capability for ScalarDB Cluster SQL. +- `scalardb.cluster.rpc.v1.sql.Metadata`: Provides a metadata view of ScalarDB Cluster SQL. + +The following sections describe how to use each service. + +## Overview of error handling in ScalarDB Cluster SQL gRPC API + +Before describing how to use each service, this section explains how error handling works in ScalarDB Cluster SQL gRPC API. + +ScalarDB Cluster SQL gRPC API employs [Richer error model](https://grpc.io/docs/guides/error/#richer-error-model) for error handling. +This model enables servers to return and enables clients to consume additional error details expressed as one or more protobuf messages. +ScalarDB Cluster SQL gRPC API uses `google.rpc.ErrorInfo`, which is one of the [standard set of error message types](https://github.com/googleapis/googleapis/blob/master/google/rpc/error_details.proto), and puts additional error details in `ErrorInfo` fields. + +`ErrorInfo` has the following fields: + +- `reason`: A string that provides a short description of the error. The following sections describe the possible values of `reason` in each service. +- `domain`: A string that indicates the error's origin. In ScalarDB Cluster SQL gRPC API, this string is always set to `com.scalar.db.cluster.sql`. +- `metadata`: A map of metadata for the specific error. In ScalarDB Cluster SQL gRPC API, a transaction ID with the `transactionId` key in the map is put if the error is related to a transaction. + +If you encounter an error, you can retrieve `ErrorInfo` from `google.rpc.Status` in the gRPC response, but the method for doing so depends on the programming language. +Please refer to the appropriate documentation to understand how to get `ErrorInfo` in your specific programming language. + +## How to use the `SqlTransaction` service + +The `SqlTransaction` service provides the following RPCs: + +- `Begin`: Begins a transaction. +- `Execute` Executes a SQL statement. +- `Commit`: Commits a transaction. +- `Rollback`: Rolls back a transaction. + +First, you call `Begin` to initiate a transaction. +Following that, you can call `Execute` to read, write, and delete records. +To finalize the transaction, call `Commit`. +Alternatively, you can call `Rollback` at any time before the transaction is committed to cancel it. +By calling `Begin`, you receive a transaction ID in the response, which you can then use to call `Execute`, `Commit`, and `Rollback`. + +Also, you can call `Execute` without a transaction ID to execute a one-shot transaction. +In this case, the transaction is automatically committed after it is executed. +You can use this method to execute DDL statements as well. +For details on the supported SQL statements, refer to [ScalarDB SQL Grammar](../scalardb-sql/grammar.mdx). +Please note, however, that `Execute` supports only DML and DDL statements. + +When you call `Begin`, you can optionally specify a transaction ID. +If you specify a transaction ID, the user is responsible for guaranteeing the uniqueness of the ID. +If you do not specify a transaction ID, ScalarDB Cluster will generate a transaction ID for the transaction. + +You need to set `RequestHeader` for each RPC request. +`RequestHeader` contains a `hop_limit` field, which restricts the number of hops for a request. +The purpose of the `hop_limit` is to prevent infinite loops within the cluster. +Each time a request is forwarded to another cluster node, the `hop_limit` decreases by one. +If the `hop_limit` reaches zero, the request will be rejected. + +### Error handling + +The table below shows the status code and the possible values of `reason` in `ErrorInfo` in each RPC in the `SqlTransaction` service: + +| RPC | Status code | `reason` in `ErrorInfo` | Description | +|----------|---------------------|----------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Begin | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | +| Begin | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | +| Begin | INTERNAL | HOP_LIMIT_EXCEEDED | The hop limit was exceeded. This occurs when the routing information between cluster nodes is inconsistent. The error is usually resolved in a short amount of time, so you can retry the transaction from the beginning after some time has passed since encountering this error. | +| Begin | INTERNAL | INTERNAL_ERROR | The operation has failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | +| Execute | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | +| Execute | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | +| Execute | NOT_FOUND | TRANSACTION_NOT_FOUND | The transaction associated with the specified transaction ID was not found. This error indicates that the transaction has expired or the routing information has been updated due to cluster topology changes. In this case, please retry the transaction from the beginning. | +| Execute | INTERNAL | HOP_LIMIT_EXCEEDED | The hop limit was exceeded. This occurs when the routing information between cluster nodes is inconsistent. The error is usually resolved in a short amount of time, so you can retry the transaction from the beginning after some time has passed since encountering this error. | +| Execute | FAILED_PRECONDITION | TRANSACTION_CONFLICT | A transaction conflict occurred. If you encounter this error, please retry the transaction from the beginning. | +| Execute | INTERNAL | INTERNAL_ERROR | The operation has failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | +| Commit | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | +| Commit | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | +| Commit | NOT_FOUND | TRANSACTION_NOT_FOUND | The transaction associated with the specified transaction ID was not found. This error indicates that the transaction has expired or the routing information has been updated due to cluster topology changes. In this case, please retry the transaction from the beginning. | +| Commit | INTERNAL | HOP_LIMIT_EXCEEDED | The hop limit was exceeded. This occurs when the routing information between cluster nodes is inconsistent. The error is usually resolved in a short amount of time, so you can retry the transaction from the beginning after some time has passed since encountering this error. | +| Commit | FAILED_PRECONDITION | TRANSACTION_CONFLICT | A transaction conflict occurred. If you encounter this error, please retry the transaction from the beginning. | +| Commit | INTERNAL | UNKNOWN_TRANSACTION_STATUS | The status of the transaction is unknown (it is uncertain whether the transaction was successfully committed or not). In this situation, you need to check whether the transaction was successfully committed, and if not, to retry it. The responsibility for determining the transaction status rests with the users. It may be beneficial to create a transaction status table and update it in conjunction with other application data so that you can determine the status of a transaction from the table itself. | +| Commit | INTERNAL | INTERNAL_ERROR | The operation has failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | +| Rollback | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | +| Rollback | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | +| Rollback | NOT_FOUND | TRANSACTION_NOT_FOUND | The transaction associated with the specified transaction ID was not found. In case of a rollback, you do not need to retry the transaction because the transaction will expire automatically. | +| Rollback | INTERNAL | HOP_LIMIT_EXCEEDED | The hop limit was exceeded. In case of a rollback, you do not need to retry the transaction because the transaction will expire automatically. | +| Rollback | INTERNAL | INTERNAL_ERROR | The operation has failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | + +If you encounter an error, you should roll back the transaction, except in the case of `Begin`. +Then, you can retry the transaction from the beginning for the errors that can be resolved by retrying. + +Besides the errors listed above, you may encounter errors returned by the gRPC library. +In these cases, the response will not contain `ErrorInfo`. +For details, refer to the [gRPC documentation](https://grpc.io/docs/guides/error/#error-status-codes). + +You can set a deadline for each RPC in gRPC. +If the deadline is exceeded, you will receive a `DEADLINE_EXCEEDED` error. +In general, you should roll back the transaction in this situation, unless the RPC is `Begin` or `Commit`. +In the case of `Commit`, the situation is equivalent to `UNKNOWN_TRANSACTION_STATUS` (it is uncertain whether the transaction was successfully committed or not), and you must handle the error in the same way. + +## How to use the `SqlTwoPhaseCommitTransaction` service + +The `SqlTwoPhaseCommitTransaction` service provides the following RPCs: + +- `Begin`: Begins a transaction. +- `Join`: Joins a transaction. +- `Execute` Executes a SQL statement. +- `Prepare`: Prepares a transaction. +- `Validate`: Validates a transaction. +- `Commit`: Commits a transaction. +- `Rollback`: Rolls back a transaction. + +First, you call `Begin` to initiate a transaction if you are the coordinator process. +Alternatively, if you are a participant process, you can call `Join` to take part in a transaction that the coordinator has already begun. +Following that, you can call `Execute` to read, write, and delete records. +To finalize the transaction, call `Prepare`, `Validate`, and then `Commit` in order. +Alternatively, you can call `Rollback` at any time before the transaction is committed to cancel it. +By calling `Begin` or `Join`, you receive a transaction ID in the response, which you can then use to call `Execute`, `Prepare`, `Validate`, `Commit`, and `Rollback`. + +In addition, you can call `Execute` without a transaction ID to execute a one-shot transaction. +In this case, the transaction is automatically committed after it is executed. +You can use this method to execute DDL statements as well. For details on the supported SQL statements, refer to [ScalarDB SQL Grammar](../scalardb-sql/grammar.mdx). +Please note, however, that `Execute` supports only DML and DDL statements. + +When you call `Begin`, you can optionally specify a transaction ID. +If you specify a transaction ID, the user is responsible for guaranteeing the uniqueness of the ID. +If you do not specify a transaction ID, ScalarDB Cluster will generate a transaction ID for the transaction. + +You need to set `RequestHeader` for each RPC request. +`RequestHeader` contains a `hop_limit` field, which restricts the number of hops for a request. +The purpose of the `hop_limit` is to prevent infinite loops within the cluster. +Each time a request is forwarded to another cluster node, the `hop_limit` decreases by one. +If the `hop_limit` reaches zero, the request will be rejected. + +### Error handling + +The table below shows the status code and the possible values of `reason` in `ErrorInfo` in each RPC in the `SqlTwoPhaseCommitTransaction` service: + +| RPC | Status code | `reason` in `ErrorInfo` | Description | +|-------------------|---------------------|----------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| Begin, Join | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | +| Begin, Join | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | +| Begin, Join | INTERNAL | HOP_LIMIT_EXCEEDED | The hop limit was exceeded. This occurs when the routing information between cluster nodes is inconsistent. The error is usually resolved in a short amount of time, so you can retry the transaction from the beginning after some time has passed since encountering this error. | +| Begin, Join | INTERNAL | INTERNAL_ERROR | The operation has failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | +| Execute | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | +| Execute | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | +| Execute | NOT_FOUND | TRANSACTION_NOT_FOUND | The transaction associated with the specified transaction ID was not found. This error indicates that the transaction has expired or the routing information has been updated due to cluster topology changes. In this case, please retry the transaction from the beginning. | +| Execute | INTERNAL | HOP_LIMIT_EXCEEDED | The hop limit was exceeded. This occurs when the routing information between cluster nodes is inconsistent. The error is usually resolved in a short amount of time, so you can retry the transaction from the beginning after some time has passed since encountering this error. | +| Execute | FAILED_PRECONDITION | TRANSACTION_CONFLICT | A transaction conflict occurred. If you encounter this error, please retry the transaction from the beginning. | +| Execute | INTERNAL | INTERNAL_ERROR | The operation has failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | +| Prepare, Validate | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | +| Prepare, Validate | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | +| Prepare, Validate | NOT_FOUND | TRANSACTION_NOT_FOUND | The transaction associated with the specified transaction ID was not found. This error indicates that the transaction has expired or the routing information has been updated due to cluster topology changes. In this case, please retry the transaction from the beginning. | +| Prepare, Validate | INTERNAL | HOP_LIMIT_EXCEEDED | The hop limit was exceeded. This occurs when the routing information between cluster nodes is inconsistent. The error is usually resolved in a short amount of time, so you can retry the transaction from the beginning after some time has passed since encountering this error. | +| Prepare, Validate | FAILED_PRECONDITION | TRANSACTION_CONFLICT | A transaction conflict occurred. If you encounter this error, please retry the transaction from the beginning. | +| Prepare, Validate | INTERNAL | INTERNAL_ERROR | The operation has failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | +| Commit | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | +| Commit | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | +| Commit | NOT_FOUND | TRANSACTION_NOT_FOUND | The transaction associated with the specified transaction ID was not found. This error indicates that the transaction has expired or the routing information has been updated due to cluster topology changes. In this case, please retry the transaction from the beginning. | +| Commit | INTERNAL | HOP_LIMIT_EXCEEDED | The hop limit was exceeded. This occurs when the routing information between cluster nodes is inconsistent. The error is usually resolved in a short amount of time, so you can retry the transaction from the beginning after some time has passed since encountering this error. | +| Commit | FAILED_PRECONDITION | TRANSACTION_CONFLICT | A transaction conflict occurred. If you encounter this error, please retry the transaction from the beginning. | +| Commit | INTERNAL | UNKNOWN_TRANSACTION_STATUS | The status of the transaction is unknown (it is uncertain whether the transaction was successfully committed or not). In this situation, you need to check whether the transaction was successfully committed, and if not, to retry it. The responsibility for determining the transaction status rests with the users. It may be beneficial to create a transaction status table and update it in conjunction with other application data so that you can determine the status of a transaction from the table itself. | +| Commit | INTERNAL | INTERNAL_ERROR | The operation has failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | +| Rollback | INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | +| Rollback | FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | +| Rollback | NOT_FOUND | TRANSACTION_NOT_FOUND | The transaction associated with the specified transaction ID was not found. In case of a rollback, you do not need to retry the transaction because the transaction will expire automatically. | +| Rollback | INTERNAL | HOP_LIMIT_EXCEEDED | The hop limit was exceeded. In case of a rollback, you do not need to retry the transaction because the transaction will expire automatically. | +| Rollback | INTERNAL | INTERNAL_ERROR | The operation has failed due to transient or nontransient faults. You can try retrying the transaction from the beginning, but the transaction may still fail if the cause is nontransient. | + +If you encounter an error, you should roll back the transaction, except in the case of `Begin` or `Join`. +Then, you can retry the transaction from the beginning for the errors that can be resolved by retrying. + +Besides the errors listed above, you may encounter errors returned by the gRPC library. +In these cases, the response will not contain `ErrorInfo`. +For details, refer to the [gRPC documentation](https://grpc.io/docs/guides/error/#error-status-codes). + +You can set a deadline for each RPC in gRPC. +If the deadline is exceeded, you will receive a `DEADLINE_EXCEEDED` error. +In general, you should roll back the transaction in this situation, unless the RPC is `Begin`, `Join`, or `Commit`. +In the case of `Commit`, the situation is equivalent to `UNKNOWN_TRANSACTION_STATUS` (it is uncertain whether the transaction was successfully committed or not), and you must handle the error in the same way. + +## How to use the `Metadata` service + +The `Metadata` service provides the following RPCs: + +- `GetNamespaceMetadata`: Retrieves namespace metadata of the specified namespace. +- `ListTableMetadataInNamespace`: Retrieves table metadata of tables in the specified namespace. +- `GetTableMetadata`: Retrieves table metadata of the specified table. + +### Error handling + +The table below shows the status code and the possible values of `reason` in `ErrorInfo` for all RPCs in the `Metadata` service: + +| Status code | `reason` in `ErrorInfo` | Description | +|---------------------|----------------------------|-------------------------------------------------| +| INVALID_ARGUMENT | ILLEGAL_ARGUMENT | The argument in the request message is invalid. | +| FAILED_PRECONDITION | ILLEGAL_STATE | The RPC was called in an invalid state. | +| INTERNAL | INTERNAL_ERROR | The operation has failed. | + +Besides the errors listed above, you may encounter errors returned by the gRPC library. +In these cases, the response will not contain `ErrorInfo`. +For details, refer to the [gRPC documentation](https://grpc.io/docs/guides/error/#error-status-codes). diff --git a/versioned_docs/version-3.X/scalardb-cluster/scalardb-cluster-status-codes.mdx b/versioned_docs/version-3.X/scalardb-cluster/scalardb-cluster-status-codes.mdx new file mode 100644 index 00000000..01693c3b --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster/scalardb-cluster-status-codes.mdx @@ -0,0 +1,542 @@ +--- +tags: + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# ScalarDB Cluster Error Codes + +This page provides a list of error codes in ScalarDB Cluster. + +## Error code classes and descriptions + +| Class | Description | +|:-------------------|:------------------------------------------| +| `DB-CLUSTER-1xxxx` | Errors for the user error category | +| `DB-CLUSTER-2xxxx` | Errors for the concurrency error category | +| `DB-CLUSTER-3xxxx` | Errors for the internal error category | + +## `DB-CLUSTER-1xxxx` status codes + +The following are status codes and messages for the user error category. + +### `DB-CLUSTER-10000` + +**Message** + +```markdown +The namespace does not exist. Namespace: %s +``` + +### `DB-CLUSTER-10001` + +**Message** + +```markdown +The table does not exist. Table: %s +``` + +### `DB-CLUSTER-10002` + +**Message** + +```markdown +The user does not exist. User: %s +``` + +### `DB-CLUSTER-10004` + +**Message** + +```markdown +The get type is unspecified +``` + +### `DB-CLUSTER-10005` + +**Message** + +```markdown +The get type is unrecognized +``` + +### `DB-CLUSTER-10006` + +**Message** + +```markdown +The value of the column is not set. Column: %s +``` + +### `DB-CLUSTER-10007` + +**Message** + +```markdown +The scan type is unspecified +``` + +### `DB-CLUSTER-10008` + +**Message** + +```markdown +The scan type is unrecognized +``` + +### `DB-CLUSTER-10009` + +**Message** + +```markdown +The order is unspecified +``` + +### `DB-CLUSTER-10010` + +**Message** + +```markdown +The order is unrecognized +``` + +### `DB-CLUSTER-10011` + +**Message** + +```markdown +The clustering order is unspecified +``` + +### `DB-CLUSTER-10012` + +**Message** + +```markdown +The clustering order is unrecognized +``` + +### `DB-CLUSTER-10013` + +**Message** + +```markdown +The put condition type is unspecified +``` + +### `DB-CLUSTER-10014` + +**Message** + +```markdown +The put condition type is unrecognized +``` + +### `DB-CLUSTER-10015` + +**Message** + +```markdown +The delete condition type is unspecified +``` + +### `DB-CLUSTER-10016` + +**Message** + +```markdown +The delete condition type is unrecognized +``` + +### `DB-CLUSTER-10017` + +**Message** + +```markdown +The operator is unspecified +``` + +### `DB-CLUSTER-10018` + +**Message** + +```markdown +The operator is unrecognized +``` + +### `DB-CLUSTER-10019` + +**Message** + +```markdown +The mutation is not set +``` + +### `DB-CLUSTER-10020` + +**Message** + +```markdown +The data type is unspecified +``` + +### `DB-CLUSTER-10021` + +**Message** + +```markdown +The data type is unrecognized +``` + +### `DB-CLUSTER-10022` + +**Message** + +```markdown +The user option is unspecified +``` + +### `DB-CLUSTER-10023` + +**Message** + +```markdown +The user option is unrecognized +``` + +### `DB-CLUSTER-10024` + +**Message** + +```markdown +The privilege is unspecified +``` + +### `DB-CLUSTER-10025` + +**Message** + +```markdown +The privilege is unrecognized +``` + +### `DB-CLUSTER-10026` + +**Message** + +```markdown +The username is not set +``` + +### `DB-CLUSTER-10027` + +**Message** + +```markdown +This feature is not supported in ScalarDB Cluster +``` + +### `DB-CLUSTER-10028` + +**Message** + +```markdown +The property 'scalar.db.contact_points' must not be empty +``` + +### `DB-CLUSTER-10029` + +**Message** + +```markdown +The property 'scalar.db.contact_points' must be prefixed with 'indirect:' or 'direct-kubernetes:' +``` + +### `DB-CLUSTER-10030` + +**Message** + +```markdown +The format of the property 'scalar.db.contact_points' for direct-kubernetes client mode is 'direct-kubernetes:/' or 'direct-kubernetes:' +``` + +### `DB-CLUSTER-10031` + +**Message** + +```markdown +The property 'scalar.db.sql.cluster_mode.contact_points' must not be empty +``` + +### `DB-CLUSTER-10032` + +**Message** + +```markdown +The property 'scalar.db.sql.cluster_mode.contact_points' must be prefixed with 'indirect:' or 'direct-kubernetes:' +``` + +### `DB-CLUSTER-10033` + +**Message** + +```markdown +The format of the property 'scalar.db.sql.cluster_mode.contact_points' for direct-kubernetes client mode is 'direct-kubernetes:/' or 'direct-kubernetes:' +``` + +### `DB-CLUSTER-10035` + +**Message** + +```markdown +The update condition type is unspecified +``` + +### `DB-CLUSTER-10036` + +**Message** + +```markdown +The update condition type is unrecognized +``` + +### `DB-CLUSTER-10037` + +**Message** + +```markdown +The two-phase commit interface is not supported +``` + +### `DB-CLUSTER-10039` + +**Message** + +```markdown +The policy state is unspecified +``` + +### `DB-CLUSTER-10040` + +**Message** + +```markdown +The policy state is unrecognized +``` + +### `DB-CLUSTER-10041` + +**Message** + +```markdown +The access mode is unspecified +``` + +### `DB-CLUSTER-10042` + +**Message** + +```markdown +The access mode is unrecognized +``` + +### `DB-CLUSTER-10043` + +**Message** + +```markdown +The service does not exist. Service Class: %s +``` + +### `DB-CLUSTER-10044` + +**Message** + +```markdown +The policy does not exist. Policy: %s +``` + +### `DB-CLUSTER-10045` + +**Message** + +```markdown +The property 'scalar.db.embedding.client.contact_points' must not be empty +``` + +### `DB-CLUSTER-10046` + +**Message** + +```markdown +The property 'scalar.db.embedding.client.contact_points' must be prefixed with 'indirect:' or 'direct-kubernetes:' +``` + +### `DB-CLUSTER-10047` + +**Message** + +```markdown +The format of the property 'scalar.db.embedding.client.contact_points' for direct-kubernetes client mode is 'direct-kubernetes:/' or 'direct-kubernetes:' +``` + +### `DB-CLUSTER-10048` + +**Message** + +```markdown +The embeddings must be provided +``` + +### `DB-CLUSTER-10049` + +**Message** + +```markdown +Only one embedding can be added with an embedding ID +``` + +### `DB-CLUSTER-10050` + +**Message** + +```markdown +Text segments cannot be provided when adding an embedding with an embedding ID +``` + +### `DB-CLUSTER-10051` + +**Message** + +```markdown +Both embedding IDs and a filter cannot be provided +``` + +### `DB-CLUSTER-10052` + +**Message** + +```markdown +Unsupported embedding store type. Type: %s +``` + +### `DB-CLUSTER-10053` + +**Message** + +```markdown +Unsupported embedding model type. Type: %s +``` + +### `DB-CLUSTER-10054` + +**Message** + +```markdown +The filter is not set +``` + +### `DB-CLUSTER-10055` + +**Message** + +```markdown +Unsupported metadata value type. Type: %s +``` + +### `DB-CLUSTER-10056` + +**Message** + +```markdown +The metadata value is not set +``` + +## `DB-CLUSTER-2xxxx` status codes + +The following are status codes and messages for the concurrency error category. + +### `DB-CLUSTER-20000` + +**Message** + +```markdown +The hop limit is exceeded +``` + +### `DB-CLUSTER-20001` + +**Message** + +```markdown +A transaction associated with the specified transaction ID is not found. The transaction might have expired, or the cluster node that handled the transaction might have been restarted. Transaction ID: %s +``` + +### `DB-CLUSTER-20002` + +**Message** + +```markdown +A scanner associated with the specified scanner ID is not found. The scanner might have expired, or the cluster node that handled the scanner might have been restarted. Transaction ID: %s; Scanner ID: %s +``` + +## `DB-CLUSTER-3xxxx` status codes + +The following are status codes and messages for the internal error category. + +### `DB-CLUSTER-30000` + +**Message** + +```markdown +Getting local IP addresses failed +``` + +### `DB-CLUSTER-30001` + +**Message** + +```markdown +Getting a cluster node object from the cache failed. Cluster Node IP Address: %s +``` + +### `DB-CLUSTER-30002` + +**Message** + +```markdown +The ring is empty +``` + +### `DB-CLUSTER-30003` + +**Message** + +```markdown +Getting the Kubernetes API client failed +``` + +### `DB-CLUSTER-30004` + +**Message** + +```markdown +Reading the Kubernetes endpoint failed. Namespace: %s; Name: %s; Code: %d; Response Headers: %s; Response Body: %s +``` + +### `DB-CLUSTER-30005` + +**Message** + +```markdown +Configuring TLS failed +``` + +### `DB-CLUSTER-30006` + +**Message** + +```markdown +No nearest cluster nodes are found +``` diff --git a/versioned_docs/version-3.X/scalardb-cluster/setup-scalardb-cluster-on-kubernetes-by-using-helm-chart.mdx b/versioned_docs/version-3.X/scalardb-cluster/setup-scalardb-cluster-on-kubernetes-by-using-helm-chart.mdx new file mode 100644 index 00000000..bb4294e1 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster/setup-scalardb-cluster-on-kubernetes-by-using-helm-chart.mdx @@ -0,0 +1,256 @@ +--- +tags: + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# How to Deploy ScalarDB Cluster Locally + +import WarningLicenseKeyContact from '/src/components/en-us/_warning-license-key-contact.mdx'; + +This guide provides instructions on how to deploy ScalarDB Cluster by using a Helm Chart on a local Kubernetes cluster, specifically designed for a test environment. + +## Prerequisites + +- [Docker](https://www.docker.com/get-started/) 20.10 or later with [Docker Compose](https://docs.docker.com/compose/install/) V2 or later +- Kubernetes cluster (either [minikube](https://minikube.sigs.k8s.io/docs/start/) or [kind](https://kind.sigs.k8s.io/docs/user/quick-start/#installation)) +- [kubectl](https://kubernetes.io/docs/tasks/tools/#kubectl) +- [Helm](https://helm.sh/docs/intro/install/) + + + +## What you will create + +You will be deploying the following components on a local Kubernetes cluster as depicted below: + +``` ++---------------------------------------------------------------------------------------------------------------------------------------+ +| [Kubernetes Cluster] | +| | +| [Pod] [Pod] [Pod] | +| | +| +-------+ | +| +---> | Envoy | ---+ | +| | +-------+ | | +| | | | +| +---------+ | +-------+ | +--------------------+ | +| | Service | ---+---> | Envoy | ---+---------> | Service | ---+ | +| | (Envoy) | | +-------+ | | (ScalarDB Cluster) | | | +| +---------+ | | +--------------------+ | +-----------------------+ | +| | +-------+ | | +---> | ScalarDB Cluster Node | ---+ | +| +---> | Envoy | ---+ | | +-----------------------+ | | +| +-------+ | | | | +| | | +-----------------------+ | +------------+ | +| +---+---> | ScalarDB Cluster Node | ---+---> | PostgreSQL | | +| | | +-----------------------+ | +------------+ | +| | | | | +| | | +-----------------------+ | | +| | +---> | ScalarDB Cluster Node | ---+ | +| | +-----------------------+ | +| +----------------------------+ | | +| | Service | ---+ | +| | (ScalarDB Cluster GraphQL) | | +| +----------------------------+ | +| | ++---------------------------------------------------------------------------------------------------------------------------------------+ +``` + +## Step 1. Start a PostgreSQL container + +ScalarDB Cluster must use some kind of database system as its backend database. The database that is used in this guide is PostgreSQL. + +You can deploy PostgreSQL on the Kubernetes cluster as follows. + +1. Add the Bitnami Helm repository by running the following command: + + ```console + helm repo add bitnami https://charts.bitnami.com/bitnami + ``` + +2. Deploy PostgreSQL by running the following command: + + ```console + helm install postgresql-scalardb-cluster bitnami/postgresql \ + --set auth.postgresPassword=postgres \ + --set primary.persistence.enabled=false + ``` + +3. Check if the PostgreSQL container is running by running the following command: + + ```console + kubectl get pod + ``` + + You should see the following output: + + ```console + NAME READY STATUS RESTARTS AGE + postgresql-scalardb-cluster-0 1/1 Running 0 17s + ``` + +## Step 2. Deploy ScalarDB Cluster on the Kubernetes cluster by using a Helm Chart + +1. Add the Scalar Helm Charts repository by running the following command: + + ```console + helm repo add scalar-labs https://scalar-labs.github.io/helm-charts + ``` + +2. Set your license key and certificate as environment variables. If you don't have a license key, please [contact us](https://www.scalar-labs.com/contact). For details about the value for ``, see [How to Configure a Product License Key](../scalar-licensing/index.mdx). + + ```console + SCALAR_DB_CLUSTER_LICENSE_KEY='' + SCALAR_DB_CLUSTER_LICENSE_CHECK_CERT_PEM='' + ``` + +3. Create a custom values file for ScalarDB Cluster (`scalardb-cluster-custom-values.yaml`) by running the following command: + + ```console + cat << 'EOF' > scalardb-cluster-custom-values.yaml + envoy: + enabled: true + service: + type: "LoadBalancer" + + scalardbCluster: + + image: + repository: "ghcr.io/scalar-labs/scalardb-cluster-node-byol-premium" + + scalardbClusterNodeProperties: | + # ScalarDB Cluster configurations + scalar.db.cluster.membership.type=KUBERNETES + scalar.db.cluster.membership.kubernetes.endpoint.namespace_name=${env:SCALAR_DB_CLUSTER_MEMBERSHIP_KUBERNETES_ENDPOINT_NAMESPACE_NAME} + scalar.db.cluster.membership.kubernetes.endpoint.name=${env:SCALAR_DB_CLUSTER_MEMBERSHIP_KUBERNETES_ENDPOINT_NAME} + + # Storage configurations + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:postgresql://postgresql-scalardb-cluster.default.svc.cluster.local:5432/postgres + scalar.db.username=${env:SCALAR_DB_CLUSTER_POSTGRES_USERNAME} + scalar.db.password=${env:SCALAR_DB_CLUSTER_POSTGRES_PASSWORD} + + # For ScalarDB Cluster GraphQL tutorial. + scalar.db.graphql.enabled=true + scalar.db.graphql.namespaces=emoney + + # For ScalarDB Cluster SQL tutorial. + scalar.db.sql.enabled=true + + ### License key configurations + scalar.db.cluster.node.licensing.license_key=${env:SCALAR_DB_CLUSTER_LICENSE_KEY} + scalar.db.cluster.node.licensing.license_check_cert_pem=${env:SCALAR_DB_CLUSTER_LICENSE_CHECK_CERT_PEM} + graphql: + enabled: true + service: + type: "LoadBalancer" + + secretName: "scalardb-credentials-secret" + EOF + ``` + +:::note + + For the purpose of this guide, the service type for ScalarDB Cluster GraphQL and Envoy is set to `LoadBalancer`. + +::: + +4. Create a secret resource named `scalardb-credentials-secret` that includes credentials and license keys. + + ```console + kubectl create secret generic scalardb-credentials-secret \ + --from-literal=SCALAR_DB_CLUSTER_POSTGRES_USERNAME=postgres \ + --from-literal=SCALAR_DB_CLUSTER_POSTGRES_PASSWORD=postgres \ + --from-literal=SCALAR_DB_CLUSTER_LICENSE_KEY="${SCALAR_DB_CLUSTER_LICENSE_KEY}" \ + --from-file=SCALAR_DB_CLUSTER_LICENSE_CHECK_CERT_PEM=<(echo ${SCALAR_DB_CLUSTER_LICENSE_CHECK_CERT_PEM} | sed 's/\\n/\ + /g') \ + -n default + ``` + +5. Set the chart version of ScalarDB Cluster. + + ```console + SCALAR_DB_CLUSTER_VERSION=3.16.0 + SCALAR_DB_CLUSTER_CHART_VERSION=$(helm search repo scalar-labs/scalardb-cluster -l | grep -F "${SCALAR_DB_CLUSTER_VERSION}" | awk '{print $2}' | sort --version-sort -r | head -n 1) + ``` + +6. Deploy ScalarDB Cluster. + + ```console + helm install scalardb-cluster scalar-labs/scalardb-cluster -f scalardb-cluster-custom-values.yaml --version ${SCALAR_DB_CLUSTER_CHART_VERSION} -n default + ``` + +7. Check if the ScalarDB Cluster pods are deployed: + + ```console + kubectl get pod + ``` + + You should see the following output: + + ```console + NAME READY STATUS RESTARTS AGE + postgresql-scalardb-cluster-0 1/1 Running 0 84s + scalardb-cluster-envoy-59899dc588-477tg 1/1 Running 0 35s + scalardb-cluster-envoy-59899dc588-dpvhx 1/1 Running 0 35s + scalardb-cluster-envoy-59899dc588-lv9hx 1/1 Running 0 35s + scalardb-cluster-node-866c756c79-5v2tk 1/1 Running 0 35s + scalardb-cluster-node-866c756c79-9zhq5 1/1 Running 0 35s + scalardb-cluster-node-866c756c79-t6v86 1/1 Running 0 35s + ``` + + If the ScalarDB Cluster Node Pods and the Envoy Pods are deployed properly, the `STATUS` for each pod will be `Running`. + +6. Check if the service resources of ScalarDB Cluster are deployed by running the following command: + + ```console + kubectl get svc + ``` + + You should see the following output: + + ```console + NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE + kubernetes ClusterIP 10.96.0.1 443/TCP 260d + postgresql-scalardb-cluster ClusterIP 10.110.97.40 5432/TCP 86s + postgresql-scalardb-cluster-hl ClusterIP None 5432/TCP 86s + scalardb-cluster-envoy LoadBalancer 10.105.121.51 localhost 60053:30641/TCP 49s + scalardb-cluster-envoy-metrics ClusterIP 10.111.131.189 9001/TCP 49s + scalardb-cluster-graphql LoadBalancer 10.105.74.214 localhost 8080:30514/TCP 49s + scalardb-cluster-headless ClusterIP None 60053/TCP 49s + scalardb-cluster-metrics ClusterIP 10.110.132.22 9080/TCP 49s + ``` + + If the service resources of ScalarDB Cluster and Envoy are deployed properly, the private IP addresses in the `CLUSTER-IP` column will be displayed. + + :::note + + `scalardb-cluster-headless` has no `CLUSTER-IP` address. + + ::: + + You can also see `EXTERNAL-IP` addresses assigned to the service resource of ScalarDB Cluster GraphQL (`scalardb-cluster-graphql`) and the service resource of Envoy (`scalardb-cluster-envoy`) with `TYPE` set to `LoadBalancer`. + + In addition, the access method to the `LoadBalancer` service from your environment depends on each Kubernetes distribution. For example: + + - If you're using minikube, you can use the [`minikube tunnel` command](https://minikube.sigs.k8s.io/docs/commands/tunnel/). + - If you're using kind, you can use [Cloud Provider KIND](https://kind.sigs.k8s.io/docs/user/loadbalancer/). + + For details on how to access the `LoadBalancer` service, see the official documentation for the Kubernetes distribution that you're using. + +## Delete all resources + +You can delete all resources created in this guide by running the following command: + +```console +helm uninstall scalardb-cluster postgresql-scalardb-cluster +``` + +## Learn more + +To get familiar with other use cases for ScalarDB Cluster, try the following tutorials: + +- [Getting Started with ScalarDB Cluster](getting-started-with-scalardb-cluster.mdx) +- [Getting Started with ScalarDB Cluster GraphQL](getting-started-with-scalardb-cluster-graphql.mdx) +- [Getting Started with ScalarDB Cluster SQL via JDBC](getting-started-with-scalardb-cluster-sql-jdbc.mdx) +- [Getting Started with ScalarDB Cluster SQL via Spring Data JDBC for ScalarDB](getting-started-with-scalardb-cluster-sql-spring-data-jdbc.mdx) diff --git a/versioned_docs/version-3.X/scalardb-cluster/standalone-mode.mdx b/versioned_docs/version-3.X/scalardb-cluster/standalone-mode.mdx new file mode 100644 index 00000000..fdfb7be4 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-cluster/standalone-mode.mdx @@ -0,0 +1,236 @@ +--- +tags: + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# ScalarDB Cluster Standalone Mode + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; +import WarningLicenseKeyContact from '/src/components/en-us/_warning-license-key-contact.mdx'; + +Instead of setting up a Kubernetes cluster and deploying ScalarDB Cluster on top of it by using a Helm Chart, you can run ScalarDB Cluster in standalone mode, which simplifies development and testing processes. A primary use case for this would be when you want to start ScalarDB Cluster in standalone mode via Docker on your local machine and use it for development and testing. + +To run ScalarDB Cluster in standalone mode, you need to set the `scalar.db.cluster.node.standalone_mode.enabled` property to `true`: + +```properties +scalar.db.cluster.node.standalone_mode.enabled=true +``` + +## Run ScalarDB Cluster in standalone mode on Docker Compose + +This section explains how to start ScalarDB Cluster in standalone mode on Docker Compose. + + + +### Clone the ScalarDB samples repository + +Open **Terminal**, then clone the ScalarDB samples repository by running the following command: + +```console +git clone https://github.com/scalar-labs/scalardb-samples +``` + +Then, go to the directory that contains the necessary files by running the following command: + +```console +cd scalardb-samples/scalardb-cluster-standalone-mode/ +``` + +### Set up your database for ScalarDB Cluster + +Select your database, and follow the instructions to configure it for ScalarDB Cluster. + +For a list of databases that ScalarDB supports, see [Databases](../requirements.mdx#databases). + + + +

Run MySQL locally

+ + You can run MySQL in Docker Compose by using the `docker-compose.yaml` file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory. + + To start MySQL, run the following command: + + ```console + docker compose up -d mysql + ``` + +

Configure ScalarDB Cluster

+ + The **scalardb-cluster-node.properties** file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Please uncomment the properties for MySQL in the **scalardb-cluster-node.properties** file so that the configuration looks as follows: + + ```properties + # For MySQL + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:mysql://mysql-1:3306/ + scalar.db.username=root + scalar.db.password=mysql + ``` + + +

Run PostgreSQL locally

+ + You can run PostgreSQL in Docker Compose by using the `docker-compose.yaml` file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory. + + To start PostgreSQL, run the following command: + + ```console + docker compose up -d postgres + ``` + +

Configure ScalarDB Cluster

+ + The **scalardb-cluster-node.properties** file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Please uncomment the properties for PostgreSQL in the **scalardb-cluster-node.properties** file so that the configuration looks as follows: + + ```properties + # For PostgreSQL + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:postgresql://postgres-1:5432/ + scalar.db.username=postgres + scalar.db.password=postgres + ``` + + +

Run Oracle Database locally

+ + You can run Oracle Database in Docker Compose by using the `docker-compose.yaml` file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory. + + To start Oracle Database, run the following command: + + ```console + docker compose up -d oracle + ``` + +

Configure ScalarDB Cluster

+ + The **scalardb-cluster-node.properties** file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Please uncomment the properties for Oracle Database in the **scalardb-cluster-node.properties** file so that the configuration looks as follows: + + ```properties + # For Oracle + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:oracle:thin:@//oracle-1:1521/FREEPDB1 + scalar.db.username=SYSTEM + scalar.db.password=Oracle + ``` + + +

Run SQL Server locally

+ + You can run SQL Server in Docker Compose by using the `docker-compose.yaml` file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory. + + To start SQL Server, run the following command: + + ```console + docker compose up -d sqlserver + ``` + +

Configure ScalarDB Cluster

+ + The **scalardb-cluster-node.properties** file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Please uncomment the properties for SQL Server in the **scalardb-cluster-node.properties** file so that the configuration looks as follows: + + ```properties + # For SQL Server + scalar.db.storage=jdbc + scalar.db.contact_points=jdbc:sqlserver://sqlserver-1:1433;encrypt=true;trustServerCertificate=true + scalar.db.username=sa + scalar.db.password=SqlServer22 + ``` + + +

Run Amazon DynamoDB Local

+ + You can run Amazon DynamoDB Local in Docker Compose by using the `docker-compose.yaml` file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory. + + To start Amazon DynamoDB Local, run the following command: + + ```console + docker compose up -d dynamodb + ``` + +

Configure ScalarDB Cluster

+ + The **scalardb-cluster-node.properties** file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Please uncomment the properties for Amazon DynamoDB Local in the **scalardb-cluster-node.properties** file so that the configuration looks as follows: + + ```properties + # For DynamoDB Local + scalar.db.storage=dynamo + scalar.db.contact_points=sample + scalar.db.username=sample + scalar.db.password=sample + scalar.db.dynamo.endpoint_override=http://dynamodb-1:8000 + ``` + + + To use Azure Cosmos DB for NoSQL, you must have an Azure account. If you don't have an Azure account, visit [Create an Azure Cosmos DB account](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/quickstart-portal#create-account). + +

Configure Cosmos DB for NoSQL

+ + Set the **default consistency level** to **Strong** according to the official document at [Configure the default consistency level](https://learn.microsoft.com/en-us/azure/cosmos-db/nosql/how-to-manage-consistency#configure-the-default-consistency-level). + +

Configure ScalarDB Cluster

+ + The following instructions assume that you have properly installed and configured the JDK in your local environment and properly configured your Cosmos DB for NoSQL account in Azure. + + The **scalardb-cluster-node.properties** file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Be sure to change the values for `scalar.db.contact_points` and `scalar.db.password` as described. + + ```properties + # For Cosmos DB + scalar.db.storage=cosmos + scalar.db.contact_points= + scalar.db.password= + ``` + +:::note + +You can use the primary key or the secondary key in your Azure Cosmos DB account as the value for `scalar.db.password`. + +::: + + +

Run Cassandra locally

+ + You can run Apache Cassandra in Docker Compose by using the `docker-compose.yaml` file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory. + + To start Apache Cassandra, run the following command: + + ```console + docker compose up -d cassandra + ``` + +

Configure ScalarDB Cluster

+ + The **scalardb-cluster-node.properties** file in the `scalardb-samples/scalardb-cluster-standalone-mode` directory contains database configurations for ScalarDB Cluster. Please uncomment the properties for Cassandra in the **scalardb-cluster-node.properties** file so that the configuration looks as follows: + + ```properties + # For Cassandra + scalar.db.storage=cassandra + scalar.db.contact_points=cassandra-1 + scalar.db.username=cassandra + scalar.db.password=cassandra + ``` + + + +### Set the license key + +Set the license key (trial license or commercial license) for the ScalarDB Clusters in the configuration file `scalardb-cluster-node.properties`. For details, see [How to Configure a Product License Key](../scalar-licensing/index.mdx). + +### Start ScalarDB Cluster in standalone mode + +To start ScalarDB Cluster in standalone mode, run the following command: + +:::note + +If you want to change other configurations for ScalarDB Cluster, update the `scalardb-cluster-node.properties` file before running the command below. + +::: + +```console +docker compose up -d scalardb-cluster-node +``` + +## Client configurations for the ScalarDB Cluster Java API + +You can use the `indirect` client mode to connect to ScalarDB Cluster in standalone mode. For details about client configurations for the ScalarDB Cluster Java API, see [Developer Guide for ScalarDB Cluster with the Java API](developer-guide-for-scalardb-cluster-with-java-api.mdx). diff --git a/versioned_docs/version-3.X/scalardb-core-status-codes.mdx b/versioned_docs/version-3.X/scalardb-core-status-codes.mdx new file mode 100644 index 00000000..bbb3a1f6 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-core-status-codes.mdx @@ -0,0 +1,1796 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# ScalarDB Core Error Codes + +This page provides a list of error codes in ScalarDB Core. + +## Error code classes and descriptions + +| Class | Description | +|:----------------|:---------------------------------------------------------| +| `DB-CORE-1xxxx` | Errors for the user error category | +| `DB-CORE-2xxxx` | Errors for the concurrency error category | +| `DB-CORE-3xxxx` | Errors for the internal error category | +| `DB-CORE-4xxxx` | Errors for the unknown transaction status error category | + +## `DB-CORE-1xxxx` status codes + +The following are status codes and messages for the user error category. + +### `DB-CORE-10000` + +**Message** + +```markdown +Only a single-column index is supported. Operation: %s +``` + +### `DB-CORE-10001` + +**Message** + +```markdown +The column of the specified index key is not indexed. Operation: %s +``` + +### `DB-CORE-10002` + +**Message** + +```markdown +The index key is not properly specified. Operation: %s +``` + +### `DB-CORE-10003` + +**Message** + +```markdown +Clustering keys cannot be specified when using an index. Operation: %s +``` + +### `DB-CORE-10004` + +**Message** + +```markdown +Orderings cannot be specified when using an index. Operation: %s +``` + +### `DB-CORE-10005` + +**Message** + +```markdown +The limit cannot be negative. Operation: %s +``` + +### `DB-CORE-10006` + +**Message** + +```markdown +Cross-partition scan is not enabled. Operation: %s +``` + +### `DB-CORE-10007` + +**Message** + +```markdown +Cross-partition scan ordering is not enabled. Operation: %s +``` + +### `DB-CORE-10008` + +**Message** + +```markdown +Cross-partition scan filtering is not enabled. Operation: %s +``` + +### `DB-CORE-10009` + +**Message** + +```markdown +The specified projection is not found. Projection: %s, Operation: %s +``` + +### `DB-CORE-10010` + +**Message** + +```markdown +The clustering key boundary is not properly specified. Operation: %s +``` + +### `DB-CORE-10011` + +**Message** + +```markdown +The start clustering key is not properly specified. Operation: %s +``` + +### `DB-CORE-10012` + +**Message** + +```markdown +The end clustering key is not properly specified. Operation: %s +``` + +### `DB-CORE-10013` + +**Message** + +```markdown +Orderings are not properly specified. Operation: %s +``` + +### `DB-CORE-10014` + +**Message** + +```markdown +The specified ordering column is not found. Ordering: %s, Operation: %s +``` + +### `DB-CORE-10015` + +**Message** + +```markdown +The condition is not properly specified. Operation: %s +``` + +### `DB-CORE-10016` + +**Message** + +```markdown +The table does not exist. Table: %s +``` + +### `DB-CORE-10017` + +**Message** + +```markdown +The column value is not properly specified. Column: %s, Operation: %s +``` + +### `DB-CORE-10018` + +**Message** + +```markdown +The mutations are empty +``` + +### `DB-CORE-10019` + +**Message** + +```markdown +The storage does not support mutations across multiple partitions. Storage: %s; Mutations: %s +``` + +### `DB-CORE-10020` + +**Message** + +```markdown +The partition key is not properly specified. Operation: %s +``` + +### `DB-CORE-10021` + +**Message** + +```markdown +The clustering key is not properly specified. Operation: %s +``` + +### `DB-CORE-10022` + +**Message** + +```markdown +The authentication and authorization feature is not enabled. To use this feature, you must enable it. Note that this feature is supported only in the ScalarDB Enterprise edition +``` + +### `DB-CORE-10023` + +**Message** + +```markdown +This condition is not allowed for the PutIf operation. Condition: %s +``` + +### `DB-CORE-10024` + +**Message** + +```markdown +This condition is not allowed for the DeleteIf operation. Condition: %s +``` + +### `DB-CORE-10025` + +**Message** + +```markdown +Operator must be LIKE or NOT_LIKE. Operator: %s +``` + +### `DB-CORE-10026` + +**Message** + +```markdown +An escape character must be a string of a single character or an empty string +``` + +### `DB-CORE-10027` + +**Message** + +```markdown +The LIKE pattern must not be null +``` + +### `DB-CORE-10028` + +**Message** + +```markdown +The LIKE pattern must not include only an escape character +``` + +### `DB-CORE-10029` + +**Message** + +```markdown +The LIKE pattern must not end with an escape character +``` + +### `DB-CORE-10030` + +**Message** + +```markdown +The column %s does not exist +``` + +### `DB-CORE-10031` + +**Message** + +```markdown +This operation is not supported when getting records of a database without using an index +``` + +### `DB-CORE-10032` + +**Message** + +```markdown +This operation is not supported when getting records of a database by using an index +``` + +### `DB-CORE-10033` + +**Message** + +```markdown +This operation is not supported when scanning all the records of a database or scanning records of a database by using an index +``` + +### `DB-CORE-10034` + +**Message** + +```markdown +This operation is supported only when scanning records of a database by using an index +``` + +### `DB-CORE-10035` + +**Message** + +```markdown +This operation is not supported when scanning records of a database by using an index +``` + +### `DB-CORE-10037` + +**Message** + +```markdown +This operation is supported only when no conditions are specified. If you want to modify a condition, please use clearConditions() to remove all existing conditions first +``` + +### `DB-CORE-10038` + +**Message** + +```markdown +One or more columns must be specified +``` + +### `DB-CORE-10039` + +**Message** + +```markdown +One or more partition keys must be specified +``` + +### `DB-CORE-10040` + +**Message** + +```markdown +The column definition must be specified since %s is specified as a partition key +``` + +### `DB-CORE-10041` + +**Message** + +```markdown +The column definition must be specified since %s is specified as a clustering key +``` + +### `DB-CORE-10042` + +**Message** + +```markdown +Invalid ID specified. ID: %d +``` + +### `DB-CORE-10043` + +**Message** + +```markdown +The transaction is not active. Status: %s +``` + +### `DB-CORE-10044` + +**Message** + +```markdown +The transaction has already been committed. Status: %s +``` + +### `DB-CORE-10045` + +**Message** + +```markdown +The transaction has not been prepared. Status: %s +``` + +### `DB-CORE-10046` + +**Message** + +```markdown +The transaction has not been prepared or validated. Status: %s +``` + +### `DB-CORE-10047` + +**Message** + +```markdown +The transaction already exists +``` + +### `DB-CORE-10048` + +**Message** + +```markdown +A transaction associated with the specified transaction ID is not found. The transaction might have expired +``` + +### `DB-CORE-10049` + +**Message** + +```markdown +%s is the system namespace name +``` + +### `DB-CORE-10050` + +**Message** + +```markdown +The namespace already exists. Namespace: %s +``` + +### `DB-CORE-10051` + +**Message** + +```markdown +The namespace does not exist. Namespace: %s +``` + +### `DB-CORE-10052` + +**Message** + +```markdown +The table already exists. Table: %s +``` + +### `DB-CORE-10053` + +**Message** + +```markdown +The namespace is not empty. Namespace: %s; Tables in the namespace: %s +``` + +### `DB-CORE-10054` + +**Message** + +```markdown +The column does not exist. Table: %s; Column: %s +``` + +### `DB-CORE-10055` + +**Message** + +```markdown +The index already exists. Table: %s; Column: %s +``` + +### `DB-CORE-10056` + +**Message** + +```markdown +The index does not exist. Table: %s; Column: %s +``` + +### `DB-CORE-10057` + +**Message** + +```markdown +The column already exists. Table: %s; Column: %s +``` + +### `DB-CORE-10058` + +**Message** + +```markdown +The operation does not have the target namespace or table name. Operation: %s +``` + +### `DB-CORE-10059` + +**Message** + +```markdown +The specified value of the property '%s' is not a number. Value: %s +``` + +### `DB-CORE-10060` + +**Message** + +```markdown +The specified value of the property '%s' is not a boolean. Value: %s +``` + +### `DB-CORE-10061` + +**Message** + +```markdown +Reading the file failed. File: %s +``` + +### `DB-CORE-10062` + +**Message** + +```markdown +The property 'scalar.db.cross_partition_scan.enabled' must be set to true to use cross-partition scan with filtering or ordering +``` + +### `DB-CORE-10063` + +**Message** + +```markdown +This column value is out of range for BigInt. Value: %s +``` + +### `DB-CORE-10064` + +**Message** + +```markdown +This type is not supported. Name: %s, Type: %s +``` + +### `DB-CORE-10065` + +**Message** + +```markdown +Storage '%s' is not found +``` + +### `DB-CORE-10066` + +**Message** + +```markdown +Transaction manager '%s' is not found +``` + +### `DB-CORE-10068` + +**Message** + +```markdown +Please use scan() for non-exact match selection. Operation: %s +``` + +### `DB-CORE-10069` + +**Message** + +```markdown +Import-related functionality is not supported in Cassandra +``` + +### `DB-CORE-10070` + +**Message** + +```markdown +The %s network strategy does not exist +``` + +### `DB-CORE-10071` + +**Message** + +```markdown +The property 'scalar.db.contact_port' must be greater than or equal to zero +``` + +### `DB-CORE-10073` + +**Message** + +```markdown +The BLOB type is not supported for clustering keys in Cosmos DB. Column: %s +``` + +### `DB-CORE-10074` + +**Message** + +```markdown +Import-related functionality is not supported in Cosmos DB +``` + +### `DB-CORE-10075` + +**Message** + +```markdown +The property 'scalar.db.contact_points' must not be empty +``` + +### `DB-CORE-10076` + +**Message** + +```markdown +Cosmos DB supports only EQ, NE, IS_NULL, and IS_NOT_NULL operations for the BLOB type in conditions. Mutation: %s +``` + +### `DB-CORE-10077` + +**Message** + +```markdown +The specified consistency level is not supported. Consistency level: %s +``` + +### `DB-CORE-10078` + +**Message** + +```markdown +0x00 bytes are not accepted in BLOB values in DESC order +``` + +### `DB-CORE-10079` + +**Message** + +```markdown +Cannot encode a Text value that contains '\u0000' +``` + +### `DB-CORE-10081` + +**Message** + +```markdown +An index column cannot be set to null or an empty value for Text or Blob in DynamoDB. Operation: %s +``` + +### `DB-CORE-10082` + +**Message** + +```markdown +DynamoDB supports only EQ, NE, IS_NULL, and IS_NOT_NULL operations for the BOOLEAN type in conditions. Mutation: %s +``` + +### `DB-CORE-10083` + +**Message** + +```markdown +Nested multi-storage definitions are not supported. Storage: %s +``` + +### `DB-CORE-10084` + +**Message** + +```markdown +Storage not found. Storage: %s +``` + +### `DB-CORE-10085` + +**Message** + +```markdown +The namespace name is not acceptable. Namespace: %s +``` + +### `DB-CORE-10086` + +**Message** + +```markdown +The table name is not acceptable. Table: %s +``` + +### `DB-CORE-10087` + +**Message** + +```markdown +Importing tables is not allowed in the RDB engine. RDB engine: %s +``` + +### `DB-CORE-10088` + +**Message** + +```markdown +The %s table must have a primary key +``` + +### `DB-CORE-10089` + +**Message** + +```markdown +The RDB engine is not supported. JDBC connection URL: %s +``` + +### `DB-CORE-10090` + +**Message** + +```markdown +Data type %s(%d) is not supported: %s +``` + +### `DB-CORE-10091` + +**Message** + +```markdown +Data type %s is not supported: %s +``` + +### `DB-CORE-10092` + +**Message** + +```markdown +Getting a transaction state is not supported in JDBC transactions +``` + +### `DB-CORE-10093` + +**Message** + +```markdown +Rolling back a transaction is not supported in JDBC transactions +``` + +### `DB-CORE-10094` + +**Message** + +```markdown +Coordinator tables already exist +``` + +### `DB-CORE-10095` + +**Message** + +```markdown +Coordinator tables do not exist +``` + +### `DB-CORE-10096` + +**Message** + +```markdown +The namespace %s is reserved. Any operations on this namespace are not allowed +``` + +### `DB-CORE-10097` + +**Message** + +```markdown +Mutating transaction metadata columns is not allowed. Table: %s; Column: %s +``` + +### `DB-CORE-10098` + +**Message** + +```markdown +A %s condition is not allowed on Put operations +``` + +### `DB-CORE-10099` + +**Message** + +```markdown +A %s condition is not allowed on Delete operations +``` + +### `DB-CORE-10100` + +**Message** + +```markdown +The condition is not allowed to target transaction metadata columns. Column: %s +``` + +### `DB-CORE-10101` + +**Message** + +```markdown +The column '%s' is reserved as transaction metadata +``` + +### `DB-CORE-10102` + +**Message** + +```markdown +Non-primary key columns with the 'before_' prefix, '%s', are reserved as transaction metadata +``` + +### `DB-CORE-10103` + +**Message** + +```markdown +Put cannot have a condition when the target record is unread and implicit pre-read is disabled. Please read the target record beforehand or enable implicit pre-read: %s +``` + +### `DB-CORE-10104` + +**Message** + +```markdown +Writing data already-deleted by the same transaction is not allowed +``` + +### `DB-CORE-10106` + +**Message** + +```markdown +Scanning data already-written or already-deleted by the same transaction is not allowed +``` + +### `DB-CORE-10107` + +**Message** + +```markdown +The transaction is not validated. When using the SERIALIZABLE isolation level, you need to call validate() before calling commit() +``` + +### `DB-CORE-10108` + +**Message** + +```markdown +DynamoDB cannot batch more than 100 mutations at once +``` + +### `DB-CORE-10126` + +**Message** + +```markdown +The mutation type is not supported. Only the Put or Delete type is supported. Mutation: %s +``` + +### `DB-CORE-10127` + +**Message** + +```markdown +This condition is not allowed for the UpdateIf operation. Condition: %s +``` + +### `DB-CORE-10128` + +**Message** + +```markdown +Cross-partition scan with ordering is not supported in Cassandra +``` + +### `DB-CORE-10129` + +**Message** + +```markdown +Cross-partition scan with ordering is not supported in Cosmos DB +``` + +### `DB-CORE-10130` + +**Message** + +```markdown +Cross-partition scan with ordering is not supported in DynamoDB +``` + +### `DB-CORE-10136` + +**Message** + +```markdown +Getting a transaction state is not supported in single CRUD operation transactions +``` + +### `DB-CORE-10137` + +**Message** + +```markdown +Rolling back a transaction is not supported in single CRUD operation transactions +``` + +### `DB-CORE-10138` + +**Message** + +```markdown +Multiple mutations are not supported in single CRUD operation transactions +``` + +### `DB-CORE-10139` + +**Message** + +```markdown +Beginning a transaction is not allowed in single CRUD operation transactions +``` + +### `DB-CORE-10140` + +**Message** + +```markdown +Resuming a transaction is not allowed in single CRUD operation transactions +``` + +### `DB-CORE-10141` + +**Message** + +```markdown +Using the group commit feature on the Coordinator table with a two-phase commit interface is not allowed +``` + +### `DB-CORE-10142` + +**Message** + +```markdown +This operation is supported only when no conditions are specified. If you want to modify a condition, please use clearConditions() to remove all existing conditions first +``` + +### `DB-CORE-10143` + +**Message** + +```markdown +The encryption feature is not enabled. To encrypt data at rest, you must enable this feature. Note that this feature is supported only in the ScalarDB Enterprise edition +``` + +### `DB-CORE-10144` + +**Message** + +```markdown +The variable key column size must be greater than or equal to 64 +``` + +### `DB-CORE-10145` + +**Message** + +```markdown +The value of the column %s in the primary key contains an illegal character. Primary-key columns must not contain any of the following characters in Cosmos DB: ':', '/', '\', '#', '?'. Value: %s +``` + +### `DB-CORE-10146` + +**Message** + +```markdown +Inserting data already-written by the same transaction is not allowed +``` + +### `DB-CORE-10147` + +**Message** + +```markdown +Deleting data already-inserted by the same transaction is not allowed +``` + +### `DB-CORE-10152` + +**Message** + +```markdown +The attribute-based access control feature is not enabled. To use this feature, you must enable it. Note that this feature is supported only in the ScalarDB Enterprise edition +``` + +### `DB-CORE-10158` + +**Message** + +```markdown +This DATE column value is out of the valid range. It must be between 1000-01-01 and 9999-12-12. Value: %s +``` + +### `DB-CORE-10159` + +**Message** + +```markdown +This TIME column value precision cannot be shorter than one microsecond. Value: %s +``` + +### `DB-CORE-10160` + +**Message** + +```markdown +This TIMESTAMP column value is out of the valid range. It must be between 1000-01-01T00:00:00.000 and 9999-12-31T23:59:59.999. Value: %s +``` + +### `DB-CORE-10161` + +**Message** + +```markdown +This TIMESTAMP column value precision cannot be shorter than one millisecond. Value: %s +``` + +### `DB-CORE-10162` + +**Message** + +```markdown +This TIMESTAMPTZ column value is out of the valid range. It must be between 1000-01-01T00:00:00.000Z to 9999-12-31T23:59:59.999Z. Value: %s +``` + +### `DB-CORE-10163` + +**Message** + +```markdown +This TIMESTAMPTZ column value precision cannot be shorter than one millisecond. Value: %s +``` + +### `DB-CORE-10164` + +**Message** + +```markdown +The underlying-storage data type %s is not supported as the ScalarDB %s data type: %s +``` + +### `DB-CORE-10188` + +**Message** + +```markdown +The replication feature is not enabled. To use this feature, you must enable it. Note that this feature is supported only in the ScalarDB Enterprise edition +``` + +### `DB-CORE-10205` + +**Message** + +```markdown +Some scanners were not closed. All scanners must be closed before committing the transaction +``` + +### `DB-CORE-10206` + +**Message** + +```markdown +Some scanners were not closed. All scanners must be closed before preparing the transaction +``` + +### `DB-CORE-10211` + +**Message** + +```markdown +Mutations are not allowed in read-only transactions. Transaction ID: %s +``` + +### `DB-CORE-10212` + +**Message** + +```markdown +The storage does not support mutations across multiple records. Storage: %s; Mutations: %s +``` + +### `DB-CORE-10213` + +**Message** + +```markdown +The storage does not support mutations across multiple tables. Storage: %s; Mutations: %s +``` + +### `DB-CORE-10214` + +**Message** + +```markdown +The storage does not support mutations across multiple namespaces. Storage: %s; Mutations: %s +``` + +### `DB-CORE-10215` + +**Message** + +```markdown +Mutations across multiple storages are not allowed. Mutations: %s +``` + +## `DB-CORE-2xxxx` status codes + +The following are status codes and messages for the concurrency error category. + +### `DB-CORE-20000` + +**Message** + +```markdown +No mutation was applied +``` + +### `DB-CORE-20001` + +**Message** + +```markdown +Logging failed in the batch +``` + +### `DB-CORE-20002` + +**Message** + +```markdown +The operation failed in the batch with type %s +``` + +### `DB-CORE-20003` + +**Message** + +```markdown +An error occurred in the batch. Details: %s +``` + +### `DB-CORE-20004` + +**Message** + +```markdown +A Paxos phase in the CAS operation failed +``` + +### `DB-CORE-20005` + +**Message** + +```markdown +The learn phase in the CAS operation failed +``` + +### `DB-CORE-20006` + +**Message** + +```markdown +A simple write operation failed +``` + +### `DB-CORE-20007` + +**Message** + +```markdown +An error occurred in the mutation. Details: %s +``` + +### `DB-CORE-20008` + +**Message** + +```markdown +A RetryWith error occurred in the mutation. Details: %s +``` + +### `DB-CORE-20009` + +**Message** + +```markdown +A transaction conflict occurred in the mutation. Details: %s +``` + +### `DB-CORE-20010` + +**Message** + +```markdown +A transaction conflict occurred in the mutation. Details: %s +``` + +### `DB-CORE-20011` + +**Message** + +```markdown +A conflict occurred. Please try restarting the transaction. Details: %s +``` + +### `DB-CORE-20012` + +**Message** + +```markdown +The %s condition of the %s operation is not satisfied. Targeting column(s): %s +``` + +### `DB-CORE-20013` + +**Message** + +```markdown +The record being prepared already exists +``` + +### `DB-CORE-20014` + +**Message** + +```markdown +A conflict occurred when preparing records +``` + +### `DB-CORE-20015` + +**Message** + +```markdown +The committing state in the coordinator failed. The transaction has been aborted +``` + +### `DB-CORE-20016` + +**Message** + +```markdown +A conflict occurred during implicit pre-read +``` + +### `DB-CORE-20017` + +**Message** + +```markdown +This record needs to be recovered +``` + +### `DB-CORE-20018` + +**Message** + +```markdown +The record does not exist, so the %s condition is not satisfied +``` + +### `DB-CORE-20019` + +**Message** + +```markdown +The record exists, so the %s condition is not satisfied +``` + +### `DB-CORE-20020` + +**Message** + +```markdown +The condition on the column '%s' is not satisfied +``` + +### `DB-CORE-20022` + +**Message** + +```markdown +An anti-dependency was found. The transaction has been aborted +``` + +### `DB-CORE-20023` + +**Message** + +```markdown +A transaction conflict occurred in the Insert operation +``` + +### `DB-CORE-20024` + +**Message** + +```markdown +The %s condition of the %s operation is not satisfied. Targeting column(s): %s +``` + +### `DB-CORE-20025` + +**Message** + +```markdown +A transaction conflict occurred in the Insert operation +``` + +### `DB-CORE-20026` + +**Message** + +```markdown +A conflict occurred when committing records +``` + +## `DB-CORE-3xxxx` status codes + +The following are status codes and messages for the internal error category. + +### `DB-CORE-30000` + +**Message** + +```markdown +Creating the namespace failed. Namespace: %s +``` + +### `DB-CORE-30001` + +**Message** + +```markdown +Dropping the namespace failed. Namespace: %s +``` + +### `DB-CORE-30002` + +**Message** + +```markdown +Creating the table failed. Table: %s +``` + +### `DB-CORE-30003` + +**Message** + +```markdown +Dropping the table failed. Table: %s +``` + +### `DB-CORE-30004` + +**Message** + +```markdown +Truncating the table failed. Table: %s +``` + +### `DB-CORE-30005` + +**Message** + +```markdown +Creating the index failed. Table: %s, Column: %s +``` + +### `DB-CORE-30006` + +**Message** + +```markdown +Dropping the index failed. Table: %s, Column: %s +``` + +### `DB-CORE-30007` + +**Message** + +```markdown +Getting the table metadata failed. Table: %s +``` + +### `DB-CORE-30008` + +**Message** + +```markdown +Getting the table names in the namespace failed. Namespace: %s +``` + +### `DB-CORE-30009` + +**Message** + +```markdown +Checking the namespace existence failed. Namespace: %s +``` + +### `DB-CORE-30010` + +**Message** + +```markdown +Checking the table existence failed. Table: %s +``` + +### `DB-CORE-30011` + +**Message** + +```markdown +Checking the index existence failed. Table: %s; Column: %s +``` + +### `DB-CORE-30012` + +**Message** + +```markdown +Repairing the namespace failed. Namespace: %s +``` + +### `DB-CORE-30013` + +**Message** + +```markdown +Repairing the table failed. Table: %s +``` + +### `DB-CORE-30014` + +**Message** + +```markdown +Adding a new column to the table failed. Table: %s; Column: %s; ColumnType: %s +``` + +### `DB-CORE-30015` + +**Message** + +```markdown +Getting the namespace names failed +``` + +### `DB-CORE-30016` + +**Message** + +```markdown +Getting the table metadata of the table being imported failed. Table: %s +``` + +### `DB-CORE-30017` + +**Message** + +```markdown +Importing the table failed. Table: %s +``` + +### `DB-CORE-30018` + +**Message** + +```markdown +Adding the raw column to the table failed. Table: %s; Column: %s; ColumnType: %s +``` + +### `DB-CORE-30019` + +**Message** + +```markdown +Upgrading the ScalarDB environment failed +``` + +### `DB-CORE-30020` + +**Message** + +```markdown +Something wrong because WriteType is neither CAS nor SIMPLE +``` + +### `DB-CORE-30021` + +**Message** + +```markdown +An error occurred in the selection. Details: %s +``` + +### `DB-CORE-30022` + +**Message** + +```markdown +An error occurred in the mutation. Details: %s +``` + +### `DB-CORE-30023` + +**Message** + +```markdown +An error occurred in the selection. Details: %s +``` + +### `DB-CORE-30024` + +**Message** + +```markdown +An error occurred in the mutation. Details: %s +``` + +### `DB-CORE-30025` + +**Message** + +```markdown +An error occurred in the selection. Details: %s +``` + +### `DB-CORE-30026` + +**Message** + +```markdown +An error occurred in the mutation. Details: %s +``` + +### `DB-CORE-30027` + +**Message** + +```markdown +An error occurred in the selection. Details: %s +``` + +### `DB-CORE-30028` + +**Message** + +```markdown +Fetching the next result failed. Details: %s +``` + +### `DB-CORE-30029` + +**Message** + +```markdown +Rolling back the transaction failed. Details: %s +``` + +### `DB-CORE-30030` + +**Message** + +```markdown +Committing the transaction failed. Details: %s +``` + +### `DB-CORE-30031` + +**Message** + +```markdown +The Get operation failed. Details: %s +``` + +### `DB-CORE-30032` + +**Message** + +```markdown +The Scan operation failed. Details: %s +``` + +### `DB-CORE-30033` + +**Message** + +```markdown +The Put operation failed. Details: %s +``` + +### `DB-CORE-30034` + +**Message** + +```markdown +The Delete operation failed. Details: %s +``` + +### `DB-CORE-30035` + +**Message** + +```markdown +Beginning a transaction failed. Details: %s +``` + +### `DB-CORE-30036` + +**Message** + +```markdown +Preparing records failed +``` + +### `DB-CORE-30037` + +**Message** + +```markdown +Validation failed +``` + +### `DB-CORE-30038` + +**Message** + +```markdown +Executing implicit pre-read failed +``` + +### `DB-CORE-30039` + +**Message** + +```markdown +Reading a record from the underlying storage failed +``` + +### `DB-CORE-30040` + +**Message** + +```markdown +Scanning records from the underlying storage failed +``` + +### `DB-CORE-30041` + +**Message** + +```markdown +Rollback failed because the transaction has already been committed +``` + +### `DB-CORE-30042` + +**Message** + +```markdown +Rollback failed +``` + +### `DB-CORE-30043` + +**Message** + +```markdown +The Insert operation failed. Details: %s +``` + +### `DB-CORE-30044` + +**Message** + +```markdown +The Upsert operation failed. Details: %s +``` + +### `DB-CORE-30045` + +**Message** + +```markdown +The Update operation failed. Details: %s +``` + +### `DB-CORE-30046` + +**Message** + +```markdown +Handling the before-preparation snapshot hook failed. Details: %s +``` + +### `DB-CORE-30054` + +**Message** + +```markdown +Getting the scanner failed. Details: %s +``` + +### `DB-CORE-30055` + +**Message** + +```markdown +Closing the scanner failed. Details: %s +``` + +### `DB-CORE-30056` + +**Message** + +```markdown +Getting the storage information failed. Namespace: %s +``` + +### `DB-CORE-30057` + +**Message** + +```markdown +Recovering records failed. Details: %s +``` + +### `DB-CORE-30058` + +**Message** + +```markdown +Committing records failed +``` + +## `DB-CORE-4xxxx` status codes + +The following are status codes and messages for the unknown transaction status error category. + +### `DB-CORE-40000` + +**Message** + +```markdown +Rolling back the transaction failed. Details: %s +``` + +### `DB-CORE-40001` + +**Message** + +```markdown +Committing state failed with NoMutationException, but the coordinator status does not exist +``` + +### `DB-CORE-40002` + +**Message** + +```markdown +The state cannot be retrieved +``` + +### `DB-CORE-40003` + +**Message** + +```markdown +The coordinator status is unknown +``` + +### `DB-CORE-40004` + +**Message** + +```markdown +Aborting state failed with NoMutationException, but the coordinator status does not exist +``` diff --git a/versioned_docs/version-3.X/scalardb-data-loader-status-codes.mdx b/versioned_docs/version-3.X/scalardb-data-loader-status-codes.mdx new file mode 100644 index 00000000..67b0012d --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-data-loader-status-codes.mdx @@ -0,0 +1,538 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# ScalarDB Data Loader Error Codes + +This page provides a list of error codes in ScalarDB Data Loader. + +## Error code classes and descriptions + +| Class | Description | +|:-----------------------|:---------------------------------------| +| `DB-DATA-LOADER-1xxxx` | Errors for the user error category | +| `DB-DATA-LOADER-3xxxx` | Errors for the internal error category | + +## `DB-DATA-LOADER-1xxxx` status codes + +The following are status codes and messages for the user error category. + +### `DB-DATA-LOADER-10000` + +**Message** + +```markdown +Data chunk queue size must be greater than 0 +``` + +### `DB-DATA-LOADER-10001` + +**Message** + +```markdown +The directory '%s' does not have write permissions. Please ensure that the current user has write access to the directory +``` + +### `DB-DATA-LOADER-10002` + +**Message** + +```markdown +Failed to create the directory '%s'. Please check if you have sufficient permissions and if there are any file system restrictions. Details: %s +``` + +### `DB-DATA-LOADER-10003` + +**Message** + +```markdown +Directory path cannot be null or empty +``` + +### `DB-DATA-LOADER-10004` + +**Message** + +```markdown +No file extension was found in the provided file name %s +``` + +### `DB-DATA-LOADER-10005` + +**Message** + +```markdown +Invalid file extension: %s. Allowed extensions are: %s +``` + +### `DB-DATA-LOADER-10006` + +**Message** + +```markdown +Invalid key: Column %s does not exist in the table %s in namespace %s +``` + +### `DB-DATA-LOADER-10007` + +**Message** + +```markdown +Invalid base64 encoding for blob value '%s' for column %s in table %s in namespace %s +``` + +### `DB-DATA-LOADER-10008` + +**Message** + +```markdown +Invalid number '%s' specified for column %s in table %s in namespace %s +``` + +### `DB-DATA-LOADER-10009` + +**Message** + +```markdown +Method null argument not allowed +``` + +### `DB-DATA-LOADER-10010` + +**Message** + +```markdown +The provided clustering key %s was not found +``` + +### `DB-DATA-LOADER-10011` + +**Message** + +```markdown +The column '%s' was not found +``` + +### `DB-DATA-LOADER-10012` + +**Message** + +```markdown +The provided partition key is incomplete. Required key: %s +``` + +### `DB-DATA-LOADER-10013` + +**Message** + +```markdown +The provided clustering-key order does not match the table schema. Required order: %s +``` + +### `DB-DATA-LOADER-10014` + +**Message** + +```markdown +The provided partition-key order does not match the table schema. Required order: %s +``` + +### `DB-DATA-LOADER-10015` + +**Message** + +```markdown +Missing namespace or table: %s, %s +``` + +### `DB-DATA-LOADER-10016` + +**Message** + +```markdown +Failed to retrieve table metadata. Details: %s +``` + +### `DB-DATA-LOADER-10017` + +**Message** + +```markdown +Duplicate data mappings found for table '%s' in the control file +``` + +### `DB-DATA-LOADER-10018` + +**Message** + +```markdown +No mapping found for column '%s' in table '%s' in the control file. Control file validation set at 'FULL'. All columns need to be mapped +``` + +### `DB-DATA-LOADER-10019` + +**Message** + +```markdown +The control file is missing data mappings +``` + +### `DB-DATA-LOADER-10020` + +**Message** + +```markdown +The target column '%s' for source field '%s' could not be found in table '%s' +``` + +### `DB-DATA-LOADER-10021` + +**Message** + +```markdown +The required partition key '%s' is missing in the control file mapping for table '%s' +``` + +### `DB-DATA-LOADER-10022` + +**Message** + +```markdown +The required clustering key '%s' is missing in the control file mapping for table '%s' +``` + +### `DB-DATA-LOADER-10023` + +**Message** + +```markdown +Duplicated data mappings found for column '%s' in table '%s' +``` + +### `DB-DATA-LOADER-10024` + +**Message** + +```markdown +Missing required field or column mapping for clustering key %s +``` + +### `DB-DATA-LOADER-10025` + +**Message** + +```markdown +Missing required field or column mapping for partition key %s +``` + +### `DB-DATA-LOADER-10026` + +**Message** + +```markdown +Missing field or column mapping for %s +``` + +### `DB-DATA-LOADER-10027` + +**Message** + +```markdown +Something went wrong while converting the ScalarDB values to strings. The table metadata and Value datatype probably do not match. Details: %s +``` + +### `DB-DATA-LOADER-10028` + +**Message** + +```markdown +The provided file format is not supported : %s +``` + +### `DB-DATA-LOADER-10029` + +**Message** + +```markdown +Could not find the partition key +``` + +### `DB-DATA-LOADER-10030` + +**Message** + +```markdown +The source record needs to contain all fields if the UPSERT turns into an INSERT +``` + +### `DB-DATA-LOADER-10031` + +**Message** + +```markdown +Record already exists +``` + +### `DB-DATA-LOADER-10032` + +**Message** + +```markdown +Record was not found +``` + +### `DB-DATA-LOADER-10033` + +**Message** + +```markdown +Could not find the clustering key +``` + +### `DB-DATA-LOADER-10034` + +**Message** + +```markdown +No table metadata found +``` + +### `DB-DATA-LOADER-10035` + +**Message** + +```markdown +The data mapping source field '%s' for table '%s' is missing in the JSON data record +``` + +### `DB-DATA-LOADER-10036` + +**Message** + +```markdown +The CSV row: %s does not match header: %s +``` + +### `DB-DATA-LOADER-10037` + +**Message** + +```markdown +Expected JSON file content to be an array +``` + +### `DB-DATA-LOADER-10038` + +**Message** + +```markdown +Missing option: either the '--namespace' and '--table' options or the '--control-file' option must be specified +``` + +### `DB-DATA-LOADER-10039` + +**Message** + +```markdown +The file '%s' specified by the argument '%s' does not exist +``` + +### `DB-DATA-LOADER-10040` + +**Message** + +```markdown +Cannot write to the log directory: %s +``` + +### `DB-DATA-LOADER-10041` + +**Message** + +```markdown +Failed to create the log directory: %s +``` + +### `DB-DATA-LOADER-10042` + +**Message** + +```markdown +Failed to parse the control file: %s +``` + +### `DB-DATA-LOADER-10043` + +**Message** + +```markdown +No permission to create or write files in the directory: %s +``` + +### `DB-DATA-LOADER-10044` + +**Message** + +```markdown +Failed to create the directory: %s +``` + +### `DB-DATA-LOADER-10045` + +**Message** + +```markdown +Path exists but is not a directory: %s +``` + +### `DB-DATA-LOADER-10046` + +**Message** + +```markdown +File path must not be blank +``` + +### `DB-DATA-LOADER-10047` + +**Message** + +```markdown +File not found: %s +``` + +### `DB-DATA-LOADER-10048` + +**Message** + +```markdown +Invalid date time value '%s' specified for column %s in table %s in namespace %s +``` + +### `DB-DATA-LOADER-10049` + +**Message** + +```markdown +Key value cannot be null or empty +``` + +### `DB-DATA-LOADER-10050` + +**Message** + +```markdown +Invalid key-value format: %s +``` + +### `DB-DATA-LOADER-10051` + +**Message** + +```markdown +Value must not be null +``` + +### `DB-DATA-LOADER-10052` + +**Message** + +```markdown +Delimiter must not be null +``` + +### `DB-DATA-LOADER-10053` + +**Message** + +```markdown +Config file path must not be blank +``` + +### `DB-DATA-LOADER-10054` + +**Message** + +```markdown +Data chunk size must be greater than 0 +``` + +### `DB-DATA-LOADER-10055` + +**Message** + +```markdown +Transaction size must be greater than 0 +``` + +### `DB-DATA-LOADER-10056` + +**Message** + +```markdown +Number of max threads must be greater than 0 +``` + +## `DB-DATA-LOADER-3xxxx` status codes + +The following are status codes and messages for the internal error category. + +### `DB-DATA-LOADER-30000` + +**Message** + +```markdown +A problem occurred while trying to save the data. Details: %s +``` + +### `DB-DATA-LOADER-30001` + +**Message** + +```markdown +A problem occurred while scanning. Are you sure you are running in the correct transaction mode? Details: %s +``` + +### `DB-DATA-LOADER-30002` + +**Message** + +```markdown +Failed to read CSV file. Details: %s +``` + +### `DB-DATA-LOADER-30003` + +**Message** + +```markdown +Failed to CSV read header line. Details: %s +``` + +### `DB-DATA-LOADER-30004` + +**Message** + +```markdown +Data chunk processing was interrupted. Details: %s +``` + +### `DB-DATA-LOADER-30005` + +**Message** + +```markdown +Failed to read JSON file. Details: %s +``` + +### `DB-DATA-LOADER-30006` + +**Message** + +```markdown +Failed to read JSON Lines file. Details: %s +``` diff --git a/versioned_docs/version-3.X/scalardb-data-loader/getting-started-export.mdx b/versioned_docs/version-3.X/scalardb-data-loader/getting-started-export.mdx new file mode 100644 index 00000000..9cea0e4f --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-data-loader/getting-started-export.mdx @@ -0,0 +1,62 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Getting started with Export + +This document explains how you can get started with the ScalarDB Data Loader Export function. + +## Features + +The ScalarDB Data Loader allows you to export data in the following formats: + +- JSON +- JSONLines +- CSV + +Each export will run a ScalarDB scan operation based on the provided CLI arguments when running data loader. + +## Usage + +The data loader export function can be started with the following minimal configuration: + +```console +./scalardb-data-loader export --config scalardb.properties --namespace namespace --table tableName +``` + + + +- --config: the path to the scalardb connection properties file +- --namespace: the namespace of the table that contains the data +- --table: name of the table that contains the data + +By default, the data loader will create the output file in the working directory if the `--output-file` argument is omitted as well. + +### Command-line flags + +Here is a list of flags (options) that can be used with the scalardb data loader. + +| Flag | Description | Usage | +| ----------------- | ------------------------------------------------------------ | ------------------------------------------------------ | +| --config | The path to the scalardb.properties file. If omitted the tool looks for a file named `scalardb.properties` in the current folder | `scalardb-data-loader --config scalardb.properties` | +| --namespace | Namespace to export table data from. Required. | `scalardb-data-loader --namespace namespace` | +| --table | Name of table to export data from. Required. | `scalardb-data-loader --table tableName` | +| --key | Export data of specific Partition key. By default, it exports all data from the specified table. | `scalardb-data-loader --key columnName=value` | +| --sort | Specify a column to sort on. The column needs to be a clustering key. The argument can be repeated to provide multiple sortings. This flag is only applicable to `--key`. | `scalardb-data-loader --sort columnName=desc` | +| --projection | Limit the columns that are exported by providing a projection. The argument can be repeated to provide multiple projections. | `scalardb-data-loader --projection columnName` | +| --start | Clustering key to mark scan start. This flag is only applicable to `--key`. | `scalardb-data-loader --start columnName=value` | +| --start-exclusive | Is the scan start exclusive or not. If omitted, the default value is `false`. This flag is only applicable to `--key` | `scalardb-data-loader --start-exclusive` | +| --end | Clustering key to mark scan end. This flag is only applicable to `--key`. | `scalardb-data-loader --end columnName=value` | +| --end-exclusive | Is the scan start exclusive or not. If omitted, the default value is `false`. This flag is only applicable to `--key` | `scalardb-data-loader --end-exclusive` | +| --limit | Limit the results of the scan. If omitted, the default value is `0` which means their is no limit. | `scalardb-data-loader --limit 1000` | +| --output-file | The name and path of the output file. If omitted, the tool will save the file in the current folder with the following name format:
`export_namespace.tableName_timestamp.json` or `export_namespace.tableName_timestamp.csv`

The ouput folder needs to exists. The dataloader does not create the output folder for you. | `scalardb-data-loader --output-file ./out/output.json` | +| --format | The output format. By default `json` is selected. | `scalardb-data-loader --format json` | +| --metadata | When set to true the transaction metadata is included in the export. By default this is set to `false` | `scalardb-data-loader --metadata` | +| --delimiter | The delimiter used in CSV files. Default value is `;` | `scalardb-data-loader --delimiter ;` | +| --no-headers | Exclude header row in CSV file. Default is `false` | `scalardb-data-loader --no-headers` | +| --threads | Thread count for concurrent processing | `scalardb-data-loader --threads 500` | + diff --git a/versioned_docs/version-3.X/scalardb-data-loader/getting-started-import.mdx b/versioned_docs/version-3.X/scalardb-data-loader/getting-started-import.mdx new file mode 100644 index 00000000..d93e5ec2 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-data-loader/getting-started-import.mdx @@ -0,0 +1,293 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Getting started with Import + +This document explains how you can get started with the ScalarDB Data Loader Import function. + +## Features + +- Import data from JSON or JSONLines files +- Automatic data mapping based on source field name mapping +- Custom Data mapping via a JSON control file +- Import data from one record or line into multiple tables +- Support for INSERT, UPDATE and UPSERT + +## Usage + +The data loader import function can be started with the following minimal configuration: + +```console +./scalardb-data-loader import --config scalardb.properties --namespace namespace --table tableName +``` + +The above configuration starts an import process where no control file is used and the data mapping is applied automatically. + + + +Execute the following steps to successfully import new or existing data + +- Prepare a source file containing data that needs to be imported. + +- Choose the right import mode. By default, the import is done in `upsert` mode which means that data + will be inserted if new or updated if the partition key and/or clustering key is found. Other + options are `insert` mode or `update` mode. + +- Find the correct `namespace` and `table` name to import data to. + +- Determine if you want to run an `all required columns` check for each data row. If enabled data + rows with missing columns will be treated as failed and not imported. + +- Specify the pathnames for the `success` and `failed` output files. By default the data loader + creates the files in the working directory. + +- When dealing with JSON data, determine if you want the JSON output for the success or failed log files to + be in `pretty print` or not. By default, this option is disabled for performance + +- Optionally specify the `threads` argument to tweak performance + +- Run the import from the command line to start importing your data. Make sure to run the ScalarDB Data + Loader in the correct `storage` or `transaction` mode depending on your running ScalarDB instance. + +### Command-line flags + +Here is a list of flags (options) that can be used with the data loader: + +| Flag | Description | Usage | +|---------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------| +| --mode | The mode in which ScalarDB is running. If omitted, the default value is `storage` | `scalardb-data-loader --mode transaction` | +| --config | the path to the scalardb.properties file. If omitted the tool looks for a file named `scalardb.properties` in the current folder | `scalardb-data-loader --config scalardb.properties` | +| --namespace | Namespace to export table data from. Required when no control file is provided. | `scalardb-data-loader --namespace namespace` | +| --table | name of the table to export data from. Required when no control file is provided. | `scalardb-data-loader --table tableName` | +| --import-mode | Mode to import the data into the ScalarDB table. Supported modes are `insert`, `update` and `upsert`. Optional. Default the value is set to `upsert` | `scalardb-data-loader --import-mode=upsert` | +| --all-columns-required | If set, data rows cannot be imported if they are missing columns. Optional. By default, the check is not executed. | `scalardb-data-loader --all-columns-required` | +| --file | Specify the path to the file that will be imported. Required | `scalardb-data-loader --file ` | +| --success | The path to the file that is created to write the succeed import results to. Both succeed and failed import results will be written to a different file.
Optional. By default, the a new file will be created in the current working directory.

Note: if the file already exists, it will be overridden. | `scalardb-data-loader --success ` | +| --failed | The path to the file that will be created to write the failed import results to.
Optional. By default, the a new file will be created in the current working directory.

Note: if the file already exists, it will be overridden. | `scalardb-data-loader --failed ` | +| --threads | Thread count for concurrent processing | `scalardb-data-loader --threads 500` | +| --format | The format of the import file. `json` and `jsonl` files are supported. Optional, default the value `json` is selected. | `scalardb-data-loader --format json` | +| --ignore-null | The null values in the source file will be ignored, which means that the existing data will not be overwritten. Optional, default the value is `false`. | `scalardb-data-loader --ignore-null` | +| --pretty | When set, the output to the success and failed files is done in `pretty print` mode. By default the option is not enabled. | `scalardb-data-loader --pretty` | +| --control-file | The path to the JSON control file specifying the rules for the custom data mapping and/or multi-table import. | `scalardb-data-loader --control-file control.json` | +| --control-file-validation-level | The validation level for the control file. `MAPPED`, `KEYS` or` FULL`.

Optional and by default the level is set to `MAPPED` | `scalardb-data-loader --control-file-validation-level FULL` | +| --log-put-value | Wether the value that was used in the ScalarDB `PUT` operation is included in the log files or not.
Optional and disabled by default. | `scalardb-data-loader --log-put-value` | +| --error-file-required | To export an optional error file of type JSON when the import file contains CSV data. By default, this option is disabled. | `scalardb-data-loader --error-file-required` | +| --error | To specify an optional error file when the import file contains CSV data. | `scalardb-data-loader --error ` | +| --delimiter | To specify a custom delimiter if the import file contains CSV data. | `scalardb-data-loader --delimiter ` | +| --header | To specify the header row data if the import file contains CSV data and does not have a header row. | `scalardb-data-loader --header ` | + +## Import mode + +The data loader supports the following import modes: + +| Mode | Description | +| ------ | ------------------------------------------------------------ | +| INSERT | Each source record is treated as new data. If the data already exists in the ScalarDB table, based on the partition and clustering key, the import for this source data will fail. | +| UPDATE | Each source record is treated as an update for existing data in the ScalarDB table. If the data does not exist in the table, based on the partition key and clustering key, the import for this source data will fail. | +| UPSERT | If the target ScalarDB table already contains the data, the import will be done via an UPDATE. If the target data is missing, it will be treated as an INSERT. | + +*Note*: + + In the case of `INSERT`, it is required to have matching fields in the source files for each target column via automatic or custom mapping via the control file. This also applies to an `UPSERT` that turns into an `INSERT`. + +## Data mapping + +### Automatic mapping + +When no control file is provided, the data loader will automatically map the fields in the source JSON data to the available columns in the ScalarDB table. If the name does not match, and if all columns are required, it will be treated as a validation error. In this case, the import for this record will fail and the result will be added to the failed output log. + +### Custom mapping + +When the source fields do not match the target column name, it is necessary to use a control file. In this control, file you can specify the custom mapping rules for the field names. + +e.g. the following control file to map the field `custom_id` in the source file to `id` in the target table. + +```json +{ + "tables": [{ + "namespace": "sample", + "table_name": "table1", + "mappings": [{ + "source_field": "custom_id", + "target_column": "id" + }] + } + ] +} +``` + +## Control file + +To allow for custom data mapping or multi-table importing, the data loader supports configuration via a JSON control file. This file needs to be passed in via the `--control-file` argument when starting the data loader. + +### Control file validation levels + +To enforce validation on the control file, the data loader allows you to specify the validation level. Based on the set level, the data loader will run a pre-check and validate the control file based on the level rules. + +The following levels are supported: + +| Level | Description | +| ------ | ------------------------------------------------------------ | +| FULL | This validation makes sure that the control file has mappings for each column in the target ScalarDB table. | +| KEYS | This validation makes sure that mappings are available for each ScalarDB table partition and, if available, clustering keys columns in the control file mappings. | +| MAPPED | The validation makes sure that the provided source fields and target columns exist for only the mappings that are provided in the control file.
No other fields are checked. | + +The validation level is optional and can be set via the `--control-file-validation-level` argument when starting the data loader. + +*Note*: + +This validation is run as a pre-check and does not mean the import process will automatically succeed. + +e.g. If the level is set to mapped and the control file does not contain mappings for each column for an INSERT, the import process will still fail as all columns are required to be mapped for an INSERT. + +## Multi-table import + +The data loader supports multi-table target importing. + +One single row in a JSON or JSONLine file can be imported into multiple tables by specifying table mapping rules in the control file. Currently, multi-table import is not supported without a control file. + +When using multi-table import in ScalarDB transaction mode, a transaction is created for each table import. e.g. If the source record is mapped to 3 tables in the control file, 3 separate transactions are created. + +e.g. The import the following source record into `table1` and `table2` we execute the following steps: + +| Table1 | Table2 | +| ------ | ------ | +| Id | Id | +| price | amount | + +**Source record** + +```json +[{ + "custom_id": 1, + "custom_price": 1000, + "custom_amount": 100 + +}] +``` + +**Control file** + +```json +{ + "tables": [{ + "namespace": "sample", + "table_name": "table1", + "mappings": [{ + "source_field": "custom_id", + "target_column": "id" + }, { + "source_field": "custom_price", + "target_column": "price" + }] + }, + { + "namespace": "sample", + "table_name": "table2", + "mappings": [{ + "source_field": "custom_id", + "target_column": "id" + }, { + "source_field": "custom_amount", + "target_column": "amount" + }] + } + ] +} +``` + + + +## Output logs + +When starting an import task, the data loader logs the import results in two files. One file contains the import data that is successfully imported and one file contains the data that cannot be imported. The failed data will contain an added field that explains why the data could not be imported. This field is called `data_loader_import_status`. + +The file containing the failed imports can be edited to correct the problems and used as the source file for a new import task as is. It is not required to first remove the `data_loader_import_status` field containing the error. This field will be ignored during the import process and the original value will not be included in the new version of the success or failed output file. + +The file with the successfully imported data also contains the `data_loader_import_status` field. In this file, each imported data row has a status message for the data. Whether a new row was created or existing data was updated. + +### Log format + +| Field | Description | +| -------------- | ------------------------------------------------------------ | +| action | The result of the import process for the data record. UPDATE, INSERT or FAILED_DURING_VALIDATION | +| namespace | The name of the namespace of the table that the data is imported in | +| tablename | The name of the table that the data is imported in | +| is_data_mapped | Whether custom data mapping was applied or not based on an available control file. | +| tx_id | The id of the transaction. Only available if the data loader is run in `transaction` mode. | +| value | The final value, after optional data mapping, that the data loader uses in the `PUT` operation. | +| row_number | The line number or record number of the source data. | +| Errors | A list of validation or other errors for things that went wrong during the import process. | + +Example of a JSON formatted success log file: + +```json +[{ + "column_1": 1, + "column_2": 2, + "column_n": 3, + "data_loader_import_status": { + "results": [{ + "action": "UPDATE", + "namespace": "namespace1", + "tableName": "table1", + "is_data_mapped": true, + "tx_id": "value", + "value": "value", + "row_number": "value" + }] + } +}] +``` + + + +Example of a JSON formatted failed log file: + +```json +[{ + "column_1": 1, + "column_2": 2, + "column_n": 3, + "data_loader_import_status": { + "results": [{ + "action": "FAILED_DURING_VALIDATION", + "namespace": "namespace1", + "tableName": "table1", + "is_data_mapped": false, + "value": "value", + "row_number": "value", + "errors": [ + "missing columns found during validation" + ] + }] + } +}] +``` + +## Data duplicates + +The data loader does not handle duplicates by itself. In ScalarDB transaction mode, trying to +update the same target data in fast succession will cause `No Mutation` errors and these are not +handled by the data loader. These failed data rows will be added to the failed import result output +file and can be re-tried for import later. + +However, it is recommended to make sure the import file does not contain updates or inserts on the +same partition keys and/or clustering keys as the correct state cannot be guaranteed by the data +loader. + + + +## Storage vs transaction mode + +ScalarDB supports both storage and transaction mode and this support is included in the data loader import process. + +When the loader is started in storage mode, each import is executed in a non-transactional way. + +Starting the loader in transaction mode will use transactions to import the data. Currently, each row is imported via a separate transaction. When importing a single record into multiple tables, a separate transaction is created for each table import. + diff --git a/versioned_docs/version-3.X/scalardb-graphql/how-to-run-two-phase-commit-transaction.mdx b/versioned_docs/version-3.X/scalardb-graphql/how-to-run-two-phase-commit-transaction.mdx new file mode 100644 index 00000000..85dfdbbc --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-graphql/how-to-run-two-phase-commit-transaction.mdx @@ -0,0 +1,157 @@ +--- +tags: + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# How to run two-phase commit transaction + +ScalarDB GraphQL supports two-phase commit style transactions +called [Two-phase Commit Transactions](../two-phase-commit-transactions.mdx). +With Two-phase Commit Transactions, you can execute a transaction that spans multiple +processes/applications (e.g., Microservices). +We name the application that starts a transaction "coordinator" while the applications that +join the transaction are named "participants". +Every two-phase commit operation requires annotating the mutation or query operation with +a `@twoPhaseCommit` directive. Below is a description of such operations. + +## Start a transaction + +To start a transaction, add the `@twoPhaseCommit` directive without setting parameters. + +```graphql +query some_query @twoPhaseCommit { + # some query +} +``` + +The transaction ID of the started transaction will be returned in the extensions object that is part +of the result. + +```json +{ + "data": { + ... + }, + "extensions": { + "transaction": { + "id": "the_transaction_id" + } + } +} +``` + +## Join a transaction (for participants) + +In a participant application, to join the transaction started by a coordinator application, set the +transaction ID with the `id` parameter and set the `join` parameter to true. + +```graphql +query some_query_from_participant @twoPhaseCommit(id:"the_transaction_id", join:true) { + # some query +} +``` + +## Resume a transaction + +To continue executing operations in the started or joined transaction, set the transaction ID value in +the `id` parameter of `@twoPhaseCommit` directive. + +```graphql +mutation some_mutation @twoPhaseCommit(id:"the_transaction_id") { + # some mutation +} +``` + +## Prepare, validate and commit a transaction + +After finishing the query and mutation operations, you need to commit the transaction. Like a +well-known +two-phase commit protocol, there are two phases: prepare and commit. +You first need to prepare the transaction in all the coordinator/participant applications, and then +you +need to commit the transaction in all the coordinator/participant applications. + +If the Consensus Commit transaction manager is configured with the `EXTRA_READ` serializable strategy +in `SERIALIZABLE` isolation level, an extra "validate" phase is required between prepare and +commit phases. +Similarly to prepare and commit, validate need to be executed in all the coordinator/participants +applications. + +Prepare, validate and commit can be executed in parallel with all the coordinator/participants +applications. + +### Prepare a transaction + +Two options are possible to prepare a two-phase commit transaction. + +#### Via the directive parameter + +By using the `prepare` parameter of the directive, the transaction will be prepared after the +execution of the operation fields and only if they do not raise an error. + +```graphql +mutation some_mutation_then_prepare_tx @twoPhaseCommit(id:"the_transaction_id", prepare:true) { + mutation1 : ... + mutation2 : ... + # the transaction will be prepared after the execution of the mutation1 and mutation2 fields +} +``` + +#### Via the mutation field + +Add a `prepare` field in a mutation operation. This field will trigger the transaction +preparation. + +```graphql +mutation prepare_tx @twoPhaseCommit(id:"the_transaction_id") { + prepare +} +``` + +### Validate a transaction + +Add a `validate` field in a mutation operation. This field will trigger the transaction +validation. + +```graphql +mutation validate_tx @twoPhaseCommit(id:"the_transaction_id") { + validate +} +``` + +### Commit a transaction + +Add a `commit` field in a mutation operation. This field will trigger the transaction commit. + +```graphql +mutation commit_tx @twoPhaseCommit(id:"the_transaction_id") { + commit +} +``` + +### Abort/Rollback a transaction + +When you need to abort/rollback a transaction explicitly, you can use the `abort` or `rollback` +mutation fields interchangeably (both have the same effect and usage). Note that you cannot mix it +with any other operations, so you must specify it alone. + +```graphql +mutation AbortTx @twoPhaseCommit(id: "the_transaction_id") { + abort +} +``` + +or + +```graphql +mutation RollbackTx @twoPhaseCommit(id: "the_transaction_id") { + rollback +} +``` + +## Error handling + +If an exception is thrown by a `@twoPhaseCommit` operation, ScalarDB GraphQL triggers a rollback procedure that recovers the transaction. +For more details about the exception handling in two-phase commit transaction, please refer to +the [exception handling guide for ScalarDB two-phase commit transaction](../two-phase-commit-transactions.mdx#handle-exceptions). diff --git a/versioned_docs/version-3.X/scalardb-graphql/index.mdx b/versioned_docs/version-3.X/scalardb-graphql/index.mdx new file mode 100644 index 00000000..c325ea85 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-graphql/index.mdx @@ -0,0 +1,23 @@ +--- +tags: + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# ScalarDB GraphQL Overview + +ScalarDB GraphQL is an interface layer that allows client applications to communicate with ScalarDB Cluster by using GraphQL. It enables you to take advantage the benefits of GraphQL, such as flexible data retrieval and type safety, while benefiting from the transaction management and data access features in ScalarDB. + +By using ScalarDB GraphQL, you can create GraphQL schemas automatically based on the ScalarDB schemas, perform CRUD operations, and execute complex transactions across multiple databases. The interface simplifies backend development by providing a unified querying mechanism, making it particularly useful for modern applications expecting advanced and responsive data interactions. + +## Getting started with GraphQL in ScalarDB Cluster + +ScalarDB GraphQL is designed to be intuitive and user-friendly, enabling developers to easily create GraphQL schemas automatically based on the ScalarDB schemas and interact with the underlying databases. + +For details on how to set up ScalarDB Cluster with GraphQL support, see [Getting Started with ScalarDB Cluster GraphQL](../scalardb-cluster/getting-started-with-scalardb-cluster-graphql.mdx). + +## Transactions with a two-phase commit + +ScalarDB GraphQL supports executing transactions with a two-phase commit interface. By using the two-phase commit interface, you can execute a transaction that spans multiple processes/applications (for example, microservices applications). + +For details on how to execute transactions by using the two-phase commit interface in ScalarDB GraphQL, see [How to Run Two-Phase Commit Transactions](./how-to-run-two-phase-commit-transaction.mdx). diff --git a/versioned_docs/version-3.X/scalardb-graphql/scalardb-graphql-status-codes.mdx b/versioned_docs/version-3.X/scalardb-graphql/scalardb-graphql-status-codes.mdx new file mode 100644 index 00000000..db653dbc --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-graphql/scalardb-graphql-status-codes.mdx @@ -0,0 +1,243 @@ +--- +tags: + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# ScalarDB GraphQL Error Codes + +This page provides a list of error codes in ScalarDB GraphQL. + +## Error code classes and descriptions + +| Class | Description | +|:-------------------|:-----------------------------------| +| `DB-GRAPHQL-1xxxx` | Errors for the user error category | + +## `DB-GRAPHQL-1xxxx` status codes + +The following are status codes and messages for the user error category. + +### `DB-GRAPHQL-10000` + +**Message** + +```markdown +A long value was expected +``` + +### `DB-GRAPHQL-10001` + +**Message** + +```markdown +The value is out of range for BigIntValue +``` + +### `DB-GRAPHQL-10002` + +**Message** + +```markdown +A long, integer, or string value was expected +``` + +### `DB-GRAPHQL-10003` + +**Message** + +```markdown +The AST type `IntValue` was expected +``` + +### `DB-GRAPHQL-10004` + +**Message** + +```markdown +A float value was expected +``` + +### `DB-GRAPHQL-10005` + +**Message** + +```markdown +An integer or float value was expected +``` + +### `DB-GRAPHQL-10006` + +**Message** + +```markdown +The AST type `IntValue` or `FloatValue` was expected +``` + +### `DB-GRAPHQL-10007` + +**Message** + +```markdown +The type is not supported. Type: %s +``` + +### `DB-GRAPHQL-10008` + +**Message** + +```markdown +The field `%s` requires a `@transaction` or `@twoPhaseCommit` directive with proper arguments +``` + +### `DB-GRAPHQL-10009` + +**Message** + +```markdown +The field `%s` cannot be used together with other fields +``` + +### `DB-GRAPHQL-10010` + +**Message** + +```markdown +The `@twoPhaseCommit` directive with the `id` argument is required to `%s` the transaction +``` + +### `DB-GRAPHQL-10011` + +**Message** + +```markdown +`%s` and `prepare` cannot be run simultaneously +``` + +### `DB-GRAPHQL-10012` + +**Message** + +```markdown +`%s` and `join` cannot be run simultaneously +``` + +### `DB-GRAPHQL-10013` + +**Message** + +```markdown +The `@transaction` directive with the `id` argument is required to `%s` the transaction +``` + +### `DB-GRAPHQL-10014` + +**Message** + +```markdown +`%s` and `commit` cannot be run simultaneously +``` + +### `DB-GRAPHQL-10015` + +**Message** + +```markdown +An object cannot be annotated with both `@transaction` and `@twoPhaseCommit` directives +``` + +### `DB-GRAPHQL-10016` + +**Message** + +```markdown +The `join` argument of the `@twoPhaseCommit` directive requires a transaction `id` argument +``` + +### `DB-GRAPHQL-10017` + +**Message** + +```markdown +`%s` requires the mutation object to be annotated with a `@twoPhaseCommit` directive +``` + +### `DB-GRAPHQL-10018` + +**Message** + +```markdown +The `%s` clustering key must have only one of the following: %s +``` + +### `DB-GRAPHQL-10019` + +**Message** + +```markdown +A string variable is expected but got %s +``` + +### `DB-GRAPHQL-10020` + +**Message** + +```markdown +Unexpected value of id: %s +``` + +### `DB-GRAPHQL-10021` + +**Message** + +```markdown +A Boolean variable is expected but got %s +``` + +### `DB-GRAPHQL-10022` + +**Message** + +```markdown +Unexpected value of %s: %s +``` + +### `DB-GRAPHQL-10023` + +**Message** + +```markdown +Invalid column. Column: %s; Type: %s +``` + +### `DB-GRAPHQL-10024` + +**Message** + +```markdown +Unexpected value of type: %s +``` + +### `DB-GRAPHQL-10025` + +**Message** + +```markdown +Only one of the following can be specified: %s +``` + +### `DB-GRAPHQL-10026` + +**Message** + +```markdown +Unexpected mutation field: %s +``` + +### `DB-GRAPHQL-10027` + +**Message** + +```markdown +Invalid type: %s +``` diff --git a/versioned_docs/version-3.X/scalardb-samples/README.mdx b/versioned_docs/version-3.X/scalardb-samples/README.mdx new file mode 100644 index 00000000..a2157779 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-samples/README.mdx @@ -0,0 +1,19 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Run Sample Applications Overview + +In this sub-category, you can learn how to run various sample applications that take advantage of ScalarDB. + +To set up and run the sample applications, see the following guides: + +- [Multi-storage Transaction Sample](multi-storage-transaction-sample/README.mdx) +- [Microservice Transaction Sample](microservice-transaction-sample/README.mdx) +- [Spring Data JDBC for ScalarDB with Multi-storage Transaction Sample](spring-data-multi-storage-transaction-sample/README.mdx) +- [Spring Data JDBC for ScalarDB with Microservice Transaction Sample](spring-data-microservice-transaction-sample/README.mdx) +- [Analytical Queries by Using ScalarDB Analytics with PostgreSQL](scalardb-analytics-postgresql-sample/README.mdx) diff --git a/versioned_docs/version-3.X/scalardb-samples/microservice-transaction-sample/README.mdx b/versioned_docs/version-3.X/scalardb-samples/microservice-transaction-sample/README.mdx new file mode 100644 index 00000000..40766fb4 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-samples/microservice-transaction-sample/README.mdx @@ -0,0 +1,534 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Create a Sample Application That Supports Microservice Transactions + +This tutorial describes how to create a sample application that supports microservice transactions in ScalarDB. + +## Overview + +This tutorial illustrates the process of creating a sample e-commerce application, where items can be ordered and paid for with a line of credit through [transactions with a two-phase commit interface](../../two-phase-commit-transactions.mdx) in ScalarDB. + +The sample application has two microservices called the *Customer Service* and the *Order Service* based on the [database-per-service pattern](https://microservices.io/patterns/data/database-per-service.html): + +- The **Customer Service** manages customer information, including line-of-credit information, credit limit, and credit total. +- The **Order Service** is responsible for order operations like placing an order and getting order histories. + +Each service has gRPC endpoints. Clients call the endpoints, and the services call each endpoint as well. + +The databases that you will be using in the sample application are Cassandra and MySQL. The Customer Service and the Order Service use Cassandra and MySQL, respectively, through ScalarDB. + +![Overview](images/overview.png) + +As shown in the diagram, both services access a small Coordinator database used for the Consensus Commit protocol. The database is service-independent and exists for managing transaction metadata for Consensus Commit in a highly available manner. + +In the sample application, for ease of setup and explanation, we co-locate the Coordinator database in the same Cassandra instance of the Order Service. Alternatively, you can manage the Coordinator database as a separate database. + +:::note + +Since the focus of the sample application is to demonstrate using ScalarDB, application-specific error handling, authentication processing, and similar functions are not included in the sample application. For details about exception handling in ScalarDB, see [How to handle exceptions](../../api-guide.mdx#how-to-handle-exceptions). + +Additionally, for the purpose of the sample application, each service has one container so that you can avoid using request routing between the services. However, for production use, because each service typically has multiple servers or hosts for scalability and availability, you should consider request routing between the services in transactions with a two-phase commit interface. For details about request routing, see [Request routing in transactions with a two-phase commit interface](../../two-phase-commit-transactions.mdx#request-routing-in-transactions-with-a-two-phase-commit-interface). + +::: + +### Service endpoints + +The endpoints defined in the services are as follows: + +- Customer Service + - `getCustomerInfo` + - `payment` + - `prepare` + - `validate` + - `commit` + - `rollback` + - `repayment` + +- Order Service + - `placeOrder` + - `getOrder` + - `getOrders` + +### What you can do in this sample application + +The sample application supports the following types of transactions: + +- Get customer information through the `getCustomerInfo` endpoint of the Customer Service. +- Place an order by using a line of credit through the `placeOrder` endpoint of the Order Service and the `payment`, `prepare`, `validate`, `commit`, and `rollback` endpoints of the Customer Service. + - Checks if the cost of the order is below the customer's credit limit. + - If the check passes, records the order history and updates the amount the customer has spent. +- Get order information by order ID through the `getOrder` endpoint of the Order Service and the `getCustomerInfo`, `prepare`, `validate`, `commit`, and `rollback` endpoints of the Customer Service. +- Get order information by customer ID through the `getOrders` endpoint of the Order Service and the `getCustomerInfo`, `prepare`, `validate`, `commit`, and `rollback` endpoints of the Customer Service. +- Make a payment through the `repayment` endpoint of the Customer Service. + - Reduces the amount the customer has spent. + +:::note + +The `getCustomerInfo` endpoint works as a participant service endpoint when receiving a transaction ID from the coordinator. + +::: + +## Prerequisites for this sample application + +- OpenJDK LTS version (8, 11, 17, or 21) from [Eclipse Temurin](https://adoptium.net/temurin/releases/) +- [Docker](https://www.docker.com/get-started/) 20.10 or later with [Docker Compose](https://docs.docker.com/compose/install/) V2 or later + +:::note + +This sample application has been tested with OpenJDK from Eclipse Temurin. ScalarDB itself, however, has been tested with JDK distributions from various vendors. For details about the requirements for ScalarDB, including compatible JDK distributions, please see [Requirements](../../requirements.mdx). + +::: + +## Set up ScalarDB + +The following sections describe how to set up the sample application that supports microservices transactions in ScalarDB. + +### Clone the ScalarDB samples repository + +Open **Terminal**, then clone the ScalarDB samples repository by running the following command: + +```console +git clone https://github.com/scalar-labs/scalardb-samples +``` + +Then, go to the directory that contains the sample application by running the following command: + +```console +cd scalardb-samples/microservice-transaction-sample +``` + +### Start Cassandra and MySQL + +Cassandra and MySQL are already configured for the sample application, as shown in [`database-cassandra.properties`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/database-cassandra.properties) and [`database-mysql.properties`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/database-mysql.properties), respectively. For details about configuring the multi-storage transactions feature in ScalarDB, see [How to configure ScalarDB to support multi-storage transactions](../../multi-storage-transactions.mdx#how-to-configure-scalardb-to-support-multi-storage-transactions). + +To start Cassandra and MySQL, which are included in the Docker container for the sample application, run the following command: + +```console +docker-compose up -d mysql cassandra +``` + +:::note + +Starting the Docker container may take more than one minute depending on your development environment. + +::: + +### Load the schema + +The database schema (the method in which the data will be organized) for the sample application has already been defined in [`customer-service-schema.json`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/customer-service-schema.json) for the Customer Service and [`order-service-schema.json`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/order-service-schema.json) for the Order Service. + +To apply the schema, go to the [ScalarDB Releases](https://github.com/scalar-labs/scalardb/releases) page and download the ScalarDB Schema Loader that matches the version of ScalarDB that you want to use to the `scalardb-samples/microservice-transaction-sample` folder. + +#### MySQL + +To load the schema for [`customer-service-schema.json`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/customer-service-schema.json) into MySQL, run the following command, replacing `` with the version of the ScalarDB Schema Loader that you downloaded: + +```console +java -jar scalardb-schema-loader-.jar --config database-mysql.properties --schema-file customer-service-schema.json +``` + +#### Cassandra + +To load the schema for [`order-service-schema.json`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/order-service-schema.json) into Cassandra, run the following command, replacing `` with the version of the ScalarDB Schema Loader that you downloaded: + +```console +java -jar scalardb-schema-loader-.jar --config database-cassandra.properties --schema-file order-service-schema.json --coordinator +``` + +#### Schema details + +As shown in [`customer-service-schema.json`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/customer-service-schema.json), all the tables for the Customer Service are created in the `customer_service` namespace. + +- `customer_service.customers`: a table that manages customers' information + - `credit_limit`: the maximum amount of money a lender will allow each customer to spend when using a line of credit + - `credit_total`: the amount of money that each customer has already spent by using their line of credit + +As shown in [`order-service-schema.json`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/order-service-schema.json), all the tables for the Order Service are created in the `order_service` namespace. + +- `order_service.orders`: a table that manages order information +- `order_service.statements`: a table that manages order statement information +- `order_service.items`: a table that manages information of items to be ordered + +The Entity Relationship Diagram for the schema is as follows: + +![ERD](images/ERD.png) + +### Load the initial data by starting the microservices + +Before starting the microservices, build the Docker images of the sample application by running the following command: + +```console +./gradlew docker +``` + +Then, start the microservices by running the following command: + +```console +docker-compose up -d customer-service order-service +``` + +After starting the microservices and the initial data has loaded, the following records should be stored in the `customer_service.customers` table: + +| customer_id | name | credit_limit | credit_total | +|-------------|---------------|--------------|--------------| +| 1 | Yamada Taro | 10000 | 0 | +| 2 | Yamada Hanako | 10000 | 0 | +| 3 | Suzuki Ichiro | 10000 | 0 | + +And the following records should be stored in the `order_service.items` table: + +| item_id | name | price | +|---------|--------|-------| +| 1 | Apple | 1000 | +| 2 | Orange | 2000 | +| 3 | Grape | 2500 | +| 4 | Mango | 5000 | +| 5 | Melon | 3000 | + +## Execute transactions and retrieve data in the sample application + +The following sections describe how to execute transactions and retrieve data in the sample e-commerce application. + +### Get customer information + +Start with getting information about the customer whose ID is `1` by running the following command: + +```console +./gradlew :client:run --args="GetCustomerInfo 1" +``` + +You should see the following output: + +```console +... +{"id": 1,"name": "Yamada Taro","credit_limit": 10000} +... +``` + +At this time, `credit_total` isn't shown, which means the current value of `credit_total` is `0`. + +### Place an order + +Then, have customer ID `1` place an order for three apples and two oranges by running the following command: + +:::note + +The order format in this command is `./gradlew run --args="PlaceOrder :,:,..."`. + +::: + +```console +./gradlew :client:run --args="PlaceOrder 1 1:3,2:2" +``` + +You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: + +```console +... +{"order_id": "4ccdb21c-ac03-4b48-bcb7-cad57eac1e79"} +... +``` + +### Check order details + +Check details about the order by running the following command, replacing `` with the UUID for the `order_id` that was shown after running the previous command: + +```console +./gradlew :client:run --args="GetOrder " +``` + +You should see a similar output as below, with different UUIDs for `order_id` and `timestamp`: + +```console +... +{"order": {"order_id": "4ccdb21c-ac03-4b48-bcb7-cad57eac1e79","timestamp": 1631605253126,"customer_id": 1,"customer_name": "Yamada Taro","statement": [{"item_id": 1,"item_name": "Apple","price": 1000,"count": 3,"total": 3000},{"item_id": 2,"item_name": "Orange","price": 2000,"count": 2,"total": 4000}],"total": 7000}} +... +``` + +### Place another order + +Place an order for one melon that uses the remaining amount in `credit_total` for customer ID `1` by running the following command: + +```console +./gradlew :client:run --args="PlaceOrder 1 5:1" +``` + +You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: + +```console +... +{"order_id": "0b10db66-faa6-4323-8a7a-474e8534a7ee"} +... +``` + +### Check order history + +Get the history of all orders for customer ID `1` by running the following command: + +```console +./gradlew :client:run --args="GetOrders 1" +``` + +You should see a similar output as below, with different UUIDs for `order_id` and `timestamp`, which shows the history of all orders for customer ID `1` in descending order by timestamp: + +```console +... +{"order": [{"order_id": "0b10db66-faa6-4323-8a7a-474e8534a7ee","timestamp": 1631605501485,"customer_id": 1,"customer_name": "Yamada Taro","statement": [{"item_id": 5,"item_name": "Melon","price": 3000,"count": 1,"total": 3000}],"total": 3000},{"order_id": "4ccdb21c-ac03-4b48-bcb7-cad57eac1e79","timestamp": 1631605253126,"customer_id": 1,"customer_name": "Yamada Taro","statement": [{"item_id": 1,"item_name": "Apple","price": 1000,"count": 3,"total": 3000},{"item_id": 2,"item_name": "Orange","price": 2000,"count": 2,"total": 4000}],"total": 7000}]} +... +``` + +### Check credit total + +Get the credit total for customer ID `1` by running the following command: + +```console +./gradlew :client:run --args="GetCustomerInfo 1" +``` + +You should see the following output, which shows that customer ID `1` has reached their `credit_limit` in `credit_total` and cannot place anymore orders: + +```console +... +{"id": 1,"name": "Yamada Taro","credit_limit": 10000,"credit_total": 10000} +... +``` + +Try to place an order for one grape and one mango by running the following command: + +```console +./gradlew :client:run --args="PlaceOrder 1 3:1,4:1" +``` + +You should see the following output, which shows that the order failed because the `credit_total` amount would exceed the `credit_limit` amount: + +```console +... +io.grpc.StatusRuntimeException: FAILED_PRECONDITION: Credit limit exceeded + at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271) + at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252) + at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165) + at sample.rpc.OrderServiceGrpc$OrderServiceBlockingStub.placeOrder(OrderServiceGrpc.java:296) + at sample.client.command.PlaceOrderCommand.call(PlaceOrderCommand.java:38) + at sample.client.command.PlaceOrderCommand.call(PlaceOrderCommand.java:12) + at picocli.CommandLine.executeUserObject(CommandLine.java:2041) + at picocli.CommandLine.access$1500(CommandLine.java:148) + at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461) + at picocli.CommandLine$RunLast.handle(CommandLine.java:2453) + at picocli.CommandLine$RunLast.handle(CommandLine.java:2415) + at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273) + at picocli.CommandLine$RunLast.execute(CommandLine.java:2417) + at picocli.CommandLine.execute(CommandLine.java:2170) + at sample.client.Client.main(Client.java:39) +... +``` + +### Make a payment + +To continue making orders, customer ID `1` must make a payment to reduce the `credit_total` amount. + +Make a payment by running the following command: + +```console +./gradlew :client:run --args="Repayment 1 8000" +``` + +Then, check the `credit_total` amount for customer ID `1` by running the following command: + +```console +./gradlew :client:run --args="GetCustomerInfo 1" +``` + +You should see the following output, which shows that a payment was applied to customer ID `1`, reducing the `credit_total` amount: + +```console +... +{"id": 1,"name": "Yamada Taro","credit_limit": 10000,"credit_total": 2000} +... +``` + +Now that customer ID `1` has made a payment, place an order for one grape and one melon by running the following command: + +```console +./gradlew :client:run --args="PlaceOrder 1 3:1,4:1" +``` + +You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: + +```console +... +{"order_id": "dd53dd9d-aaf4-41db-84b2-56951fed6425"} +... +``` + +## Stop the sample application + +To stop the sample application, you need to stop the Docker containers that are running Cassandra, MySQL, and the microservices. To stop the Docker containers, run the following command: + +```console +docker-compose down +``` + +## Reference - How the microservice transaction is achieved + +The transactions for placing an order, getting a single order, and getting the history of orders achieve the microservice transaction. This section focuses on how the transactions that span the Customer Service and the Order Service are implemented by placing an order as an example. + +The following sequence diagram shows the transaction for placing an order: + +![Microservice transaction sequence diagram](images/sequence_diagram.png) + +### 1. Transaction with a two-phase commit interface is started + +When a client sends a request to place an order to the Order Service, `OrderService.placeOrder()` is called, and the microservice transaction starts. + +At first, the Order Service starts a transaction with a two-phase commit interface with `start()` as follows. For reference, see [`OrderService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/order-service/src/main/java/sample/order/OrderService.java). + +```java +transaction = twoPhaseCommitTransactionManager.start(); +``` + +### 2. CRUD operations are executed + +After the transaction with a two-phase commit interface starts, CRUD operations are executed. The Order Service puts the order information in the `order_service.orders` table and the detailed information in the `order_service.statements` table as follows. For reference, see [`OrderService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/order-service/src/main/java/sample/order/OrderService.java): + +```java +// Put the order info into the `orders` table. +Order.put(transaction, orderId, request.getCustomerId(), System.currentTimeMillis()); + +int amount = 0; +for (ItemOrder itemOrder : request.getItemOrderList()) { + // Put the order statement into the `statements` table. + Statement.put(transaction, orderId, itemOrder.getItemId(), itemOrder.getCount()); + + // Retrieve the item info from the `items` table. + Optional item = Item.get(transaction, itemOrder.getItemId()); + if (!item.isPresent()) { + responseObserver.onError( + Status.NOT_FOUND.withDescription("Item not found").asRuntimeException()); + return; + } + + // Calculate the total amount. + amount += item.get().price * itemOrder.getCount(); +} +``` + +Then, the Order Service calls the `payment` gRPC endpoint of the Customer Service along with the transaction ID. For reference, see [`OrderService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/order-service/src/main/java/sample/order/OrderService.java). + +```java +customerServiceStub.payment( + PaymentRequest.newBuilder() + .setTransactionId(transactionId) + .setCustomerId(customerId) + .setAmount(amount) + .build()); +``` + +The `payment` endpoint of the Customer Service first joins the transaction with `join()` as follows. For reference, see [`CustomerService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/customer-service/src/main/java/sample/customer/CustomerService.java). + +```java +private void execOperationsAsParticipant(String funcName, String transactionId, + TransactionFunction operations, + StreamObserver responseObserver) { + try { + // Join the transaction + TwoPhaseCommitTransaction transaction = twoPhaseCommitTransactionManager.join(transactionId); + + // Execute operations + T response = operations.apply(transaction); +``` + +The endpoint then gets the customer information and checks if the customer's credit total exceeds the credit limit after the payment. If the credit total does not exceed the credit limit, the endpoint updates the customer's credit total. For reference, see [`CustomerService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/customer-service/src/main/java/sample/customer/CustomerService.java). + +```java +execOperationsAsParticipant("Payment", request.getTransactionId(), + transaction -> { + // Retrieve the customer info for the customer ID + Optional result = Customer.get(transaction, request.getCustomerId()); + if (!result.isPresent()) { + throw Status.NOT_FOUND.withDescription("Customer not found").asRuntimeException(); + } + + int updatedCreditTotal = result.get().creditTotal + request.getAmount(); + + // Check if the credit total exceeds the credit limit after payment + if (updatedCreditTotal > result.get().creditLimit) { + throw Status.FAILED_PRECONDITION + .withDescription("Credit limit exceeded") + .asRuntimeException(); + } + + // Update `credit_total` for the customer + Customer.updateCreditTotal(transaction, request.getCustomerId(), updatedCreditTotal); + + return Empty.getDefaultInstance(); + }, responseObserver +); +``` + +### 3. Transaction is committed by using the two-phase commit protocol + +After the Order Service receives the update that the payment succeeded, the Order Service tries to commit the transaction. + +To commit the transaction, the Order Service starts with preparing the transaction. The Order Service calls `prepare()` from its transaction object and calls the `prepare` gRPC endpoint of the Customer Service. For reference, see [`OrderService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/order-service/src/main/java/sample/order/OrderService.java): + +```java +transaction.prepare(); +callPrepareEndpoint(transaction.getId()); +``` + +In this endpoint, the Customer Service resumes the transaction and calls `prepare()` from its transaction object, as well. For reference, see [`CustomerService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/customer-service/src/main/java/sample/customer/CustomerService.java): + +```java +// Resume the transaction. +transaction = twoPhaseCommitTransactionManager.resume(request.getTransactionId()); + +// Prepare the transaction. +transaction.prepare(); +``` + +Similarly, the Order Service and the Customer Service call `validate()` from their transaction objects. For reference, see [`OrderService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/order-service/src/main/java/sample/order/OrderService.java) and [`CustomerService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/customer-service/src/main/java/sample/customer/CustomerService.java). For details about `validate()`, see [Validate the transaction](../../two-phase-commit-transactions.mdx#validate-the-transaction). + +After preparing and validating the transaction succeeds in both the Order Service and the Customer Service, the transaction can be committed. In this case, the Order Service calls `commit()` from its transaction object and then calls the `commit` gRPC endpoint of the Customer Service. For reference, see [`OrderService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/order-service/src/main/java/sample/order/OrderService.java). + +```java +transaction.commit(); +callCommitEndpoint(transaction.getId()); +``` + +In this endpoint, the Customer Service resumes the transaction and calls `commit()` from its transaction object, as well. For reference, see [`CustomerService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/customer-service/src/main/java/sample/customer/CustomerService.java). + +```java +// Resume the transaction. +transaction = twoPhaseCommitTransactionManager.resume(request.getTransactionId()); + +// Commit the transaction. +transaction.commit(); +``` + +### Error handling + +If an error happens while executing a transaction, you will need to roll back the transaction. In this case, the Order Service calls `rollback()` from its transaction object and then calls the `rollback` gRPC endpoint of the Customer Service. For reference, see [`OrderService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/order-service/src/main/java/sample/order/OrderService.java). + +```java +transaction.rollback(); +callRollbackEndpoint(transaction.getId()); +``` + +In this endpoint, the Customer Service resumes the transaction and calls `rollback()` from its transaction object, as well. For reference, see [`CustomerService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/microservice-transaction-sample/customer-service/src/main/java/sample/customer/CustomerService.java). + +```java +// Resume the transaction. +TwoPhaseCommitTransaction transaction = + twoPhaseCommitTransactionManager.resume(request.getTransactionId()); + +// Roll back the transaction. +transaction.rollback(); +``` + +For details about how to handle exceptions in ScalarDB, see [How to handle exceptions](../../api-guide.mdx#how-to-handle-exceptions). diff --git a/versioned_docs/version-3.X/scalardb-samples/microservice-transaction-sample/images/ERD.png b/versioned_docs/version-3.X/scalardb-samples/microservice-transaction-sample/images/ERD.png new file mode 100644 index 00000000..c0468efa Binary files /dev/null and b/versioned_docs/version-3.X/scalardb-samples/microservice-transaction-sample/images/ERD.png differ diff --git a/versioned_docs/version-3.X/scalardb-samples/microservice-transaction-sample/images/overview.png b/versioned_docs/version-3.X/scalardb-samples/microservice-transaction-sample/images/overview.png new file mode 100644 index 00000000..4340b4f5 Binary files /dev/null and b/versioned_docs/version-3.X/scalardb-samples/microservice-transaction-sample/images/overview.png differ diff --git a/versioned_docs/version-3.X/scalardb-samples/microservice-transaction-sample/images/sequence_diagram.png b/versioned_docs/version-3.X/scalardb-samples/microservice-transaction-sample/images/sequence_diagram.png new file mode 100644 index 00000000..0317b5f3 Binary files /dev/null and b/versioned_docs/version-3.X/scalardb-samples/microservice-transaction-sample/images/sequence_diagram.png differ diff --git a/versioned_docs/version-3.X/scalardb-samples/multi-storage-transaction-sample/README.mdx b/versioned_docs/version-3.X/scalardb-samples/multi-storage-transaction-sample/README.mdx new file mode 100644 index 00000000..6fe2af65 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-samples/multi-storage-transaction-sample/README.mdx @@ -0,0 +1,314 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Create a Sample Application That Supports Multi-Storage Transactions + +This tutorial describes how to create a sample application that supports the multi-storage transactions feature in ScalarDB. + +## Overview + +This tutorial illustrates the process of creating a sample e-commerce application, where items can be ordered and paid for with a line of credit by using the [multi-storage transactions](../../multi-storage-transactions.mdx) feature in ScalarDB. + +In this tutorial, you will build an application that uses both Cassandra and MySQL. By using the multi-storage transactions feature in ScalarDB, you can execute a transaction that spans both Cassandra and MySQL. + +![Overview](images/overview.png) + +:::note + +Since the focus of the sample application is to demonstrate using ScalarDB, application-specific error handling, authentication processing, and similar functions are not included in the sample application. For details about exception handling in ScalarDB, see [How to handle exceptions](../../api-guide.mdx#how-to-handle-exceptions). + +::: + +### What you can do in this sample application + +The sample application supports the following types of transactions: + +- Get customer information. +- Place an order by using a line of credit. + - Checks if the cost of the order is below the customer's credit limit. + - If the check passes, records the order history and updates the amount the customer has spent. +- Get order information by order ID. +- Get order information by customer ID. +- Make a payment. + - Reduces the amount the customer has spent. + +## Prerequisites for this sample application + +- OpenJDK LTS version (8, 11, 17, or 21) from [Eclipse Temurin](https://adoptium.net/temurin/releases/) +- [Docker](https://www.docker.com/get-started/) 20.10 or later with [Docker Compose](https://docs.docker.com/compose/install/) V2 or later + +:::note + +This sample application has been tested with OpenJDK from Eclipse Temurin. ScalarDB itself, however, has been tested with JDK distributions from various vendors. For details about the requirements for ScalarDB, including compatible JDK distributions, please see [Requirements](../../requirements.mdx). + +::: + +## Set up ScalarDB + +The following sections describe how to set up the sample application that supports the multi-storage transactions feature in ScalarDB. + +### Clone the ScalarDB samples repository + +Open **Terminal**, then clone the ScalarDB samples repository by running the following command: + +```console +git clone https://github.com/scalar-labs/scalardb-samples +``` + +Then, go to the directory that contains the sample application by running the following command: + +```console +cd scalardb-samples/multi-storage-transaction-sample +``` + +### Start Cassandra and MySQL + +Cassandra and MySQL are already configured for the sample application, as shown in [`database.properties`](https://github.com/scalar-labs/scalardb-samples/tree/master/multi-storage-transaction-sample/database.properties). For details about configuring the multi-storage transactions feature in ScalarDB, see [How to configure ScalarDB to support multi-storage transactions](../../multi-storage-transactions.mdx#how-to-configure-scalardb-to-support-multi-storage-transactions). + +To start Cassandra and MySQL, which are included in the Docker container for the sample application, make sure Docker is running and then run the following command: + +```console +docker-compose up -d +``` + +:::note + +Starting the Docker container may take more than one minute depending on your development environment. + +::: + +### Load the schema + +The database schema (the method in which the data will be organized) for the sample application has already been defined in [`schema.json`](https://github.com/scalar-labs/scalardb-samples/tree/master/multi-storage-transaction-sample/schema.json). + +To apply the schema, go to the [ScalarDB Releases](https://github.com/scalar-labs/scalardb/releases) page and download the ScalarDB Schema Loader that matches the version of ScalarDB that you want to use to the `scalardb-samples/multi-storage-transaction-sample` folder. + +Then, run the following command, replacing `` with the version of the ScalarDB Schema Loader that you downloaded: + +```console +java -jar scalardb-schema-loader-.jar --config database.properties --schema-file schema.json --coordinator +``` + +#### Schema details + +As shown in [`schema.json`](https://github.com/scalar-labs/scalardb-samples/tree/master/multi-storage-transaction-sample/schema.json) for the sample application, all the tables are created in the `customer` and `order` namespaces. + +- `customer.customers`: a table that manages customers' information + - `credit_limit`: the maximum amount of money a lender will allow each customer to spend when using a line of credit + - `credit_total`: the amount of money that each customer has already spent by using their line of credit +- `order.orders`: a table that manages order information +- `order.statements`: a table that manages order statement information +- `order.items`: a table that manages information of items to be ordered + +The Entity Relationship Diagram for the schema is as follows: + +![ERD](images/ERD.png) + +### Load the initial data + +After the Docker container has started, load the initial data by running the following command: + +```console +./gradlew run --args="LoadInitialData" +``` + +After the initial data has loaded, the following records should be stored in the tables. + +**`customer.customers` table** + +| customer_id | name | credit_limit | credit_total | +|-------------|---------------|--------------|--------------| +| 1 | Yamada Taro | 10000 | 0 | +| 2 | Yamada Hanako | 10000 | 0 | +| 3 | Suzuki Ichiro | 10000 | 0 | + +**`order.items` table** + +| item_id | name | price | +|---------|--------|-------| +| 1 | Apple | 1000 | +| 2 | Orange | 2000 | +| 3 | Grape | 2500 | +| 4 | Mango | 5000 | +| 5 | Melon | 3000 | + +## Execute transactions and retrieve data in the sample application + +The following sections describe how to execute transactions and retrieve data in the sample e-commerce application. + +### Get customer information + +Start with getting information about the customer whose ID is `1` by running the following command: + +```console +./gradlew run --args="GetCustomerInfo 1" +``` + +You should see the following output: + +```console +... +{"id": 1, "name": "Yamada Taro", "credit_limit": 10000, "credit_total": 0} +... +``` + +### Place an order + +Then, have customer ID `1` place an order for three apples and two oranges by running the following command: + +:::note + +The order format in this command is `./gradlew run --args="PlaceOrder :,:,..."`. + +::: + +```console +./gradlew run --args="PlaceOrder 1 1:3,2:2" +``` + +You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: + +```console +... +{"order_id": "dea4964a-ff50-4ecf-9201-027981a1566e"} +... +``` + +### Check order details + +Check details about the order by running the following command, replacing `` with the UUID for the `order_id` that was shown after running the previous command: + +```console +./gradlew run --args="GetOrder " +``` + +You should see a similar output as below, with different UUIDs for `order_id` and `timestamp`: + +```console +... +{"order": {"order_id": "dea4964a-ff50-4ecf-9201-027981a1566e","timestamp": 1650948340914,"customer_id": 1,"customer_name": "Yamada Taro","statement": [{"item_id": 1,"item_name": "Apple","price": 1000,"count": 3,"total": 3000},{"item_id": 2,"item_name": "Orange","price": 2000,"count": 2,"total": 4000}],"total": 7000}} +... +``` + +### Place another order + +Place an order for one melon that uses the remaining amount in `credit_total` for customer ID `1` by running the following command: + +```console +./gradlew run --args="PlaceOrder 1 5:1" +``` + +You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: + +```console +... +{"order_id": "bcc34150-91fa-4bea-83db-d2dbe6f0f30d"} +... +``` + +### Check order history + +Get the history of all orders for customer ID `1` by running the following command: + +```console +./gradlew run --args="GetOrders 1" +``` + +You should see a similar output as below, with different UUIDs for `order_id` and `timestamp`, which shows the history of all orders for customer ID `1` in descending order by timestamp: + +```console +... +{"order": [{"order_id": "dea4964a-ff50-4ecf-9201-027981a1566e","timestamp": 1650948340914,"customer_id": 1,"customer_name": "Yamada Taro","statement": [{"item_id": 1,"item_name": "Apple","price": 1000,"count": 3,"total": 3000},{"item_id": 2,"item_name": "Orange","price": 2000,"count": 2,"total": 4000}],"total": 7000},{"order_id": "bcc34150-91fa-4bea-83db-d2dbe6f0f30d","timestamp": 1650948412766,"customer_id": 1,"customer_name": "Yamada Taro","statement": [{"item_id": 5,"item_name": "Melon","price": 3000,"count": 1,"total": 3000}],"total": 3000}]} +... +``` + +### Check credit total + +Get the credit total for customer ID `1` by running the following command: + +```console +./gradlew run --args="GetCustomerInfo 1" +``` + +You should see the following output, which shows that customer ID `1` has reached their `credit_limit` in `credit_total` and cannot place anymore orders: + +```console +... +{"id": 1, "name": "Yamada Taro", "credit_limit": 10000, "credit_total": 10000} +... +``` + +Try to place an order for one grape and one mango by running the following command: + +```console +./gradlew run --args="PlaceOrder 1 3:1,4:1" +``` + +You should see the following output, which shows that the order failed because the `credit_total` amount would exceed the `credit_limit` amount: + +```console +... +java.lang.RuntimeException: Credit limit exceeded + at sample.Sample.placeOrder(Sample.java:205) + at sample.command.PlaceOrderCommand.call(PlaceOrderCommand.java:33) + at sample.command.PlaceOrderCommand.call(PlaceOrderCommand.java:8) + at picocli.CommandLine.executeUserObject(CommandLine.java:1783) + at picocli.CommandLine.access$900(CommandLine.java:145) + at picocli.CommandLine$RunLast.handle(CommandLine.java:2141) + at picocli.CommandLine$RunLast.handle(CommandLine.java:2108) + at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:1975) + at picocli.CommandLine.execute(CommandLine.java:1904) + at sample.command.SampleCommand.main(SampleCommand.java:35) +... +``` + +### Make a payment + +To continue making orders, customer ID `1` must make a payment to reduce the `credit_total` amount. + +Make a payment by running the following command: + +```console +./gradlew run --args="Repayment 1 8000" +``` + +Then, check the `credit_total` amount for customer ID `1` by running the following command: + +```console +./gradlew run --args="GetCustomerInfo 1" +``` + +You should see the following output, which shows that a payment was applied to customer ID `1`, reducing the `credit_total` amount: + +```console +... +{"id": 1, "name": "Yamada Taro", "credit_limit": 10000, "credit_total": 2000} +... +``` + +Now that customer ID `1` has made a payment, place an order for one grape and one melon by running the following command: + +```console +./gradlew run --args="PlaceOrder 1 3:1,4:1" +``` + +You should see a similar output as below, with a different UUID for `order_id`, which confirms that the order was successful: + +```console +... +{"order_id": "8911cab3-1c2b-4322-9386-adb1c024e078"} +... +``` + +## Stop the sample application + +To stop the sample application, stop the Docker container by running the following command: + +```console +docker-compose down +``` diff --git a/versioned_docs/version-3.X/scalardb-samples/multi-storage-transaction-sample/images/ERD.png b/versioned_docs/version-3.X/scalardb-samples/multi-storage-transaction-sample/images/ERD.png new file mode 100644 index 00000000..02100437 Binary files /dev/null and b/versioned_docs/version-3.X/scalardb-samples/multi-storage-transaction-sample/images/ERD.png differ diff --git a/versioned_docs/version-3.X/scalardb-samples/multi-storage-transaction-sample/images/overview.png b/versioned_docs/version-3.X/scalardb-samples/multi-storage-transaction-sample/images/overview.png new file mode 100644 index 00000000..16749f3b Binary files /dev/null and b/versioned_docs/version-3.X/scalardb-samples/multi-storage-transaction-sample/images/overview.png differ diff --git a/versioned_docs/version-3.X/scalardb-samples/scalardb-analytics-postgresql-sample/README.mdx b/versioned_docs/version-3.X/scalardb-samples/scalardb-analytics-postgresql-sample/README.mdx new file mode 100644 index 00000000..d29a7a24 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-samples/scalardb-analytics-postgresql-sample/README.mdx @@ -0,0 +1,305 @@ +--- +tags: + - Community +displayed_sidebar: docsEnglish +--- + +# Run Analytical Queries on Sample Data by Using ScalarDB Analytics with PostgreSQL + +This tutorial describes how to run analytical queries on sample data by using ScalarDB Analytics with PostgreSQL. + +## Overview + +This sample tutorial shows how you can run two types of queries: a single-table query and a multi-table query. + +### What you can do in this sample tutorial + +This sample tutorial shows how you can run the following types of queries: + +- Read data and calculate summaries. +- Join tables that span multiple storages. + +:::note + +You can run any arbitrary query that PostgreSQL supports on the imported tables in this sample tutorial. Since ScalarDB Analytics with PostgreSQL supports all queries that PostgreSQL supports, you can use not only join, aggregation, filtering, and ordering as shown in the example, but also the window function, lateral join, or various analytical operations. + +To see which types of queries PostgreSQL supports, see the [PostgreSQL documentation](https://www.postgresql.org/docs/current/index.html). + +::: + +## Prerequisites + +- [Docker](https://www.docker.com/get-started/) 20.10 or later with [Docker Compose](https://docs.docker.com/compose/install/) V2 or later +- [psql](https://www.postgresql.org/docs/current/app-psql.html) + +## Set up ScalarDB Analytics with PostgreSQL + +First, you must set up the database to run analytical queries with ScalarDB Analytics with PostgreSQL. If you haven't set up the database yet, please follow the instructions in [Getting Started](../../scalardb-analytics-postgresql/getting-started.mdx). + +### Schema details in ScalarDB + +In this sample tutorial, you have tables with the following schema in the ScalarDB database: + +```mermaid +erDiagram + "dynamons.customer" ||--|{ "postgresns.orders" : "custkey" + "dynamons.customer" { + int c_custkey + text c_name + text c_address + int c_nationkey + text c_phone + double c_acctbal + text c_mktsegment + text c_comment + } + "postgresns.orders" ||--|{ "cassandrans.lineitem" : "orderkey" + "postgresns.orders" { + int o_orderkey + int o_custkey + text o_orderstatus + double o_totalprice + text o_orderdate + text o_orderpriority + text o_clerk + int o_shippriority + text o_comment + } + "cassandrans.lineitem" { + int l_orderkey + int l_partkey + int l_suppkey + int l_linenumber + double l_quantity + double l_extendedprice + double l_discount + double l_tax + text l_returnflag + text l_linestatus + text l_shipdate + text l_commitdate + text l_receiptdate + text l_shipinstruct + text l_shipmode + text l_comment + } +``` + +For reference, this diagram shows the following: + +- `dynamons`, `postgresns`, and `cassandrans`. Namespaces that are mapped to the back-end storages of DynamoDB, PostgreSQL, and Cassandra, respectively. +- `dynamons.customer`. A table that represents information about customers. This table includes attributes like customer key, name, address, phone number, and account balance. +- `postgresns.orders`. A table that contains information about orders that customers have placed. This table includes attributes like order key, customer key, order status, order date, and order priority. +- `cassandrans.lineitem`. A table that represents line items associated with orders. This table includes attributes such as order key, part key, supplier key, quantity, price, and shipping date. + +### Schema details in PostgreSQL + +By running the Schema Importer when setting up ScalarDB, you can import the table schema in the ScalarDB database into the PostgreSQL database. More precisely, for each `namespace_name.table_name` table in the ScalarDB database, you will have a foreign table for `namespace_name._table_name` and a view for `namespace_name.table_name` in the PostgreSQL database. + +The created foreign table contains columns that are identical to the ScalarDB table and the transaction metadata columns that ScalarDB manages internally. Since the created view is defined to exclude the transaction metadata columns from the foreign table, the created view contains only the same columns as the ScalarDB table. + +You can find the schema of the ScalarDB tables in `schema.json`. For example, the `dynamons.customer` table is defined as follows: + +```json + "dynamons.customer": { + "transaction": true, + "partition-key": [ + "c_custkey" + ], + "columns": { + "c_custkey": "INT", + "c_name": "TEXT", + "c_address": "TEXT", + "c_nationkey": "INT", + "c_phone": "TEXT", + "c_acctbal": "DOUBLE", + "c_mktsegment": "TEXT", + "c_comment": "TEXT" + } + }, +``` + +To see the foreign table for `dynamons._customer` in the PostgreSQL database, run the following command and enter your PostgreSQL user password when prompted: + +```console +psql -U postgres -h localhost test -c '\d dynamons._customer'; +``` + +After entering your password, you should see the following output, which shows the same `c_` columns as in the `dynamons.customer` table: + +```console + Foreign table "dynamons._customer" + Column | Type | Collation | Nullable | Default | FDW options +------------------------+------------------+-----------+----------+---------+------------- + c_custkey | integer | | | | + c_name | text | | | | + c_address | text | | | | + c_nationkey | integer | | | | + c_phone | text | | | | + c_acctbal | double precision | | | | + c_mktsegment | text | | | | + c_comment | text | | | | + tx_id | text | | | | + tx_version | integer | | | | + tx_state | integer | | | | + tx_prepared_at | bigint | | | | + tx_committed_at | bigint | | | | + before_tx_id | text | | | | + before_tx_version | integer | | | | + before_tx_state | integer | | | | + before_tx_prepared_at | bigint | | | | + before_tx_committed_at | bigint | | | | + before_c_name | text | | | | + before_c_address | text | | | | + before_c_nationkey | integer | | | | + before_c_phone | text | | | | + before_c_acctbal | double precision | | | | + before_c_mktsegment | text | | | | + before_c_comment | text | | | | +Server: multi_storage_dynamodb +FDW options: (namespace 'dynamons', table_name 'customer') +``` + +As you can see in the foreign table, the table also contains the transaction metadata columns. These columns are required to ensure the Read Committed isolation level. + +To see the view for `dynamons.customer`, run the following command and enter your PostgreSQL user password when prompted: + +```console +psql -U postgres -h localhost test -c '\d dynamons.customer'; +``` + +After entering your password, you should see the following output: + +```console + View "dynamons.customer" + Column | Type | Collation | Nullable | Default +--------------+------------------+-----------+----------+--------- + c_custkey | integer | | | + c_name | text | | | + c_address | text | | | + c_nationkey | integer | | | + c_phone | text | | | + c_acctbal | double precision | | | + c_mktsegment | text | | | + c_comment | text | | | +``` + +The column definitions in this view are the same as the original table in the ScalarDB database. This view is created based on the foreign table explained above to expose only the valid data with the Read Committed isolation level by interpreting the transaction metadata columns. + +:::note + +Normally, you don't need to access the foreign tables directly. Instead, you can equate the views with the tables in the ScalarDB database. + +::: + +For details about type mapping between ScalarDB and PostgreSQL, see [Data-type mapping between ScalarDB and other databases](../../schema-loader.mdx#data-type-mapping-between-scalardb-and-other-databases). + +## Run analytical queries + +The following sections describe how to read data, calculate summaries, and join tables that span multiple storages. + +### Read data and calculate summaries + +You can run a query that reads data from `cassandrans.lineitem`, with the actual data stored in the Cassandra back-end, and calculates several summaries of the ordered line items by aggregating the data. + +To run the query, log in to the psql terminal by running the following command: + +```console +psql -U postgres -h localhost test +``` + +After entering your password, enter the following query into the psql terminal: + +```console +SELECT + l_returnflag, + l_linestatus, + sum(l_quantity) AS sum_qty, + sum(l_extendedprice) AS sum_base_price, + sum(l_extendedprice * (1 - l_discount)) AS sum_disc_price, + sum(l_extendedprice * (1 - l_discount) * (1 + l_tax)) AS sum_charge, + avg(l_quantity) AS avg_qty, + avg(l_extendedprice) AS avg_price, + avg(l_discount) AS avg_disc, + count(*) AS count_order +FROM + cassandrans.lineitem +WHERE + to_date(l_shipdate, 'YYYY-MM-DD') <= date '1998-12-01' - 3 +GROUP BY + l_returnflag, + l_linestatus +ORDER BY + l_returnflag, + l_linestatus; +``` + +You should see the following output: + +```console + l_returnflag | l_linestatus | sum_qty | sum_base_price | sum_disc_price | sum_charge | avg_qty | avg_price | avg_disc | count_order +--------------+--------------+---------+--------------------+--------------------+--------------------+---------------------+--------------------+---------------------+------------- + A | F | 1519 | 2374824.6560430005 | 1387363.5818635763 | 1962762.9341866106 | 26.6491228070175439 | 41663.590456894744 | 0.4150182982456142 | 57 + N | F | 98 | 146371.22954200002 | 85593.92837883368 | 121041.52567369482 | 32.6666666666666667 | 48790.409847333336 | 0.4098473333333333 | 3 + N | O | 5374 | 8007373.247144971 | 4685645.630765834 | 6624209.157932242 | 24.4272727272727273 | 36397.15112338623 | 0.414759749999999 | 220 + R | F | 1461 | 2190869.967642001 | 1284177.8484816086 | 1814150.7929095028 | 25.1896551724137931 | 37773.62013175864 | 0.41323520689655185 | 58 +(4 rows) +``` + +### Join tables that span multiple storages + +You can also run a query to join tables that are connected to the three back-end storages and calculate the unshipped orders with the highest revenue on a particular date. + +To run the query, log in to the psql terminal by running the following command: + +```console +psql -U postgres -h localhost test +``` + +After entering your password, enter the following query into the psql terminal: + +```console +SELECT + l_orderkey, + sum(l_extendedprice * (1 - l_discount)) AS revenue, + o_orderdate, + o_shippriority +FROM + dynamons.customer, + postgresns.orders, + cassandrans.lineitem +WHERE + c_mktsegment = 'AUTOMOBILE' + AND c_custkey = o_custkey + AND l_orderkey = o_orderkey + AND o_orderdate < '1995-03-15' + AND l_shipdate > '1995-03-15' +GROUP BY + l_orderkey, + o_orderdate, + o_shippriority +ORDER BY + revenue DESC, + o_orderdate, + l_orderkey +LIMIT 10; +``` + +You should see the following output: + +```console + l_orderkey | revenue | o_orderdate | o_shippriority +------------+--------------------+-------------+---------------- + 1071617 | 128186.94002748765 | 1995-03-10 | 0 + 1959075 | 33104.49713665398 | 1994-12-23 | 0 + 430243 | 19476.107574179696 | 1994-12-24 | 0 +(3 rows) +``` + +## Stop ScalarDB Analytics with PostgreSQL and the database + +To stop ScalarDB Analytics with PostgreSQL and the database, stop the Docker container by running the following command: + +```console +docker-compose down +``` diff --git a/versioned_docs/version-3.X/scalardb-samples/scalardb-analytics-spark-sample/README.mdx b/versioned_docs/version-3.X/scalardb-samples/scalardb-analytics-spark-sample/README.mdx new file mode 100644 index 00000000..93581b2e --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-samples/scalardb-analytics-spark-sample/README.mdx @@ -0,0 +1,455 @@ +--- +tags: + - Enterprise Option +displayed_sidebar: docsEnglish +--- + +# Getting Started with ScalarDB Analytics + +import WarningLicenseKeyContact from '/src/components/en-us/_warning-license-key-contact.mdx'; + +This tutorial describes how to run analytical queries on sample data by using ScalarDB Analytics. The source code is available at [https://github.com/scalar-labs/scalardb-samples/tree/main/scalardb-analytics-spark-sample](https://github.com/scalar-labs/scalardb-samples/tree/main/scalardb-analytics-spark-sample). + +ScalarDB Analytics in its current version leverages Apache Spark as its execution engine. It provides a unified view of ScalarDB-managed and non-ScalarDB-managed data sources by using a Spark custom catalog. By using ScalarDB Analytics, you can treat tables from these data sources as native Spark tables. This allows you to execute arbitrary Spark SQL queries seamlessly. For example, you can join a table stored in Cassandra with a table in PostgreSQL to perform cross-database analysis with ease. + +## Overview of the sample application + +This sample tutorial demonstrates how to configure Spark to enable ScalarDB Analytics and perform interactive analyses using `spark-sql` on tables provided by ScalarDB Analytics. + +## Prerequisites for this sample application + +- [Docker](https://www.docker.com/get-started/) 20.10 or later with [Docker Compose](https://docs.docker.com/compose/install/) V2 or later + + + +## Step 1: Set up ScalarDB Analytics + +### Clone the ScalarDB samples repository + +Open **Terminal**, and clone the ScalarDB samples repository by running the following command: + +```console +git clone https://github.com/scalar-labs/scalardb-samples +``` + +Then, go to the directory that contains the sample application by running the following command: + +```console +cd scalardb-samples/scalardb-analytics-spark-sample +``` + +### Set your license information + +ScalarDB Analytics requires valid license information to be specified in the Spark configuration. You can provide your license details in the **spark-defaults.conf** file. + + +Open the **spark-defaults.conf** file located in the **conf** directory of your Spark installation. Then, replace `` with your license key and `` with the PEM-encoded contents of your license certificate. + +```console +spark.sql.catalog.test_catalog.license.key +spark.sql.catalog.test_catalog.license.cert_pem +``` + +For additional configuration details required in the **spark-defaults.conf** file for setting up ScalarDB Analytics, refer to [ScalarDB Analytics configuration](#scalardb-analytics-configuration). + +## Step 2: Set up the sample databases + +To set up the sample databases, run the following command: + +```console +docker compose up -d --wait +``` + +This command starts three services locally: PostgreSQL, Cassandra, and MySQL. + +- **PostgreSQL:** Used independently (non-ScalarDB-managed). +- **Cassandra and MySQL:** Used as backend databases for ScalarDB (ScalarDB-managed). + +In this guide, PostgreSQL is referred to as a **non-ScalarDB-managed database**, which is not managed by ScalarDB transactions, while Cassandra and DynamoDB are referred to as **ScalarDB-managed databases**, which are managed by ScalarDB transactions. + +For non-ScalarDB-managed databases, sample data is automatically loaded when the Docker container is initialized, so no additional steps are required. For ScalarDB-managed databases, run the following command to load the sample data after starting the containers: + +```console +docker compose run --rm sample-data-loader +``` + +After completing the setup, the following tables should be available: + +- In PostgreSQL: + - `sample_ns.customer` +- In ScalarDB (backed by Cassandra): + - `cassandrans.lineitem` +- In ScalarDB (backed by MySQL): + - `mysqlns.order` + +According to the above, within ScalarDB, `cassandrans` and `mysqlns` are mapped to Cassandra and MySQL, respectively. + +For details about the table schema, including column definitions and data types, refer to [Schema details](#schema-details). Ensure that the sample data has been successfully loaded into these tables. + +## Step 3: Launch the Spark SQL console + +To launch the Spark SQL console, run the following command: + +```console +docker compose run --rm spark-sql +``` + +While launching the Spark SQL console, the ScalarDB Analytics catalog is initialized with the configuration in **spark-defaults.conf** and is registered as a Spark catalog named `test_catalog`. + +### Namespace mapping + +The following tables in the configured data sources are mapped to Spark SQL tables, allowing seamless querying across different data sources: + +- For PostgreSQL: + - `test_catalog.postgresql.sample_ns.customer` +- For ScalarDB (backed by Cassandra): + - `test_catalog.scalardb.cassandrans.lineitem` +- For ScalarDB (backed by MySQL): + - `test_catalog.scalardb.mysqlns.orders` + +For more details about how tables are mapped to Spark SQL tables, refer to [Namespace-mapping details](#namespace-mapping-details). + +Additionally, ScalarDB Analytics offers WAL-interpreted views for ScalarDB tables, simplifying common use cases. In this sample application, you have the following WAL-interpreted views available: + +- For ScalarDB (backed by Cassandra): + - `test_catalog.view.scalardb.cassandrans.lineitem` +- For ScalarDB (backed by MySQL): + - `test_catalog.view.scalardb.mysqlns.orders` + +In most cases, WAL-interpreted views are preferred over raw tables. In this tutorial, we will use the WAL-interpreted views for the ScalarDB tables. For detailed information on WAL-interpreted views, including their use cases and benefits, see [WAL-interpreted views for ScalarDB tables](#wal-interpreted-views-for-scalardb-tables). + +## Step 4: Run analytical queries + +Now, everything is set up, and you can run analytical queries on the sample data using the Spark SQL console. + +### Read data and calculate summaries + +You can run the following query to retrieve data from `test_catalog.scalardb.cassandrans.lineitem` in Cassandra and calculate aggregated metrics, including total quantity, average price, and total revenue for line items grouped by their return flag and line status. + +```sql +SELECT + l_returnflag, + l_linestatus, + sum(l_quantity) AS sum_qty, + sum(l_extendedprice) AS sum_base_price, + sum(l_extendedprice * (1 - l_discount)) AS sum_disc_price, + sum(l_extendedprice * (1 - l_discount) * (1 + l_tax)) AS sum_charge, + avg(l_quantity) AS avg_qty, + avg(l_extendedprice) AS avg_price, + avg(l_discount) AS avg_disc, + count(*) AS count_order +FROM + test_catalog.view.scalardb.cassandrans.lineitem +WHERE + to_date(l_shipdate, 'yyyy-MM-dd') <= date '1998-12-01' - 3 +GROUP BY + l_returnflag, + l_linestatus +ORDER BY + l_returnflag, + l_linestatus; +``` + +You should see the following output: + +```console +A F 1519 2374824.6560278563 1387364.2207725341 1962763.4654265852 26.649122807017545 41663.590456629056 0.41501802923479575 57 +N F 98 146371.2295412012 85593.96776336085 121041.55837332775 32.666666666666664 48790.409847067065 0.40984706454007996 3 +N O 5374 8007373.247086477 4685647.785126835 6624210.945739046 24.427272727272726 36397.15112312035 0.4147594809559689 220 +R F 1461 2190869.9676265526 1284178.4378283697 1814151.2807494882 25.189655172413794 37773.62013149229 0.41323493790730753 58 +``` + +### Join tables that span multiple data sources + +You can also run the following query to join tables from multiple data sources, including both ScalarDB-managed and non-ScalarDB-managed tables. This query joins customer, order, and line item data from PostgreSQL, MySQL, and Cassandra, identifying the top unshipped orders with the highest revenue on a specific date. This analysis helps prioritize shipments for maximum financial impact. + +```sql +SELECT + l_orderkey, + sum(l_extendedprice * (1 - l_discount)) AS revenue, + o_orderdate, + o_shippriority +FROM + test_catalog.postgresql.sample_ns.customer, + test_catalog.scalardb.mysqlns.orders, + test_catalog.scalardb.cassandrans.lineitem +WHERE + c_mktsegment = 'AUTOMOBILE' + AND c_custkey = o_custkey + AND l_orderkey = o_orderkey + AND o_orderdate < '1995-03-15' + AND l_shipdate > '1995-03-15' +GROUP BY + l_orderkey, + o_orderdate, + o_shippriority +ORDER BY + revenue DESC, + o_orderdate, + l_orderkey +LIMIT 10; +``` + +You should see the following output: + +```console +1071617 128186.99915996166 1995-03-10 0 +1959075 33104.51278645416 1994-12-23 0 +430243 19476.115819260962 1994-12-24 0 +``` + +:::note + +You can also run any arbitrary query that Apache Spark and Spark SQL support on the imported tables in this sample tutorial. Since ScalarDB Analytics supports all queries that Spark SQL supports, you can do not only selections (filtering), joins, aggregations, and ordering, as shown in the example, but also window functions, lateral joins, and other various operations. + +To see which types of queries Spark SQL supports, see the [Spark SQL documentation](https://spark.apache.org/docs/latest/sql-ref.html). + +::: + +## Step 5: Stop the sample application + +To stop the sample application and remove all associated volumes, run the following command. This action shuts down all services and deletes any persisted data stored in the volumes, resetting the application state: + +```console +docker compose down -v +``` + +## Reference + +This section contains other details related to ScalarDB Analytics, like configurations and schema details. + +### ScalarDB Analytics configuration + +You can configure ScalarDB Analytics in the Spark configuration, such as in the `spark-defaults.conf` file. This section contains brief explanations of the configurations for ScalarDB Analytics in this sample application. + +#### Common configurations + +The following are common configurations for ScalarDB Analytics: + +```console +spark.sql.catalog.test_catalog com.scalar.db.analytics.spark.ScalarDbAnalyticsCatalog +spark.sql.extensions com.scalar.db.analytics.spark.extension.ScalarDbAnalyticsExtensions +``` + +The first line specifies the Spark catalog plugin implementation class. You must always set this to `com.scalar.db.analytics.spark.ScalarDbAnalyticsCatalog` to enable the ScalarDB Analytics catalog in Spark SQL. + +:::note + +You can set an arbitrary string as the catalog name, which is `test_catalog` in this example. The configured catalog name will be used as a part of the table identifier in the Spark SQL query. + +::: + +The second line specifies the Spark SQL extension implementation class. You must always set this to `com.scalar.db.analytics.spark.extension.ScalarDbAnalyticsExtensions`, along with the `spark.sql.catalog.test_catalog` configuration, mentioned above. + +#### License information + +The following are the license configurations for ScalarDB Analytics: + +```apacheconf +spark.sql.catalog.test_catalog.license.key +spark.sql.catalog.test_catalog.license.cert_pem +``` + +These lines provide the license information for ScalarDB Analytics. As explained above, you must replace the placeholders with your license information before launching the Spark SQL console. + +#### Data source configurations for ScalarDB-managed databases + +The following are the data source configurations for ScalarDB-managed databases for ScalarDB Analytics: + +```apacheconf +spark.sql.catalog.test_catalog.data_source.scalardb.type scalardb +spark.sql.catalog.test_catalog.data_source.scalardb.config_path /etc/scalardb.properties +``` + +The first line specifies the data source type. You must always set this to `scalardb` to configure the data source for ScalarDB-managed databases. The second line specifies the path to the configuration file for the ScalarDB data source, which is the only required configuration for the ScalarDB data source. + +:::note + +You can set an arbitrary string as the data source name, which is `scalardb` in this example. The configured data source names will be used as a part of the table identifier in the Spark SQL query. + +::: + +#### Data source configurations for non-ScalarDB-managed databases + +The following are the data source configurations for non-ScalarDB-managed databases for ScalarDB Analytics: + +```apacheconf +spark.sql.catalog.test_catalog.data_source.postgresql.type postgresql +spark.sql.catalog.test_catalog.data_source.postgresql.host postgres +spark.sql.catalog.test_catalog.data_source.postgresql.port 5432 +spark.sql.catalog.test_catalog.data_source.postgresql.username postgres +spark.sql.catalog.test_catalog.data_source.postgresql.password postgres +spark.sql.catalog.test_catalog.data_source.postgresql.database sampledb +``` + +These lines configure the data source PostgreSQL as a non-ScalarDB-managed database. The first line specifies the data source type, and the rest of the lines specify the data source-specific configurations, which is the connection information for the PostgreSQL data source. The data source–specific configurations may vary depending on the data source type. + +:::note + +You can set an arbitrary string as the data source name, which is `postgresql` in this example, in the same way as the ScalarDB data source. + +::: + +### Schema details + +The following entity relationship diagram illustrates the relationships between the tables across PostgreSQL, MySQL, and Cassandra, with foreign keys linking customers, orders, and line items. + +```mermaid +erDiagram + "postgresql.sample_ns.customer" ||--|{ "scalardb.mysqlns.orders" : "custkey" + "postgresql.sample_ns.customer" { + int c_custkey + text c_name + text c_address + int c_nationkey + text c_phone + double c_acctbal + text c_mktsegment + text c_comment + } + "scalardb.mysqlns.orders" ||--|{ "scalardb.cassandrans.lineitem" : "orderkey" + "scalardb.mysqlns.orders" { + int o_orderkey + int o_custkey + text o_orderstatus + double o_totalprice + text o_orderdate + text o_orderpriority + text o_clerk + int o_shippriority + text o_comment + } + "scalardb.cassandrans.lineitem" { + int l_orderkey + int l_partkey + int l_suppkey + int l_linenumber + double l_quantity + double l_extendedprice + double l_discount + double l_tax + text l_returnflag + text l_linestatus + text l_shipdate + text l_commitdate + text l_receiptdate + text l_shipinstruct + text l_shipmode + text l_comment + } +``` + +- `postgresql.sample_ns.customer` comes from PostgreSQL, which is not managed by ScalarDB. +- `scalardb.mysqlns.orders` and `scalardb.cassandrans.lineitem` come from ScalarDB, which are backed by MySQL and Cassandra, respectively. + +The following are brief descriptions of the tables: + +- **`postgresql.sample_ns.customer`.** A table that represents information about customers. This table includes attributes like customer key, name, address, phone number, and account balance. +- **`scalardb.mysqlns.orders`.** A table that contains information about orders that customers have placed. This table includes attributes like order key, customer key, order status, order date, and order priority. +- **`scalardb.cassandrans.lineitem`.** A table that represents line items associated with orders. This table includes attributes such as order key, part key, supplier key, quantity, price, and shipping date. + +### Namespace-mapping details + +The tables of each configured data source are mapped to the Spark SQL identifier by using the following format: + +```console +...`. +``` + +The following explains each part of the table identifier: + +- **``.** The catalog name configured in spark-defaults.conf. This identifies the ScalarDB Analytics catalog in Spark SQL. +- **``.** The data source name configured in spark-defaults.conf, representing the specific type of data source, such as postgresql or scalardb. +- **``.** The namespace name in the data source. For example: + - In an RDBMS like PostgreSQL or MySQL, this corresponds to the schema. + - In NoSQL databases like Cassandra, this may refer to a keyspace. +- **``.** The name of the table in the namespace. + +In this example, the following tables are available: + +- For PostgreSQL: + - test_catalog.postgresql.sample_ns.customer +- For ScalarDB (backed by Cassandra): + - test_catalog.scalardb.cassandrans.lineitem +- For ScalarDB (backed by MySQL): + - test_catalog.scalardb.mysqlns.orders + +This mapping allows you to access and query tables from different data sources seamlessly by using Spark SQL. + +### WAL-interpreted views for ScalarDB tables + +ScalarDB tables that are transaction-enabled include transaction metadata columns in the raw tables stored in the underlying data sources. Since ScalarDB Analytics maps these raw tables directly to Spark SQL tables, you will see transaction metadata columns when describing these tables in Spark SQL. You can see these columns by running the following command: + +```sql +DESCRIBE test_catalog.scalardb.mysqlns.orders; +``` + +You should see the following output: + +```console +o_orderkey int +o_custkey int +o_orderstatus string +o_totalprice double +o_orderdate string +o_orderpriority string +o_clerk string +o_shippriority int +o_comment string +tx_id string +tx_state int +tx_version int +tx_prepared_at bigint +tx_committed_at bigint +before_tx_id string +before_tx_state int +before_tx_version int +before_tx_prepared_at bigint +before_tx_committed_at bigint +before_o_orderstatus string +before_o_clerk string +before_o_orderdate string +before_o_shippriority int +before_o_custkey int +before_o_totalprice double +before_o_comment string +before_o_orderpriority string +``` + +In many cases, you may not need the transaction metadata columns in your queries. To simplify this, ScalarDB Analytics provides WAL-interpreted views. WAL-interpreted views hide transaction metadata columns and expose only user-defined columns, simplifying queries. For example, use WAL-interpreted views when performing read-only analytics or when transaction metadata is not needed for analysis. Additionally, WAL-interpreted views guarantee read-committed consistency by interpreting the transaction metadata columns internally. + +#### WAL-interpreted view naming convention in Spark SQL + +WAL-interpreted views are prefixed with `view.` before the data source part of the table identifier. For example, the following WAL-interpreted views are available for ScalarDB tables: + +- For ScalarDB (backed by Cassandra): + - test_catalog.view.scalardb.cassandrans.lineitem +- For ScalarDB (backed by MySQL): + - test_catalog.view.scalardb.mysqlns.orders + +For example, to see the WAL-interpreted view for the ScalarDB table backed by Cassandra, run the following command: + +```sql +DESCRIBE test_catalog.view.scalardb.cassandrans.lineitem; +``` + +You should see the following output: + +```console +l_orderkey int +l_linenumber int +l_comment string +l_commitdate string +l_discount double +l_extendedprice double +l_linestatus string +l_partkey int +l_quantity int +l_receiptdate string +l_returnflag string +l_shipdate string +l_shipinstruct string +l_shipmode string +l_suppkey int +l_tax double +``` diff --git a/versioned_docs/version-3.X/scalardb-samples/spring-data-microservice-transaction-sample/README.mdx b/versioned_docs/version-3.X/scalardb-samples/spring-data-microservice-transaction-sample/README.mdx new file mode 100644 index 00000000..08a602db --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-samples/spring-data-microservice-transaction-sample/README.mdx @@ -0,0 +1,529 @@ +--- +tags: + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Sample application of Spring Data JDBC for ScalarDB with Microservice Transactions + +import WarningLicenseKeyContact from '/src/components/en-us/_warning-license-key-contact.mdx'; + +This tutorial describes how to create a sample Spring Boot application for microservice transactions by using Spring Data JDBC for ScalarDB. + +For details about these features, see [Two-phase Commit Transactions](../../two-phase-commit-transactions.mdx) and [Guide of Spring Data JDBC for ScalarDB](../../scalardb-sql/spring-data-guide.mdx). + +## Prerequisites for this sample application + +- OpenJDK LTS version (8, 11, 17, or 21) from [Eclipse Temurin](https://adoptium.net/temurin/releases/) +- [Docker](https://www.docker.com/get-started/) 20.10 or later with [Docker Compose](https://docs.docker.com/compose/install/) V2 or later + +:::note + +This sample application has been tested with OpenJDK from Eclipse Temurin. ScalarDB itself, however, has been tested with JDK distributions from various vendors. For details about the requirements for ScalarDB, including compatible JDK distributions, please see [Requirements](../../requirements.mdx). + +::: + + + +## Sample application + +### Overview + +This tutorial illustrates the process of creating a sample e-commerce application, where items can be ordered and paid for with a line of credit through [transactions with a two-phase commit interface](../../two-phase-commit-transactions.mdx) in ScalarDB. + +There are two microservices called the *Customer Service* and the *Order Service* based on the [*Database-per-service* pattern](https://microservices.io/patterns/data/database-per-service.html) in this sample application. + +The Customer Service manages customers' information including credit card information like a credit limit and a credit total. +The Order Service is responsible for order operations like placing an order and getting order histories. +Each service has gRPC endpoints. Clients call the endpoints, and the services call the endpoints each other as well. +The Customer Service and the Order Service use MySQL and Cassandra through ScalarDB, respectively. + +![Overview](images/overview.png) + +Each service accesses the databases through its own dedicated ScalarDB Cluster. + +:::note +Both ScalarDB Clusters access a small coordinator database used for the Consensus Commit protocol. In this sample application, for ease of setup and explanation, the coordinator database is co-located in the same Cassandra instance of the Order Service, but of course, the coordinator database can be managed as a separate database. + +Also, application-specific error handling, authentication processing, etc. are omitted in the sample application since it focuses on explaining how to use ScalarDB. +For details about handling exceptions in ScalarDB, see [How to handle exceptions](../../two-phase-commit-transactions.mdx#how-to-handle-exceptions). + +::: + +### Service endpoints + +The endpoints defined in the services are as follows: + +- Customer Service + - `getCustomerInfo` + - `payment` + - `prepare` + - `validate` + - `commit` + - `rollback` + - `repayment` + +- Order Service + - `placeOrder` + - `getOrder` + - `getOrders` + +### What you can do in this sample application + +The sample application supports the following types of transactions: + +- Get customer information through the `getCustomerInfo` endpoint of the Customer Service. +- Place an order by using a line of credit through the `placeOrder` endpoint of the Order Service and the `payment`, `prepare`, `validate`, `commit`, and `rollback` endpoints of the Customer Service. + - Checks if the cost of the order is below the customer's credit limit. + - If the check passes, records the order history and updates the amount the customer has spent. +- Get order information by order ID through the `getOrder` endpoint of the Order Service and the `getCustomerInfo`, `prepare`, `validate`, `commit`, and `rollback` endpoints of the Customer Service. +- Get order information by customer ID through the `getOrders` endpoint of the Order Service and the `getCustomerInfo`, `prepare`, `validate`, `commit`, and `rollback` endpoints of the Customer Service. +- Make a payment through the `repayment` endpoint of the Customer Service. + - Reduces the amount the customer has spent. + +:::note + +The `getCustomerInfo` endpoint works as a participant service endpoint when receiving a transaction ID from the coordinator. + +::: + +## Configuration for ScalarDB Cluster + +[The configuration for ScalarDB Cluster for the Customer Service](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/scalardb-cluster-node-for-customer-service.properties) is as follows: + +```properties +scalar.db.storage=multi-storage +scalar.db.multi_storage.storages=cassandra,mysql +scalar.db.multi_storage.storages.cassandra.storage=cassandra +scalar.db.multi_storage.storages.cassandra.contact_points=cassandra-1 +scalar.db.multi_storage.storages.cassandra.username=cassandra +scalar.db.multi_storage.storages.cassandra.password=cassandra +scalar.db.multi_storage.storages.mysql.storage=jdbc +scalar.db.multi_storage.storages.mysql.contact_points=jdbc:mysql://mysql-1:3306/ +scalar.db.multi_storage.storages.mysql.username=root +scalar.db.multi_storage.storages.mysql.password=mysql +scalar.db.multi_storage.namespace_mapping=customer_service:mysql,coordinator:cassandra +scalar.db.multi_storage.default_storage=mysql +scalar.db.consensus_commit.isolation_level=SERIALIZABLE + +scalar.db.cluster.node.standalone_mode.enabled=true +scalar.db.sql.enabled=true + +# License key configurations +scalar.db.cluster.node.licensing.license_key= +scalar.db.cluster.node.licensing.license_check_cert_pem= +``` + +- `scalar.db.sql.connection_mode`: This configuration decides how to connect to ScalarDB. +- `scalar.db.storage`: Specifying `multi-storage` is necessary to use Multi-storage Transactions in ScalarDB. +- `scalar.db.multi_storage.storages`: Your storage names must be defined here. +- `scalar.db.multi_storage.storages.cassandra.*`: These configurations are for the `cassandra` storage, which is one of the storage names defined in `scalar.db.multi_storage.storages`. You can configure all the `scalar.db.*` properties for the `cassandra` storage here. +- `scalar.db.multi_storage.storages.mysql.*`: These configurations are for the `mysql` storage, which is one of the storage names defined in `scalar.db.multi_storage.storages`. You can configure all the `scalar.db.*` properties for the `mysql` storage here. +- `scalar.db.multi_storage.namespace_mapping`: This configuration maps the namespaces to the storage. In this sample application, operations for `customer_service` namespace tables are mapped to the `mysql` storage and operations for `order_service` namespace tables are mapped to the `cassandra` storage. You can also define which storage is mapped for the `coordinator` namespace that is used in Consensus Commit transactions. +- `scalar.db.multi_storage.default_storage`: This configuration sets the default storage that is used for operations on unmapped namespace tables. +- `scalar.db.sql.default_transaction_mode`: Specifying `two_phase_commit_transaction` is necessary to use Two-Phase Commit Transactions mode in ScalarDB. +- `scalar.db.consensus_commit.isolation_level`: This configuration decides the isolation level used for ConsensusCommit. + +For details, see [Multi-Storage Transactions](../../multi-storage-transactions.mdx). + +[The configuration for ScalarDB Cluster for the Order Service](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/scalardb-cluster-node-for-order-service.properties) is as follows: + +```properties +scalar.db.storage=cassandra +scalar.db.contact_points=cassandra-1 +scalar.db.username=cassandra +scalar.db.password=cassandra +scalar.db.consensus_commit.isolation_level=SERIALIZABLE + +scalar.db.cluster.node.standalone_mode.enabled=true +scalar.db.sql.enabled=true + +# License key configurations +scalar.db.cluster.node.licensing.license_key= +scalar.db.cluster.node.licensing.license_check_cert_pem= +``` + +- `scalar.db.storage`: `cassandra` is specified since this servcise uses only Cassandra as an underlying database. +- `scalar.db.contact_points`: This configuration specifies the contact points (e.g., host) for connecting to Cassandra. +- `scalar.db.username`: This configuration specifies the username for connecting to Cassandra. +- `scalar.db.password`: This configuration specifies the password for connecting to Cassandra. +- `scalar.db.sql.default_namespace_name`: This configuration sets the default namespace to `order_service`, eliminating the need for the application to specify namespaces. +- `scalar.db.sql.default_transaction_mode`: Specifying `two_phase_commit_transaction` is necessary to use Two-Phase Commit Transactions mode in ScalarDB. +- `scalar.db.consensus_commit.isolation_level`: This configuration decides the isolation level used for ConsensusCommit. + +In this sample application, the ScalarDB Clusters are running in standalone mode (`scalar.db.cluster.node.standalone_mode.enabled=true`). + +Also, you need to set the license key (trial license or commercial license) for the ScalarDB Clusters in the configuration file. For details, see [How to Configure a Product License Key](../../scalar-licensing/index.mdx). + +## Setup + +### Clone the ScalarDB samples repository + +Open Terminal, then clone the ScalarDB samples repository by running the following command: + +```console +git clone https://github.com/scalar-labs/scalardb-samples +``` + +Then, go to the directory with this sample by running the following command: + +```console +cd scalardb-samples/spring-data-microservice-transaction-sample +``` + +### Set the license key + +Set the license key (trial license or commercial license) for the ScalarDB Clusters in the configuration files [`scalardb-cluster-node-for-customer-service.properties`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/scalardb-cluster-node-for-customer-service.properties) and [`scalardb-cluster-node-for-order-service.properties`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/scalardb-cluster-node-for-order-service.properties). For details, see [How to Configure a Product License Key](../../scalar-licensing/index.mdx). + +### Start Cassandra, MySQL, and ScalarDB Clusters + +To start Cassandra, MySQL, and ScalarDB Clusters, you need to run the following `docker-compose` command: + +```console +docker-compose up -d cassandra mysql scalardb-cluster-node-for-customer-service scalardb-cluster-node-for-order-service +``` + +:::note + +You will need to wait more than one minute for the containers to be fully started. + +::: + +### Load schema + +The database schema (the method in which the data will be organized) for the sample application has already been defined in [`schema-for-customer-service.sql`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/schema-for-customer-service.sql) for the Customer Service and [`schema-for-order-service.sql`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/schema-for-order-service.sql) for the Order Service. + +To apply the schema, go to the [ScalarDB Releases](https://github.com/scalar-labs/scalardb/releases) page and download the SQL CLI tool that matches the version of ScalarDB that you want to use. + +#### MySQL + +To load the schema for [`schema-for-customer-service.sql`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/schema-for-customer-service.sql) into MySQL, run the following command, replacing `` with the version of the ScalarDB Schema Loader that you downloaded: + +```console +java -jar scalardb-cluster-sql-cli--all.jar --config scalardb-cluster-node-for-customer-service-client.properties --file schema-for-customer-service.sql +``` + +#### Cassandra + +To load the schema for [`schema-for-order-service.sql`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/schema-for-order-service.sql) into Cassandra, run the following command, replacing `` with the version of the ScalarDB Schema Loader that you downloaded: + +```console +java -jar scalardb-cluster-sql-cli--all.jar --config scalardb-cluster-node-for-order-service-client.properties --file schema-for-order-service.sql +``` + +#### Schema details + +As shown in [`schema-for-customer-service.sql`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/schema-for-customer-service.sql), all the tables for the Customer Service are created in the `customer_service` namespace. + +- `customer_service.customers`: a table that manages customers' information + - `credit_limit`: the maximum amount of money a lender will allow each customer to spend when using a line of credit + - `credit_total`: the amount of money that each customer has already spent by using their line of credit + +As shown in [`schema-for-order-service.sql`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/schema-for-order-service.sql), all the tables for the Order Service are created in the `order_service` namespace. + +- `order_service.orders`: a table that manages order information +- `order_service.statements`: a table that manages order statement information +- `order_service.items`: a table that manages information of items to be ordered + +The Entity Relationship Diagram for the schema is as follows: + +![ERD](images/ERD.png) + +### Start Microservices + +First, you need to build the docker images of the sample application with the following command: + +```console +./gradlew docker +``` + +Then, you can start the microservices with the following `docker-compose` command: + +```console +docker-compose up -d customer-service order-service +``` + +### Initial data + +When the microservices start, the initial data is loaded automatically. + +After the initial data has loaded, the following records should be stored in the tables: + +- For the `customer_service.customers` table: + +| customer_id | name | credit_limit | credit_total | +|-------------|---------------|--------------|--------------| +| 1 | Yamada Taro | 10000 | 0 | +| 2 | Yamada Hanako | 10000 | 0 | +| 3 | Suzuki Ichiro | 10000 | 0 | + +- For the `order_service.items` table: + +| item_id | name | price | +|---------|--------|-------| +| 1 | Apple | 1000 | +| 2 | Orange | 2000 | +| 3 | Grape | 2500 | +| 4 | Mango | 5000 | +| 5 | Melon | 3000 | + +## Run the sample application + +Let's start with getting information about the customer whose ID is `1`: + +```console +./gradlew :client:run --args="GetCustomerInfo 1" +... +{"id": 1,"name": "Yamada Taro","credit_limit": 10000} +... +``` + +At this time, `credit_total` isn't shown, which means the current value of `credit_total` is `0`. + +Then, place an order for three apples and two oranges by using customer ID `1`. +Note that the order format is `:,:,...`: + +```console +./gradlew :client:run --args="PlaceOrder 1 1:3,2:2" +... +{"order_id": "415a453b-cfee-4c48-b8f6-d103d3e10bdb"} +... +``` + +You can see that running this command shows the order ID. + +Let's check the details of the order by using the order ID: + +```console +./gradlew :client:run --args="GetOrder 415a453b-cfee-4c48-b8f6-d103d3e10bdb" +... +{"order": {"order_id": "415a453b-cfee-4c48-b8f6-d103d3e10bdb","timestamp": 1686555272435,"customer_id": 1,"customer_name": "Yamada Taro","statement": [{"item_id": 1,"item_name": "Apple","price": 1000,"count": $ +,"total": 3000},{"item_id": 2,"item_name": "Orange","price": 2000,"count": 2,"total": 4000}],"total": 7000}} +... +``` + +Then, let's place another order and get the order history of customer ID `1`: + +```console +./gradlew :client:run --args="PlaceOrder 1 5:1" +... +{"order_id": "069be075-98f7-428c-b2e0-6820693fc41b"} +... +./gradlew :client:run --args="GetOrders 1" +... +{"order": [{"order_id": "069be075-98f7-428c-b2e0-6820693fc41b","timestamp": 1686555279366,"customer_id": 1,"customer_name": "Yamada Taro","statement": [{"item_id": 5,"item_name": "Melon","price": 3000,"count": 1,"total": 3000}],"total": 3000},{"order_id": "415a453b-cfee-4c48-b8f6-d103d3e10bdb","timestamp": 1686555272435,"customer_id": 1,"customer_name": "Yamada Taro","statement": [{"item_id": 1,"item_name": "Apple","price": 1000,"count": 3,"total": 3000},{"item_id": 2,"item_name": "Orange","price": 2000,"count": 2,"total": 4000}],"total": 7000}]} +... +``` + +This order history is shown in descending order by timestamp. + +The customer's current `credit_total` is `10000`. +Since the customer has now reached their `credit_limit`, which was shown when retrieving their information, they cannot place anymore orders. + +```console +./gradlew :client:run --args="GetCustomerInfo 1" +... +{"id": 1,"name": "Yamada Taro","credit_limit": 10000,"credit_total": 10000} +... +./gradlew :client:run --args="PlaceOrder 1 3:1,4:1" +... +io.grpc.StatusRuntimeException: FAILED_PRECONDITION: Credit limit exceeded. creditTotal:10000, payment:7500 + at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271) + at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252) + at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165) + at sample.rpc.OrderServiceGrpc$OrderServiceBlockingStub.placeOrder(OrderServiceGrpc.java:296) + at sample.client.command.PlaceOrderCommand.call(PlaceOrderCommand.java:38) + at sample.client.command.PlaceOrderCommand.call(PlaceOrderCommand.java:12) + at picocli.CommandLine.executeUserObject(CommandLine.java:2041) + at picocli.CommandLine.access$1500(CommandLine.java:148) + at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461) + at picocli.CommandLine$RunLast.handle(CommandLine.java:2453) + at picocli.CommandLine$RunLast.handle(CommandLine.java:2415) + at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273) + at picocli.CommandLine$RunLast.execute(CommandLine.java:2417) + at picocli.CommandLine.execute(CommandLine.java:2170) + at sample.client.Client.main(Client.java:39) +... +``` + +After making a payment, the customer will be able to place orders again. + +```console +./gradlew :client:run --args="Repayment 1 8000" +... +./gradlew :client:run --args="GetCustomerInfo 1" +... +{"id": 1,"name": "Yamada Taro","credit_limit": 10000,"credit_total": 2000} +... +./gradlew :client:run --args="PlaceOrder 1 3:1,4:1" +... +{"order_id": "b6adabd8-0a05-4109-9618-3420fea3161f"} +... +``` + +## Clean up + +To stop Cassandra, MySQL and the Microservices, run the following command: + +```console +docker-compose down +``` + +## Reference - How the microservice transaction is achieved + +The transactions for placing an order, getting a single order, and getting the history of orders achieve the microservice transaction. This section focuses on how the transactions that span the Customer Service and the Order Service are implemented by placing an order as an example. + +The following sequence diagram shows the transaction for placing an order: + +![Sequence Diagram](images/sequence_diagram.png) + +### 1. Transaction with a two-phase commit interface is started + +When a client sends a request to place an order to the Order Service, `OrderService.placeOrder()` is called, and the microservice transaction starts. + +At first, the Order Service starts a transaction with a two-phase commit interface with `ScalarDbTwoPcRepository.executeTwoPcTransaction()` as follows. For reference, see [`OrderService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/order-service/src/main/java/sample/order/OrderService.java). + +```java +// Start a two-phase commit interface transaction +TwoPcResult result = orderRepository.executeTwoPcTransaction(txId -> { + ... +}, ...); +``` + +The actions in [CRUD operations are executed](#2-crud-operations-are-executed), [Transaction is committed by using the two-phase commit protocol](#3-transaction-is-committed-by-using-the-two-phase-commit-protocol), and [Error handling](#error-handling) are automatically performed by the API. + +### 2. CRUD operations are executed + +After the transaction with a two-phase commit interface starts, CRUD operations are executed by `ScalarDbTwoPcRepository.executeTwoPcTransaction()`. The Order Service puts the order information in the `order_service.orders` table and the detailed information in the `order_service.statements` table as follows. For reference, see [`OrderService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/order-service/src/main/java/sample/order/OrderService.java). + +```java +// Put the order info into the `orders` table +orderRepository.insert(order); + +AtomicInteger amount = new AtomicInteger(); +for (ItemOrder itemOrder : request.getItemOrderList()) { + int itemId = itemOrder.getItemId(); + int count = itemOrder.getCount(); + // Retrieve the item info from the `items` table + Optional itemOpt = itemRepository.findById(itemId); + if (!itemOpt.isPresent()) { + String message = "Item not found: " + itemId; + responseObserver.onError( + Status.NOT_FOUND.withDescription(message).asRuntimeException()); + throw new ScalarDbNonTransientException(message); + } + Item item = itemOpt.get(); + + int cost = item.price * count; + // Put the order statement into the `statements` table + statementRepository.insert(new Statement(itemId, orderId, count)); + // Calculate the total amount + amount.addAndGet(cost); +} +``` + +Then, the Order Service calls the `payment` gRPC endpoint of the Customer Service along with the transaction ID. For reference, see [`OrderService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/order-service/src/main/java/sample/order/OrderService.java). + +```java +customerServiceStub.payment( + PaymentRequest.newBuilder() + .setTransactionId(transactionId) + .setCustomerId(customerId) + .setAmount(amount) + .build()); +``` + +The `payment` endpoint of the Customer Service first joins the transaction with `ScalarDbTwoPcRepository.joinTransactionOnParticipant()` as follows. For reference, see [`CustomerService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/customer-service/src/main/java/sample/customer/CustomerService.java). + +```java +customerRepository.joinTransactionOnParticipant(request.getTransactionId(), ...); +``` + +The endpoint then gets the customer information and checks if the customer's credit total exceeds the credit limit after the payment. If the credit total does not exceed the credit limit, the endpoint updates the customer's credit total. For reference, see [`CustomerService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/customer-service/src/main/java/sample/customer/CustomerService.java). + +```java +Customer customer = getCustomer(responseObserver, request.getCustomerId()); + +int updatedCreditTotal = customer.creditTotal + request.getAmount(); +// Check if the credit total exceeds the credit limit after payment +if (updatedCreditTotal > customer.creditLimit) { + String message = String.format( + "Credit limit exceeded. creditTotal:%d, payment:%d", customer.creditTotal, request.getAmount()); + responseObserver.onError( + Status.FAILED_PRECONDITION.withDescription(message).asRuntimeException()); + throw new ScalarDbNonTransientException(message); +} + +// Reduce `credit_total` for the customer +customerRepository.update(customer.withCreditTotal(updatedCreditTotal)); +``` + +### 3. Transaction is committed by using the two-phase commit protocol + +After the Order Service receives the update that the payment succeeded, the Order Service tries to commit the transaction. + +The `ScalarDbTwoPcRepository.executeTwoPcTransaction()` API, which called on the Order Service, automatically performs preparations, validations, and commits of both the local Order Service and the remote Customer Service. These steps are executed sequentially after the above CRUD operations successfully finish. The implementations to invoke `prepare`, `validate`, and `commit` gRPC endpoints of the Customer Service need to be passed as parameters to the API. For reference, see [`OrderService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/order-service/src/main/java/sample/order/OrderService.java). + +```java +TwoPcResult result = orderRepository.executeTwoPcTransaction(txId ->{ + ... + }, + + Collections.singletonList( + RemotePrepareCommitPhaseOperations.createSerializable( + this::callPrepareEndpoint, + this::callValidateEndpoint, + this::callCommitEndpoint, + this::callRollbackEndpoint + ) + ) +); +``` + +![Sequence Diagram of High Level 2PC API](images/seq-diagram-high-level-2pc-api.png) + +In the `prepare` endpoint of the Customer Service, the endpoint resumes and prepares the transaction by using `ScalarDbTwoPcRepository.prepareTransactionOnParticipant()`. For reference, see [`CustomerService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/customer-service/src/main/java/sample/customer/CustomerService.java). + +```java +customerRepository.prepareTransactionOnParticipant(request.getTransactionId()); +``` + +In the `validate` endpoint of the Customer Service, the endpoint resumes and validates the transaction by using `ScalarDbTwoPcRepository.validateTransactionOnParticipant()`. For reference, see [`CustomerService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/customer-service/src/main/java/sample/customer/CustomerService.java). + +```java +customerRepository.validateTransactionOnParticipant(request.getTransactionId()); +``` + +In the `commit` endpoint of the Customer Service, the endpoint resumes and commits the transaction by using `ScalarDbTwoPcRepository.commitTransactionOnParticipant()`. For reference, see [`CustomerService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/customer-service/src/main/java/sample/customer/CustomerService.java). + +```java +customerRepository.commitTransactionOnParticipant(request.getTransactionId()); +``` + +### Error handling + +If an error happens while executing a transaction, `ScalarDbTwoPcRepository.executeTwoPcTransaction()` will automatically roll back the transaction in both the local Order Service and the remote Customer Service. The implementation to invoke the `rollback` gRPC endpoint of the Customer Service also needs to be passed as a parameter to the API with other ones. For reference, see [`OrderService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/order-service/src/main/java/sample/order/OrderService.java). + +```java +TwoPcResult result = orderRepository.executeTwoPcTransaction(txId ->{ + ... + }, + + Collections.singletonList( + RemotePrepareCommitPhaseOperations.createSerializable( + this::callPrepareEndpoint, + this::callValidateEndpoint, + this::callCommitEndpoint, + this::callRollbackEndpoint + ) + ) +); +``` + +In the `rollback` endpoint of the Customer Service, the endpoint resumes and rolls back the transaction. For reference, see [`CustomerService.java`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/customer-service/src/main/java/sample/customer/CustomerService.java). + +```java +customerRepository.rollbackTransactionOnParticipant(request.getTransactionId()); +``` + +For details about how to handle exceptions in ScalarDB, see [How to handle exceptions](../../two-phase-commit-transactions.mdx#how-to-handle-exceptions). diff --git a/versioned_docs/version-3.X/scalardb-samples/spring-data-microservice-transaction-sample/images/ERD.png b/versioned_docs/version-3.X/scalardb-samples/spring-data-microservice-transaction-sample/images/ERD.png new file mode 100644 index 00000000..c0468efa Binary files /dev/null and b/versioned_docs/version-3.X/scalardb-samples/spring-data-microservice-transaction-sample/images/ERD.png differ diff --git a/versioned_docs/version-3.X/scalardb-samples/spring-data-microservice-transaction-sample/images/overview.png b/versioned_docs/version-3.X/scalardb-samples/spring-data-microservice-transaction-sample/images/overview.png new file mode 100644 index 00000000..223c8ad7 Binary files /dev/null and b/versioned_docs/version-3.X/scalardb-samples/spring-data-microservice-transaction-sample/images/overview.png differ diff --git a/versioned_docs/version-3.X/scalardb-samples/spring-data-microservice-transaction-sample/images/seq-diagram-high-level-2pc-api.png b/versioned_docs/version-3.X/scalardb-samples/spring-data-microservice-transaction-sample/images/seq-diagram-high-level-2pc-api.png new file mode 100644 index 00000000..724e52b5 Binary files /dev/null and b/versioned_docs/version-3.X/scalardb-samples/spring-data-microservice-transaction-sample/images/seq-diagram-high-level-2pc-api.png differ diff --git a/versioned_docs/version-3.X/scalardb-samples/spring-data-microservice-transaction-sample/images/sequence_diagram.png b/versioned_docs/version-3.X/scalardb-samples/spring-data-microservice-transaction-sample/images/sequence_diagram.png new file mode 100644 index 00000000..0317b5f3 Binary files /dev/null and b/versioned_docs/version-3.X/scalardb-samples/spring-data-microservice-transaction-sample/images/sequence_diagram.png differ diff --git a/versioned_docs/version-3.X/scalardb-samples/spring-data-multi-storage-transaction-sample/README.mdx b/versioned_docs/version-3.X/scalardb-samples/spring-data-multi-storage-transaction-sample/README.mdx new file mode 100644 index 00000000..f5f29cac --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-samples/spring-data-multi-storage-transaction-sample/README.mdx @@ -0,0 +1,333 @@ +--- +tags: + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Sample application of Spring Data JDBC for ScalarDB with Multi-storage Transactions + +import WarningLicenseKeyContact from '/src/components/en-us/_warning-license-key-contact.mdx'; + +This tutorial describes how to create a sample Spring Boot application by using Spring Data JDBC for ScalarDB with Multi-storage Transactions. + +## Prerequisites for this sample application + +- OpenJDK LTS version (8, 11, 17, or 21) from [Eclipse Temurin](https://adoptium.net/temurin/releases/) +- [Docker](https://www.docker.com/get-started/) 20.10 or later with [Docker Compose](https://docs.docker.com/compose/install/) V2 or later + +:::note + +This sample application has been tested with OpenJDK from Eclipse Temurin. ScalarDB itself, however, has been tested with JDK distributions from various vendors. For details about the requirements for ScalarDB, including compatible JDK distributions, please see [Requirements](../../requirements.mdx). + +::: + + + +## Sample application + +### Overview + +This tutorial illustrates the process of creating a sample e-commerce application, where items can be ordered and paid for with a line of credit by using the [multi-storage transactions](../../multi-storage-transactions.mdx) feature in ScalarDB. + +:::note + +Application-specific error handling, authentication processing, etc. are omitted in the sample application since this tutorial focuses on explaining how to use Spring Data JDBC for ScalarDB with multi-storage transactions. + +For details, see [Guide of Spring Data JDBC for ScalarDB](../../scalardb-sql/spring-data-guide.mdx). + +::: + +![Overview](images/overview.png) + +The application accesses the databases through ScalarDB Cluster. + +### Schema + +[The schema](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-multi-storage-transaction-sample/schema.sql) is as follows: + +```sql +CREATE COORDINATOR TABLES IF NOT EXIST; + +CREATE NAMESPACE IF NOT EXISTS customer; + +CREATE TABLE IF NOT EXISTS customer.customers ( + customer_id INT PRIMARY KEY, + name TEXT, + credit_limit INT, + credit_total INT +); + +CREATE NAMESPACE IF NOT EXISTS "order"; + +CREATE TABLE IF NOT EXISTS "order".orders ( + customer_id INT, + timestamp BIGINT, + order_id TEXT, + PRIMARY KEY (customer_id, timestamp) +); + +CREATE INDEX IF NOT EXISTS ON "order".orders (order_id); + +CREATE TABLE IF NOT EXISTS "order".statements ( + order_id TEXT, + item_id INT, + count INT, + PRIMARY KEY (order_id, item_id) +); + +CREATE TABLE IF NOT EXISTS "order".items ( + item_id INT PRIMARY KEY, + name TEXT, + price INT +); +``` + +All the tables are created in the `customer` and `order` namespaces. + +- `customer.customers`: a table that manages customers' information + - `credit_limit`: the maximum amount of money a lender will allow each customer to spend when using a credit card + - `credit_total`: the amount of money that each customer has already spent by using the credit card +- `order.orders`: a table that manages order information +- `order.statements`: a table that manages order statement information +- `order.items`: a table that manages information of items to be ordered + +The Entity Relationship Diagram for the schema is as follows: + +![ERD](images/ERD.png) + +### Transactions + +The following five transactions are implemented in this sample application: + +1. Getting customer information +2. Placing an order by credit card (checks if the cost of the order is below the credit limit, then records order history and updates the `credit_total` if the check passes) +3. Getting order information by order ID +4. Getting order information by customer ID +5. Repayment (reduces the amount in the `credit_total`) + +## Configuration for ScalarDB Cluster + +[The configuration for ScalarDB Cluster](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-multi-storage-transaction-sample/scalardb-cluster-node.properties) are as follows: + +```properties +scalar.db.storage=multi-storage +scalar.db.multi_storage.storages=cassandra,mysql +scalar.db.multi_storage.storages.cassandra.storage=cassandra +scalar.db.multi_storage.storages.cassandra.contact_points=cassandra-1 +scalar.db.multi_storage.storages.cassandra.username=cassandra +scalar.db.multi_storage.storages.cassandra.password=cassandra +scalar.db.multi_storage.storages.mysql.storage=jdbc +scalar.db.multi_storage.storages.mysql.contact_points=jdbc:mysql://mysql-1:3306/ +scalar.db.multi_storage.storages.mysql.username=root +scalar.db.multi_storage.storages.mysql.password=mysql +scalar.db.multi_storage.namespace_mapping=customer:mysql,order:cassandra,coordinator:cassandra +scalar.db.multi_storage.default_storage=cassandra + +scalar.db.cluster.node.standalone_mode.enabled=true +scalar.db.sql.enabled=true + +# License key configurations +scalar.db.cluster.node.licensing.license_key= +scalar.db.cluster.node.licensing.license_check_cert_pem= +``` + +- `scalar.db.storage`: Specifying `multi-storage` is necessary to use Multi-storage Transactions in ScalarDB. +- `scalar.db.multi_storage.storages`: Your storage names must be defined here. +- `scalar.db.multi_storage.storages.cassandra.*`: These configurations are for the `cassandra` storage, which is one of the storage names defined in `scalar.db.multi_storage.storages`. You can configure all the `scalar.db.*` properties for the `cassandra` storage here. +- `scalar.db.multi_storage.storages.mysql.*`: These configurations are for the `mysql` storage, which is one of the storage names defined in `scalar.db.multi_storage.storages`. You can configure all the `scalar.db.*` properties for the `mysql` storage here. +- `scalar.db.multi_storage.namespace_mapping`: This configuration maps the namespaces to the storage. In this sample application, operations for `customer` namespace tables are mapped to the `mysql` storage and operations for `order` namespace tables are mapped to the `cassandra` storage. You can also define which storage is mapped for the `coordinator` namespace that is used in Consensus Commit transactions. +- `scalar.db.multi_storage.default_storage`: This configuration sets the default storage that is used for operations on unmapped namespace tables. + +For details, see [Multi-Storage Transactions](../../multi-storage-transactions.mdx). + +In this sample application, ScalarDB Cluster is running in standalone mode (`scalar.db.cluster.node.standalone_mode.enabled=true`). + +Also, you need to set the license key (trial license or commercial license) for ScalarDB Cluster in the configuration file. For details, see [How to Configure a Product License Key](../../scalar-licensing/index.mdx). + +## Client Configuration + +[The client configuration](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-multi-storage-transaction-sample/scalardb-sql.properties) is as follows: + +```properties +scalar.db.sql.connection_mode=cluster +scalar.db.sql.cluster_mode.contact_points=indirect:localhost +``` + +## Setup + +### Clone the ScalarDB samples repository + +Open Terminal, then clone the ScalarDB samples repository by running the following command: + +```console +git clone https://github.com/scalar-labs/scalardb-samples +``` + +Then, go to the directory with this sample by running the following command: + +```console +cd scalardb-samples/spring-data-multi-storage-transaction-sample +``` + +### Set the license key + +Set the license key (trial license or commercial license) for the ScalarDB Clusters in the configuration file [`scalardb-cluster-node.properties`](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-multi-storage-transaction-sample/scalardb-cluster-node.properties). For details, see [How to Configure a Product License Key](../../scalar-licensing/index.mdx). + +### Start Cassandra, MySQL, and ScalarDB Cluster + +To start Cassandra, MySQL, and ScalarDB Cluster, you need to run the following `docker-compose` command: + +```console +docker-compose up -d +``` + +Please note that starting the containers may take more than one minute. + +### Load schema + +You then need to apply the schema with the following command. +To download the SQL CLI tool, `scalardb-cluster-sql-cli--all.jar`, see the [Releases](https://github.com/scalar-labs/scalardb/releases) of ScalarDB and download the version that you want to use. + +```console +java -jar scalardb-cluster-sql-cli--all.jar --config scalardb-sql.properties --file schema.sql +``` + +### Load initial data + +After the containers have started, you need to load the initial data by running the following command: + +```console +./gradlew run --args="LoadInitialData" +``` + +After the initial data has loaded, the following records should be stored in the tables: + +- For the `customer.customers` table: + +| customer_id | name | credit_limit | credit_total | +|-------------|---------------|--------------|--------------| +| 1 | Yamada Taro | 10000 | 0 | +| 2 | Yamada Hanako | 10000 | 0 | +| 3 | Suzuki Ichiro | 10000 | 0 | + +- For the `order.items` table: + +| item_id | name | price | +|---------|--------|-------| +| 1 | Apple | 1000 | +| 2 | Orange | 2000 | +| 3 | Grape | 2500 | +| 4 | Mango | 5000 | +| 5 | Melon | 3000 | + +## Run the sample application + +Let's start with getting information about the customer whose ID is `1`: + +```console +./gradlew run --args="GetCustomerInfo 1" +... +{"customer_id":1,"name":"Yamada Taro","credit_limit":10000,"credit_total":0} +... +``` + +Then, place an order for three apples and two oranges by using customer ID `1`. Note that the order format is `:,:,...`: + +```console +./gradlew run --args="PlaceOrder 1 1:3,2:2" +... +{"order_id":"5d49eb62-fcb9-4dd2-9ae5-e714d989937f","customer_id":1,"timestamp":1677564659810} +... +``` + +You can see that running this command shows the order ID. + +Let's check the details of the order by using the order ID: + +```console +./gradlew run --args="GetOrder 5d49eb62-fcb9-4dd2-9ae5-e714d989937f" +... +{"order_id":"5d49eb62-fcb9-4dd2-9ae5-e714d989937f","timestamp":1677564659810,"customer_id":1,"customer_name":"Yamada Taro","statements":[{"item_id":1,"item_name":"Apple","price":1000,"count":3,"total":3000},{"item_id":2,"item_name":"Orange","price":2000,"count":2,"total":4000}],"total":7000} +... +``` + +Then, let's place another order and get the order history of customer ID `1`: + +```console +./gradlew run --args="PlaceOrder 1 5:1" +... +{"order_id":"ccd97d75-ee57-4393-a0bb-5230c4a8c68a","customer_id":1,"timestamp":1677564776069} +... +./gradlew run --args="GetOrders 1" +... +[{"order_id":"ccd97d75-ee57-4393-a0bb-5230c4a8c68a","timestamp":1677564776069,"customer_id":1,"customer_name":"Yamada Taro","statements":[{"item_id":5,"item_name":"Melon","price":3000,"count":1,"total":3000}],"total":3000},{"order_id":"5d49eb62-fcb9-4dd2-9ae5-e714d989937f","timestamp":1677564659810,"customer_id":1,"customer_name":"Yamada Taro","statements":[{"item_id":1,"item_name":"Apple","price":1000,"count":3,"total":3000},{"item_id":2,"item_name":"Orange","price":2000,"count":2,"total":4000}],"total":7000}] +... +``` + +This order history is shown in descending order by timestamp. + +The customer's current `credit_total` is `10000`. Since the customer has now reached their `credit_limit`, which was shown when retrieving their information, they cannot place anymore orders. + +```console +./gradlew run --args="GetCustomerInfo 1" +... +{"customer_id":1,"name":"Yamada Taro","credit_limit":10000,"credit_total":10000} +... +./gradlew run --args="PlaceOrder 1 3:1,4:1" +... +java.lang.RuntimeException: Credit limit exceeded. limit:10000, total:17500 + at sample.SampleService.placeOrder(SampleService.java:102) + at sample.SampleService$$FastClassBySpringCGLIB$$1123c447.invoke() + at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) + at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:793) + at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) + at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) + at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123) + at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388) + at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) + at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) + at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:763) + at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:708) + at sample.SampleService$$EnhancerBySpringCGLIB$$1cb0cc8c.placeOrder() + at sample.command.PlaceOrderCommand.call(PlaceOrderCommand.java:37) + at sample.command.PlaceOrderCommand.call(PlaceOrderCommand.java:13) + at picocli.CommandLine.executeUserObject(CommandLine.java:2041) + at picocli.CommandLine.access$1500(CommandLine.java:148) + at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461) + at picocli.CommandLine$RunLast.handle(CommandLine.java:2453) + at picocli.CommandLine$RunLast.handle(CommandLine.java:2415) + at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273) + at picocli.CommandLine$RunLast.execute(CommandLine.java:2417) + at picocli.CommandLine.execute(CommandLine.java:2170) + at sample.SampleApp.run(SampleApp.java:26) + at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:768) + at org.springframework.boot.SpringApplication.callRunners(SpringApplication.java:752) + at org.springframework.boot.SpringApplication.run(SpringApplication.java:314) + at org.springframework.boot.SpringApplication.run(SpringApplication.java:1303) + at org.springframework.boot.SpringApplication.run(SpringApplication.java:1292) + at sample.SampleApp.main(SampleApp.java:35) +... +``` + +After making a payment, the customer will be able to place orders again. + +```console +./gradlew run --args="Repayment 1 8000" +... +./gradlew run --args="GetCustomerInfo 1" +... +{"customer_id":1,"name":"Yamada Taro","credit_limit":10000,"credit_total":2000} +... +./gradlew run --args="PlaceOrder 1 3:1,4:1" +... +{"order_id":"3ac4a1bf-a724-4f26-b948-9f03281a971e","customer_id":1,"timestamp":1677565028204} +... +``` + +## Cleanup + +To stop Cassandra, MySQL, and ScalarDB Cluster, run the following command: + +```console +docker-compose down +``` diff --git a/versioned_docs/version-3.X/scalardb-samples/spring-data-multi-storage-transaction-sample/images/ERD.png b/versioned_docs/version-3.X/scalardb-samples/spring-data-multi-storage-transaction-sample/images/ERD.png new file mode 100644 index 00000000..02100437 Binary files /dev/null and b/versioned_docs/version-3.X/scalardb-samples/spring-data-multi-storage-transaction-sample/images/ERD.png differ diff --git a/versioned_docs/version-3.X/scalardb-samples/spring-data-multi-storage-transaction-sample/images/overview.png b/versioned_docs/version-3.X/scalardb-samples/spring-data-multi-storage-transaction-sample/images/overview.png new file mode 100644 index 00000000..360b26c6 Binary files /dev/null and b/versioned_docs/version-3.X/scalardb-samples/spring-data-multi-storage-transaction-sample/images/overview.png differ diff --git a/versioned_docs/version-3.X/scalardb-schema-loader-status-codes.mdx b/versioned_docs/version-3.X/scalardb-schema-loader-status-codes.mdx new file mode 100644 index 00000000..23c452e1 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-schema-loader-status-codes.mdx @@ -0,0 +1,165 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# ScalarDB Schema Loader Error Codes + +This page provides a list of error codes in ScalarDB Schema Loader. + +## Error code classes and descriptions + +| Class | Description | +|:-------------------------|:---------------------------------------| +| `DB-SCHEMA-LOADER-1xxxx` | Errors for the user error category. | + +## `DB-SCHEMA-LOADER-1xxxx` status codes + +The following are status codes and messages for the user error category. + +### `DB-SCHEMA-LOADER-10000` + +**Message** + +```markdown +The table does not exist. Table: %s +``` + +### `DB-SCHEMA-LOADER-10001` + +**Message** + +```markdown +The partition keys for the table %s.%s were modified, but altering partition keys is not supported +``` + +### `DB-SCHEMA-LOADER-10002` + +**Message** + +```markdown +The clustering keys for the table %s.%s were modified, but altering clustering keys is not supported +``` + +### `DB-SCHEMA-LOADER-10003` + +**Message** + +```markdown +The clustering order of the table %s.%s were modified, but altering the clustering order is not supported +``` + +### `DB-SCHEMA-LOADER-10004` + +**Message** + +```markdown +The column %s in the table %s.%s has been deleted. Column deletion is not supported when altering a table +``` + +### `DB-SCHEMA-LOADER-10005` + +**Message** + +```markdown +The data type for the column %s in the table %s.%s was modified, but altering data types is not supported +``` + +### `DB-SCHEMA-LOADER-10006` + +**Message** + +```markdown +Specifying the '--schema-file' option is required when using the '--repair-all' option +``` + +### `DB-SCHEMA-LOADER-10007` + +**Message** + +```markdown +Specifying the '--schema-file' option is required when using the '--alter' option +``` + +### `DB-SCHEMA-LOADER-10008` + +**Message** + +```markdown +Specifying the '--schema-file' option is required when using the '--import' option +``` + +### `DB-SCHEMA-LOADER-10009` + +**Message** + +```markdown +Specifying the '--coordinator' option with the '--import' option is not allowed. Create Coordinator tables separately +``` + +### `DB-SCHEMA-LOADER-10010` + +**Message** + +```markdown +Reading the configuration file failed. File: %s +``` + +### `DB-SCHEMA-LOADER-10011` + +**Message** + +```markdown +Reading the schema file failed. File: %s +``` + +### `DB-SCHEMA-LOADER-10012` + +**Message** + +```markdown +Parsing the schema JSON failed. Details: %s +``` + +### `DB-SCHEMA-LOADER-10013` + +**Message** + +```markdown +The table name must contain the namespace and the table. Table: %s +``` + +### `DB-SCHEMA-LOADER-10014` + +**Message** + +```markdown +The partition key must be specified. Table: %s +``` + +### `DB-SCHEMA-LOADER-10015` + +**Message** + +```markdown +Invalid clustering-key format. The clustering key must be in the format of 'column_name' or 'column_name ASC/DESC'. Table: %s; Clustering key: %s +``` + +### `DB-SCHEMA-LOADER-10016` + +**Message** + +```markdown +Columns must be specified. Table: %s +``` + +### `DB-SCHEMA-LOADER-10017` + +**Message** + +```markdown +Invalid column type. Table: %s; Column: %s; Type: %s +``` diff --git a/versioned_docs/version-3.X/scalardb-sql/grammar.mdx b/versioned_docs/version-3.X/scalardb-sql/grammar.mdx new file mode 100644 index 00000000..b0a19b4d --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-sql/grammar.mdx @@ -0,0 +1,2499 @@ +--- +tags: + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# ScalarDB SQL Grammar + +- DDL + - [CREATE NAMESPACE](#create-namespace) + - [CREATE TABLE](#create-table) + - [CREATE INDEX](#create-index) + - [TRUNCATE TABLE](#truncate-table) + - [DROP INDEX](#drop-index) + - [DROP TABLE](#drop-table) + - [DROP NAMESPACE](#drop-namespace) + - [CREATE COORDINATOR TABLES](#create-coordinator-tables) + - [TRUNCATE COORDINATOR TABLES](#truncate-coordinator-tables) + - [DROP COORDINATOR TABLES](#drop-coordinator-tables) + - [ALTER TABLE](#alter-table) +- DML + - [SELECT](#select) + - [INSERT](#insert) + - [UPSERT](#upsert) + - [UPDATE](#update) + - [DELETE](#delete) +- DCL + - [CREATE USER](#create-user) + - [ALTER USER](#alter-user) + - [DROP USER](#drop-user) + - [GRANT](#grant) + - [REVOKE](#revoke) +- Others + - [USE](#use) + - [BEGIN](#begin) + - [START TRANSACTION](#start-transaction) + - [JOIN](#join) + - [PREPARE](#prepare) + - [VALIDATE](#validate) + - [COMMIT](#commit) + - [ROLLBACK](#rollback) + - [ABORT](#abort) + - [SET MODE](#set-mode) + - [SHOW NAMESPACES](#show-namespaces) + - [SHOW TABLES](#show-tables) + - [DESCRIBE](#describe) + - [SUSPEND](#suspend) + - [RESUME](#resume) + - [SHOW USERS](#show-users) + - [SHOW GRANTS](#show-grants) +- Literal + - [Text](#text) + - [Numeric](#numeric) + - [Date and time](#date-and-time) + +## DDL + +### CREATE NAMESPACE + +Before creating tables, namespaces must be created since a table belongs to one namespace. +The `CREATE NAMESPACE` command creates a namespace. + +#### Grammar + +```sql +CREATE NAMESPACE [IF NOT EXISTS] [WITH creation_options] + +creation_options:
( + data_type PRIMARY KEY, + data_type, + ... +) [WITH creation_options] + +data_type: BOOLEAN | INT | BIGINT | FLOAT | DOUBLE | TEXT | BLOB | DATE | TIME | TIMESTAMP | TIMESTAMPTZ +creation_options:
( + data_type, + data_type, + ..., + data_type [ENCRYPTED], + ..., + PRIMARY KEY (, [, ] ...) +) [WITH [clustering_order_definition [AND creation_options]] | creation_options] + +clustering_order_definition: CLUSTERING ORDER BY ( [clustering_order] [, [clustering_order]] ...) +clustering_order: ASC | DESC +``` + +- If you omit `clustering_order`, the default clustering order `ASC` will be used. + +Create a table with a primary key that is composed of multiple partition key columns and multiple clustering key columns: +```sql +CREATE TABLE [IF NOT EXISTS] [.]
( + data_type, + ..., + data_type, + ..., + data_type [ENCRYPTED], + data_type [ENCRYPTED], + ..., + PRIMARY KEY (( [, ] ...), [, ] ...) +) [WITH [clustering_order_definition [AND creation_options]] | creation_options] +``` + +#### Examples + +Examples of `CREATE TABLE` are as follows: + +```sql +-- Create a table with a primary key ("c1") and creation options +CREATE TABLE ns.tbl ( + c1 INT PRIMARY KEY, + c2 TEXT, + c3 FLOAT, + c4 BIGINT, + c5 BOOLEAN +) WITH 'option1' = 'value1' AND 'option2' = 'value2' AND 'option3' = 'value3'; + +-- Create a table with a partition key ("c1") and a clustering key ("c2" and "c3") with clustering order definition only if it does not already exist +CREATE TABLE IF NOT EXISTS tbl ( + c1 INT, + c2 TEXT, + c3 FLOAT, + c4 BIGINT, + c5 BOOLEAN, + PRIMARY KEY (c1, c2, c3) +) WITH CLUSTERING ORDER BY (c2 DESC, c3 ASC); + +-- Create a table with a partition key ("c1", "c2") and a clustering key ("c3" and "c4") with clustering order definition and creation options +CREATE TABLE ns.tbl ( + c1 INT, + c2 TEXT, + c3 FLOAT, + c4 BIGINT, + c5 BOOLEAN, + PRIMARY KEY ((c1, c2), c3, c4) +) WITH CLUSTERING ORDER BY (c3 DESC, c4 ASC) AND 'option1' = 'value1' AND 'option2' = 'value2' AND 'option3' = 'value3'; + +-- Create a table with a primary key ("c1") and encrypted columns ("c2", "c3", "c4", and "c5") +CREATE TABLE ns.tbl ( + c1 INT PRIMARY KEY, + c2 TEXT ENCRYPTED, + c3 FLOAT ENCRYPTED, + c4 BIGINT ENCRYPTED, + c5 BOOLEAN ENCRYPTED +); +``` + +Examples of building statement objects for `CREATE TABLE` are as follows: + +```java +// Create a table with a primary key ("c1") and creation options +CreateTableStatement statement1 = + StatementBuilder.createTable("ns", "tbl") + .withPartitionKey("c1", DataType.INT) + .withColumn("c2", DataType.TEXT) + .withColumn("c3", DataType.FLOAT) + .withColumn("c4", DataType.BIGINT) + .withColumn("c5", DataType.BOOLEAN) + .withOption("option1", "value1") + .withOption("option2", "value2") + .withOption("option3", "value3") + .build(); + +// Create a table with a partition key ("c1") and a clustering key ("c2" and "c3") with clustering order definition +CreateTableStatement statement2 = + StatementBuilder.createTable("tbl") + .ifNotExists() + .withPartitionKey("c1", DataType.INT) + .withClusteringKey("c2", DataType.TEXT) + .withClusteringKey("c3", DataType.FLOAT) + .withColumn("c4", DataType.BIGINT) + .withColumn("c5", DataType.BOOLEAN) + .withClusteringOrder("c2", ClusteringOrder.DESC) + .withClusteringOrder("c3", ClusteringOrder.ASC) + .build(); + +// Create a table with a partition key ("c1", "c2") and a clustering key ("c3" and "c4") with clustering order definition and creation options +CreateTableStatement statement3 = + StatementBuilder.createTable("ns", "tbl") + .withPartitionKey("c1", DataType.INT) + .withPartitionKey("c2", DataType.TEXT) + .withClusteringKey("c3", DataType.FLOAT) + .withClusteringKey("c4", DataType.BIGINT) + .withColumn("c5", DataType.BOOLEAN) + .withClusteringOrder("c3", ClusteringOrder.DESC) + .withClusteringOrder("c4", ClusteringOrder.ASC) + .withOption("option1", "value1") + .withOption("option2", "value2") + .withOption("option3", "value3") + .build(); + +// Create a table with a primary key ("c1") and encrypted columns ("c2", "c3", "c4", and "c5") +CreateTableStatement statement4 = + StatementBuilder.createTable("ns", "tbl") + .withPartitionKey("c1", DataType.INT) + .withColumn("c2", DataType.TEXT, true) + .withColumn("c3", DataType.FLOAT, true) + .withColumn("c4", DataType.BIGINT, true) + .withColumn("c5", DataType.BOOLEAN, true) + .build(); +``` + +### CREATE INDEX + +The `CREATE INDEX` command creates a secondary index on a table. + +:::note + +You cannot create a secondary index on encrypted columns. + +::: + +#### Grammar + +```sql +CREATE INDEX [IF NOT EXISTS] ON [.]
() +``` + +#### Examples + +Examples of `CREATE INDEX` are as follows: + +```sql +-- Create a secondary index on a column "c4" of a table "ns.tbl" +CREATE INDEX ON ns.tbl (c4); + +-- Create a secondary index only if it does not already exist +CREATE INDEX IF NOT EXISTS ON tbl (c4); + +-- Create a secondary index with options +CREATE INDEX ON ns.tbl (c4) WITH 'option1' = 'value1' AND 'option2' = 'value2' AND 'option3' = 'value3'; +``` + +Examples of building statement objects for `CREATE INDEX` are as follows: + +```java +// Create a secondary index on a column "c4" of a table "ns.tbl" +CreateIndexStatement statement1 = + StatementBuilder.createIndex().onTable("ns", "tbl").column("c4").build(); + +// Create a secondary index only if it does not already exist +CreateIndexStatement statement2 = + StatementBuilder.createIndex().ifNotExists().onTable("tbl").column("c4").build(); + +// Create a secondary index with options +CreateIndexStatement statement3 = + StatementBuilder.createIndex() + .onTable("ns", "tbl") + .column("c4") + .withOption("option1", "value1") + .withOption("option2", "value2") + .withOption("option3", "value3") + .build(); +``` + +### TRUNCATE TABLE + +The `TRUNCATE TABLE` command truncates a table. + +#### Grammar + +```sql +TRUNCATE TABLE [.]
+``` + +#### Examples + +Examples of `TRUNCATE TABLE` are as follows: + +```sql +-- Truncate a table "ns.tbl" +TRUNCATE TABLE ns.tbl; +``` + +Examples of building statement objects for `TRUNCATE TABLE` are as follows: + +```java +// Truncate a table "ns.tbl" +TruncateTableStatement statement = StatementBuilder.truncateTable("ns", "tbl").build(); +``` + +### DROP INDEX + +The `DROP INDEX` command drops a secondary index. + +#### Grammar + +```sql +DROP INDEX [IF EXISTS] ON [.]
() +``` + +#### Examples + +Examples of `DROP INDEX` are as follows: + +```sql +-- Drop a secondary index on a column "c4" of a table "ns.tbl" +DROP INDEX ON ns.tbl (c4); + +-- Drop a secondary index only if it exists +DROP INDEX IF EXISTS ON tbl (c4); +``` + +Examples of building statement objects for `DROP INDEX` are as follows: + +```java +// Drop a secondary index on a column "c4" of a table "ns.tbl" +DropIndexStatement statement1 = + StatementBuilder.dropIndex().onTable("ns", "tbl").column("c4").build(); + +// Drop a secondary index only if it exists +DropIndexStatement statement2 = + StatementBuilder.dropIndex().ifExists().onTable("ns", "tbl").column("c4").build(); +``` + +### DROP TABLE + +The `DROP TABLE` command drops a table. + +#### Grammar + +```sql +DROP TABLE [IF EXISTS] [.]
+``` + +#### Examples + +Examples of `DROP TABLE` are as follows: + +```sql +-- Drop a table "ns.tbl" +DROP TABLE ns.tbl; + +-- Drop a table only if it exists +DROP TABLE IF EXISTS tbl; +``` + +Examples of building statement objects for `DROP TABLE` are as follows: + +```java +// Drop a table "ns.tbl" +DropTableStatement statement1 = StatementBuilder.dropTable("ns", "tbl").build(); + +// Drop a table only if it exists +DropTableStatement statement2 = StatementBuilder.dropTable("ns", "tbl").ifExists().build(); +``` + +### DROP NAMESPACE + +The `DROP NAMESPACE` command drops a namespace. + +#### Grammar + +```sql +DROP NAMESPACE [IF EXISTS] [CASCADE] +``` + +#### Examples + +Examples of `DROP NAMESPACE` are as follows: + +```sql +-- Drop a namespace "ns" +DROP NAMESPACE ns; + +-- Drop a namespace only if it exists +DROP NAMESPACE IF EXISTS ns; + +-- Drop a namespace with cascade +DROP NAMESPACE ns CASCADE; +``` + +Examples of building statement objects for `DROP NAMESPACE` are as follows: + +```java +// Drop a namespace "ns" +DropNamespaceStatement statement1 = StatementBuilder.dropNamespace("ns").build(); + +// Drop a namespace only if it exists +DropNamespaceStatement statement2 = StatementBuilder.dropNamespace("ns").ifExists().build(); + +// Drop a namespace with cascade +DropNamespaceStatement statement3 = StatementBuilder.dropNamespace("ns").cascade().build(); +``` + +### CREATE COORDINATOR TABLES + +The `CREATE COORDINATOR TABLES` command creates coordinator tables. + +#### Grammar + +```sql +CREATE COORDINATOR TABLES [IF NOT {EXIST|EXISTS}] [WITH creation_options] + +creation_options:
ADD [COLUMN] data_type [ENCRYPTED] + +data_type: BOOLEAN | INT | BIGINT | FLOAT | DOUBLE | TEXT | BLOB | DATE | TIME | TIMESTAMP | TIMESTAMPTZ +``` + +- You can specify `ENCRYPTED` for the column to encrypt the data in that column. + +#### Examples + +Examples of `ALTER TABLE` are as follows: + +```sql +-- Add a new column "new_col" to "ns.tbl" +ALTER TABLE ns.tbl ADD COLUMN new_col INT; + +-- Add a new encrypted column "new_col" to "ns.tbl" +ALTER TABLE ns.tbl ADD COLUMN new_col TEXT ENCRYPTED; +``` + +Examples of building statement objects for `ALTER TABLE` are as follows: + +```java +// Add a new column "new_col" to "ns.tbl" +AlterTableAddColumnStatement statement1 = + StatementBuilder.alterTable("ns", "tbl").addColumn("new_col", DataType.INT).build(); + +// Add a new encrypted column "new_col" to "ns.tbl" +AlterTableAddColumnStatement statement2 = + StatementBuilder.alterTable("ns", "tbl").addColumn("new_col", DataType.TEXT, true).build(); +``` + +## DML + +### SELECT + +The `SELECT` command retrieves records from the database. ScalarDB SQL creates an execution plan for a `SELECT` command by using one of the `Get`, partition `Scan`, and cross-partition `Scan` operations in ScalarDB to retrieve records from the database. For the best results, specify primary key columns uniquely as much as possible in the `WHERE` clause to avoid using the cross-partition scan since it might cause performance and consistency issues, especially in non-JDBC databases. See the following rules for selecting operations. The former ones are prior to the latter ones and more efficient. [ScalarDB data model](../data-modeling.mdx) also helps in understanding the best practices for modeling and accessing data. + +1. If you fully specify the primary-key columns in the `WHERE` clause, the `SELECT` command will use a `Get` operation for a single record in a single partition. +2. If you fully specify the partition key and properly specify the clustering key and order in the `WHERE` and `ORDER BY` clauses, the `SELECT` command will use a `Scan` operation for records in a single partition. For more details, see the [Examples of partition scans and index scans](#examples-of-partition-scans-and-index-scans). +3. If you specify the indexed column value literal with the `equal to` (`=`) operator in the `WHERE` clause without the `ORDER BY` clause, the `SELECT` command will use an index `Scan` operation. +4. For other cases, the `SELECT` command will be converted to a cross-partition `Scan` operation. + +You need to enable the cross-partition scan option and the cross-partition scan with filtering and ordering options if you want to flexibly retrieve records across partitions with arbitrary conditions and orderings. Currently, the ordering option is available only for JDBC databases. For details about configurations, see [Cross-partition scan configurations](../configurations.mdx#cross-partition-scan-configurations) and [ScalarDB Cluster SQL configurations](../scalardb-cluster/developer-guide-for-scalardb-cluster-with-java-api.mdx#scalardb-cluster-sql-client-configurations). + +:::warning + +For non-JDBC databases, transactions could be executed at read-committed snapshot isolation (`SNAPSHOT`), which is a lower isolation level, even if you enable cross-partition scan with the `SERIALIZABLE` isolation level. When using non-JDBC databases, use cross-partition scan only if consistency does not matter for your transactions. + +::: + +#### Grammar + +```sql +SELECT projection [, projection] ... + FROM [.]
[AS ] [join_specification [join_specification] ...] + [WHERE and_predicates [OR and_predicates ...] | or_predicates [AND or_predicates ...]] + [ORDER BY identifier [order] [, identifier [order]] ...] + [LIMIT ] + [WITH operation_attributes] + +projection: * | identifier [AS ] +join_specification: [INNER] JOIN [.]
[AS ] ON join_predicate [AND join_predicate] ... | {LEFT|RIGHT} [OUTER] JOIN [.]
[AS ] ON join_predicate [AND join_predicate] ... +join_predicate: identifier = identifier +and_predicates: predicate | (predicate [AND predicate ...]) +or_predicates: predicate | (predicate [OR predicate ...]) +predicate: identifier operator | identifier BETWEEN AND | identifier [NOT] LIKE [ESCAPE ] | identifier IS [NOT] NULL +identifier: [[.]
.] | [alias.] +operator: = | <> | != | > | >= | < | <= +order: ASC | DESC +operation_attributes: operation_attribute [AND operation_attribute] ... +operation_attribute: = +``` + +##### Note + +`JOIN` clause: + +- For `[INNER] JOIN` and `LEFT [OUTER] JOIN`: + - The `join_predicate`s must include either all primary-key columns or a secondary-index column from the right table. + - The `WHERE` predicates and the `ORDER BY` clause can only include columns from the table specified in the `FROM` clause. +- For `RIGHT [OUTER] JOIN`: + - It must be specified as the first `join_specification`. + - The `join_predicate`s must contain all primary-key columns or a secondary-index column from the left table. + - The `WHERE` predicates and the `ORDER BY` clause can only specify columns from the table specified in the `RIGHT OUTER JOIN` clause. + +`WHERE` clauses: + +- You can use arbitrary predicates for any columns in the `WHERE` clause. +- In the `WHERE` clause, predicates must be an OR-wise of `and_predicates` (known as disjunctive normal form) or an AND-wise of `or_predicates` (known as conjunctive normal form). +- When connecting multiple `and_predicates` or `or_predicates`, which have more than one predicate, you need to put parentheses around `and_predicates` and `or_predicates`. +- You can specify `` to a bind marker (positional `?` and named `:`). See the [Literal](#literal) section for the literal syntax. +- You cannot specify encrypted columns in the `WHERE` clause. + +`LIKE` predicate: + +- `_` in `` matches any single character. +- `%` in `` matches any sequence of zero or more characters. +- `\` in `` works as the escape character by default. + - You can change the escape character by specifying the `ESCAPE` clause. + - You can disable the escape character by specifying an empty escape character, `ESCAPE ''`. + +`ORDER BY` clause: + +- You can specify `order` for any columns in the `ORDER BY` clause. +- If you omit `order`, the default order `ASC` will be used. +- You cannot specify encrypted columns in the `ORDER BY` clause. + +`LIMIT` clause: + +- You can specify `` to a bind marker (positional `?` and named `:`). + +For details about retrieving data from a database in ScalarDB, see [Get operation](../api-guide.mdx#get-operation) and [Scan operation](../api-guide.mdx#scan-operation). + +#### Examples of partition scans and index scans + +Let's say you have the following table and index: +```sql +CREATE TABLE ns.tbl ( + c1 INT, + c2 TEXT, + c3 FLOAT, + c4 BIGINT, + c5 BOOLEAN, + PRIMARY KEY (c1, c2, c3) +) WITH CLUSTERING ORDER BY (c2 DESC, c3 ASC); + +CREATE INDEX ON ns.tbl (c4); +``` + +Examples of `SELECT` are as follows: + +```sql +-- With a full primary key +SELECT * FROM ns.tbl WHERE c1 = 10 AND c2 = 'aaa' AND c3 = 1.23; + +-- With a full primary key and predicates for non-primary-key columns +SELECT * FROM ns.tbl WHERE c1 = 10 AND c2 = 'aaa' AND c3 = 1.23 AND c4 < 100; + +-- With a partial primary key +SELECT * FROM ns.tbl WHERE c1 = 10 AND c2 = 'aaa'; + +-- With a partial primary key and predicates for non-primary-key columns +SELECT * FROM ns.tbl WHERE c1 = 10 AND c2 = 'aaa' AND (c4 < 100 OR c4 > 500); + +-- With projections and a partition key and clustering-key boundaries +SELECT c1, c2, c3, c5 FROM ns.tbl WHERE c1 = 10 AND c2 = 'aaa' AND c3 >= 1.23 AND c3 < 4.56; + +-- With projections and a partition key and clustering-key boundaries and orders and limit +SELECT c1 AS a, c2 AS b, c3 AS c, c5 FROM ns.tbl WHERE c1 = 10 AND c2 > 'aaa' AND c2 <= 'ddd' ORDER BY c2 ASC, c3 DESC LIMIT 10; + +-- With an equality predicate for an indexed column +SELECT * FROM ns.tbl WHERE c4 = 100; + +-- With an equality predicate for an indexed column and predicates for non-primary-key columns +SELECT * FROM ns.tbl WHERE c4 = 100 AND c5 = false; + +-- With projections and an indexed column and limit +SELECT c1, c2, c3, c4 FROM ns.tbl WHERE c4 = 100 LIMIT 10; + +-- With positional bind markers +SELECT * FROM ns.tbl WHERE c1 = ? AND c2 > ? AND c2 <= ? ORDER BY c2 ASC, c3 DESC LIMIT ?; + +-- With operations attributes +SELECT * FROM ns.tbl WHERE c1 = 10 AND c2 = 'aaa' AND c3 = 1.23 WITH 'attribute1' = 'value1' AND 'attribute2' = 'value2'; +``` + +Examples of building statement objects for `SELECT` are as follows: + +```java +// With a full primary key +SelectStatement statement1 = + StatementBuilder.select() + .from("ns", "tbl") + .where(Predicate.column("c1").isEqualTo(Value.of(10))) + .and(Predicate.column("c2").isEqualTo(Value.of("aaa"))) + .and(Predicate.column("c3").isEqualTo(Value.of(1.23F))) + .and(Predicate.column("c4").isLessThan(Value.of(100))) + .build(); + +// With a full primary key and predicates for non-primary-key columns +SelectStatement statement1 = + StatementBuilder.select() + .from("ns", "tbl") + .where(Predicate.column("c1").isEqualTo(Value.of(10))) + .and(Predicate.column("c2").isEqualTo(Value.of("aaa"))) + .and(Predicate.column("c4").isEqualTo(Value.of(1.23F))) + .and(Predicate.column("c4").isLessThan(Value.of(100))) + .build(); + +// With a partial primary key +SelectStatement statement2 = + StatementBuilder.select() + .from("ns", "tbl") + .where(Predicate.column("c1").isEqualTo(Value.of(10))) + .and(Predicate.column("c2").isEqualTo(Value.of("aaa"))) + .build(); + +// With a partial primary key and predicates for non-primary-key columns +SelectStatement statement2 = + StatementBuilder.select() + .from("ns", "tbl") + .where(Predicate.column("c1").isEqualTo(Value.of(10))) + .and(Predicate.column("c2").isEqualTo(Value.of("aaa"))) + .and( + AndPredicateList.predicate(Predicate.column("c4").isLessThan(Value.of(100))) + .and(Predicate.column("c4").isGreaterThan(Value.of(500))) + .build()) + .build(); + +// With projections and a partition key and clustering-key boundaries +SelectStatement statement3 = + StatementBuilder.select("c1", "c2", "c3", "c5") + .from("ns", "tbl") + .where(Predicate.column("c1").isEqualTo(Value.of(10))) + .and(Predicate.column("c2").isEqualTo(Value.of("aaa"))) + .and(Predicate.column("c3").isGreaterThanOrEqualTo(Value.of(1.23F))) + .and(Predicate.column("c3").isLessThan(Value.of(4.56F))) + .build(); + +// With projections and a partition key and clustering key boundaries and orders and limit +SelectStatement statement4 = + StatementBuilder.select( + Projection.column("c1").as("a"), + Projection.column("c2").as("b"), + Projection.column("c3").as("c"), + Projection.column("c5").as("d")) + .from("ns", "tbl") + .where(Predicate.column("c1").isEqualTo(Value.of(10))) + .and(Predicate.column("c2").isGreaterThan(Value.of("aaa"))) + .and(Predicate.column("c2").isLessThanOrEqualTo(Value.of("ddd"))) + .orderBy(Ordering.column("c2").asc(), Ordering.column("c3").desc()) + .limit(10) + .build(); + +// With an equality predicate for an indexed column +SelectStatement statement5 = + StatementBuilder.select() + .from("ns", "tbl") + .where(Predicate.column("c4").isEqualTo(Value.of(100))) + .build(); + +// With an equality predicate for an indexed column and predicates for non-primary-key columns +SelectStatement statement5 = + StatementBuilder.select() + .from("ns", "tbl") + .where(Predicate.column("c4").isEqualTo(Value.of(100))) + .and(Predicate.column("c5").isEqualTo(Value.of(false))) + .build(); + +// With projections and an indexed column and limit +SelectStatement statement6 = + StatementBuilder.select("c1", "c2", "c3", "c4") + .from("ns", "tbl") + .where(Predicate.column("c4").isEqualTo(Value.of(100))) + .limit(10) + .build(); + +// With positional bind markers +SelectStatement statement7 = + StatementBuilder.select() + .from("ns", "tbl") + .where(Predicate.column("c1").isEqualTo(BindMarker.positional())) + .and(Predicate.column("c2").isGreaterThan(BindMarker.positional())) + .and(Predicate.column("c2").isLessThanOrEqualTo(BindMarker.positional())) + .orderBy(Ordering.column("c2").asc(), Ordering.column("c3").desc()) + .limit(BindMarker.positional()) + .build(); + +// With operations attributes +SelectStatement statement8 = + StatementBuilder.select() + .from("ns", "tbl") + .where(Predicate.column("c1").isEqualTo(Value.of(10))) + .and(Predicate.column("c2").isEqualTo(Value.of("aaa"))) + .and(Predicate.column("c3").isEqualTo(Value.of(1.23F))) + .withAttribute("attribute1", "value1") + .withAttribute("attribute2", "value2") + .build(); +``` + +Examples of `SELECT` with `JOIN` are as follows: + +```sql +-- For INNER JOIN and LEFT OUTER JOIN: +SELECT * FROM tbl1 as t1 + INNER JOIN tbl2 as t2 on t1.col1=t2.id1 and t1.col2=t2.id2 -- This part must have all primary key columns or a secondary index column of `tbl2`. + WHERE t1.pkey=1 -- Only columns of `tbl1` can be specified here. + ORDER BY t1.ckey DESC; -- Only columns of `tbl1` can be specified here. + +SELECT * FROM tbl1 as t1 + INNER JOIN tbl2 as t2 on t1.col1=t2.id -- This part must have all primary key columns or a secondary index column of `tbl2`. + LEFT OUTER JOIN tbl3 as t3 on t1.col2=t3.id -- This part must have all primary key columns or a secondary index column of `tbl3`. + WHERE t1.pkey=1 -- Only columns of `tbl1` can be specified here. + ORDER BY t1.ckey DESC; -- Only columns of `tbl1` can be specified here. + +-- For RIGHT OUTER JOIN: +SELECT * FROM tbl1 as t1 + RIGHT OUTER JOIN tbl2 as t2 on t1.id=t2.col -- Acceptable as the first join. And this part must have all primary key columns or a secondary index column of `tbl1`. + LEFT OUTER JOIN tbl3 as t3 on t1.col2=t3.id + WHERE t2.pkey=1 -- Only columns of `tbl2` can be specified here. + ORDER BY t2.ckey DESC; -- Only columns of `tbl2` can be specified here. + +SELECT * FROM tbl1 as t1 + RIGHT OUTER JOIN tbl2 as t2 on t1.id1=t2.col1 and t1.id2=t2.col2 -- This part must have all primary key columns or a secondary index column of `tbl1`. + WHERE t2.pkey=1 -- Only columns of `tbl2` can be specified here. + ORDER BY t2.ckey DESC; -- Only columns of `tbl2` can be specified here. +``` + +Examples of building statement objects for `SELECT` with `JOIN` are as follows: + +```java +// For INNER JOIN and LEFT OUTER JOIN: +SelectStatement statement1 = + StatementBuilder.select() + .from("tbl1", "t1") + .innerJoin("tbl2", "t2") + .on(JoinPredicate.column("t1", "col1").isEqualTo("t2", "id1")) + .and(JoinPredicate.column("t1", "col2").isEqualTo("t2", "id2")) // This part must have all primary key columns or a secondary index column of `tbl2`. + .where(Predicate.column("t1", "pkey").isEqualTo(Value.of(1))) // Only columns of `tbl1` can be specified here. + .orderBy(Ordering.column("t1", "ckey").desc()) // Only columns of `tbl1` can be specified here. + .build(); + +SelectStatement statement2 = + StatementBuilder.select() + .from("tbl1", "t1") + .innerJoin("tbl2", "t2") + .on(JoinPredicate.column("t1", "col1").isEqualTo("t2", "id")) // This part must have all primary key columns or a secondary index column of `tbl2`. + .leftOuterJoin("tbl3", "t3") + .on(JoinPredicate.column("t1", "col2").isEqualTo("t3", "id")) // This part must have all primary key columns or a secondary index column of `tbl3`. + .where(Predicate.column("t1", "pkey").isEqualTo(Value.of(1))) // Only columns of `tbl1` can be specified here. + .orderBy(Ordering.column("t1", "ckey").desc()) // Only columns of `tbl1` can be specified here. + .build(); + +// For RIGHT OUTER JOIN: +SelectStatement statement3 = + StatementBuilder.select() + .from("tbl1", "t1") + .rightOuterJoin("tbl2", "t2") + .on(JoinPredicate.column("t1", "id").isEqualTo("t2", "col")) // Acceptable as the first join. And this part must have all primary key columns or a secondary index column of `tbl1`. + .leftOuterJoin("tbl3", "t3") + .on(JoinPredicate.column("t1", "col2").isEqualTo("t3", "id")) + .where(Predicate.column("t2", "pkey").isEqualTo(Value.of(1))) // Only columns of `tbl2` can be specified here. + .orderBy(Ordering.column("t2", "ckey").desc()) // Only columns of `tbl2` can be specified here. + .build(); + +SelectStatement statement4 = + StatementBuilder.select() + .from("tbl1", "t1") + .rightOuterJoin("tbl2", "t2") + .on(JoinPredicate.column("t1", "id1").isEqualTo("t2", "col1")) + .and(JoinPredicate.column("t1", "id2").isEqualTo("t2", "col2")) // This part must have all primary key columns or a secondary index column of `tbl1`. + .where(Predicate.column("t2", "pkey").isEqualTo(Value.of(1))) // Only columns of `tbl2` can be specified here. + .orderBy(Ordering.column("t2", "ckey").desc()) // Only columns of `tbl2` can be specified here. + .build(); +``` + +#### Examples of cross-partition scan + +If you have the following table, for example: + +```sql +CREATE TABLE ns.user ( + id INT, + name TEXT, + age INT, + height FLOAT, + PRIMARY KEY (id) +) +``` + +Examples of `SELECT` with cross-partition scan are as follows: + +```sql +-- Without the WHERE clause to retrieve all the records of a table +SELECT * FROM ns.user; + +-- Without the WHERE clause and with projections and a limit +SELECT id, name FROM ns.user LIMIT 10; + +-- With AND predicates for non-primary-key columns +SELECT * FROM ns.user WHERE age > 10 AND height > 140; + +-- With OR predicates for non-primary key columns +SELECT * FROM ns.user WHERE age > 10 OR height > 140; + +-- With OR-wise of AND predicates +SELECT * FROM ns.user WHERE (age > 10 AND height > 150) OR (age > 15 AND height > 145); + +-- With AND-wise of OR predicates +SELECT * FROM ns.user WHERE (age < 10 OR age > 65) AND (height < 145 OR height > 175); + +-- With LIKE predicates +SELECT * FROM ns.user WHERE name LIKE 'A%' OR name NOT LIKE 'B_b'; + +-- With LIKE predicates with an escape character +SELECT * FROM ns.user WHERE name LIKE '+%Alice' ESCAPE '+'; + +-- With IS NULL predicates +SELECT * FROM ns.user WHERE name IS NOT NULL AND age IS NULL; + +-- With projections +SELECT name, age, height FROM ns.user WHERE (age < 10 OR age > 65) AND age <> 0; + +-- With limit +SELECT name, age, height FROM ns.user WHERE age < 10 OR age > 65 LIMIT 10; + +-- With orderings +SELECT * FROM ns.user WHERE age < 10 ORDER BY height DESC; + +-- With orderings without the WHERE clause +SELECT * FROM ns.user ORDER BY height; + +-- With positional bind markers +SELECT * FROM ns.user WHERE age < ? ORDER BY age ASC, height DESC LIMIT ?; +``` + +For examples that use the `JOIN` clause, see [Examples of partition scans and index scans](#examples-of-partition-scans-and-index-scans). + +Examples of building statement objects for `SELECT` are as follows: + +```java +// Without the WHERE clause to retrieve all the records of a table +SelectStatement statement1 = StatementBuilder.select().from("ns", "user").build(); + +// Without the WHERE clause and with projections and a limit +SelectStatement statement2 = + StatementBuilder.select("id", "name").from("ns", "user").limit(10).build(); + +// With AND predicates for non-primary-key columns +SelectStatement statement2 = + StatementBuilder.select() + .from("ns", "user") + .where(Predicate.column("age").isGreaterThan(Value.of(10))) + .and(Predicate.column("height").isGreaterThan(Value.of(140.0F))) + .build(); + +// With OR predicates for non-primary key columns +SelectStatement statement3 = + StatementBuilder.select() + .from("ns", "user") + .where(Predicate.column("age").isGreaterThan(Value.of(10))) + .or(Predicate.column("height").isGreaterThan(Value.of(140.0F))) + .build(); + +// With OR-wise of AND predicates +SelectStatement statement4 = + StatementBuilder.select() + .from("ns", "user") + .where( + AndPredicateList.predicate(Predicate.column("age").isGreaterThan(Value.of(10))) + .and(Predicate.column("height").isGreaterThan(Value.of(140.0F))) + .build()) + .or( + AndPredicateList.predicate(Predicate.column("age").isGreaterThan(Value.of(15))) + .and(Predicate.column("height").isGreaterThan(Value.of(145.0F))) + .build()) + .build(); + +// With AND-wise of OR predicates +SelectStatement statement5 = + StatementBuilder.select() + .from("ns", "user") + .where( + OrPredicateList.predicate(Predicate.column("age").isLessThan(Value.of(10))) + .or(Predicate.column("age").isGreaterThan(Value.of(65))) + .build()) + .and( + OrPredicateList.predicate(Predicate.column("height").isLessThan(Value.of(145.0F))) + .or(Predicate.column("height").isGreaterThan(Value.of(175.0F))) + .build()) + .build(); + +// With LIKE predicates +SelectStatement statement6 = + StatementBuilder.select() + .from("ns", "user") + .where(Predicate.column("name").isLike(Value.of("A%"))) + .or(Predicate.column("name").isNotLike(Value.of("B_b"))) + .build(); + +// With LIKE predicates with an escape character +SelectStatement statement7 = + StatementBuilder.select() + .from("ns", "user") + .where(Predicate.column("name").isLike(Value.of("+%Alice"), Value.of("+"))) + .build(); + +// With IS NULL predicates +SelectStatement statement8 = + StatementBuilder.select() + .from("ns", "user") + .where(Predicate.column("name").isNotNull()) + .and(Predicate.column("age").isNull()) + .build(); + +// With projections +SelectStatement statement9 = + StatementBuilder.select("name", "age", "height") + .from("ns", "user") + .where( + OrPredicateList.predicate(Predicate.column("age").isLessThan(Value.of(10))) + .or(Predicate.column("age").isGreaterThan(Value.of(65))) + .build()) + .and(Predicate.column("height").isNotEqualTo(Value.of(0))) + .build(); + +// With limit +SelectStatement statement10 = + StatementBuilder.select("name", "age", "height") + .from("ns", "user") + .where(Predicate.column("age").isLessThan(Value.of(10))) + .or(Predicate.column("age").isGreaterThan(Value.of(65))) + .limit(10) + .build(); + +// With orderings +SelectStatement statement11 = + StatementBuilder.select() + .from("ns", "user") + .where(Predicate.column("age").isLessThan(Value.of(10))) + .orderBy(Ordering.column("height").desc()) + .build(); + +// With orderings without the WHERE clause +SelectStatement statement12 = + StatementBuilder.select() + .from("ns", "user") + .orderBy(Ordering.column("height").desc()) + .build(); + +// With positional bind markers +SelectStatement statement13 = + StatementBuilder.select() + .from("ns", "user") + .where(Predicate.column("age").isLessThan(BindMarker.positional())) + .orderBy(Ordering.column("age").asc(), Ordering.column("height").desc()) + .limit(BindMarker.positional()) + .build(); +``` + +For examples that use the `JOIN` clause, see [Examples of partition scans and index scans](#examples-of-partition-scans-and-index-scans). + +### INSERT + +The `INSERT` command inserts new records into the database. If any of the target records already exist, a transaction conflict error will be thrown. + +This command returns the following column: + +- `updateCount`: `INT` - the number of inserted records + +#### Grammar + +```sql +INSERT INTO [.]
[( [, ] ...)] + VALUES ( [, ] ...) [, ( [, ] ...)] ... + [WITH operation_attributes] + +operation_attributes: = [AND =] ... +``` + +- You must specify a full primary key in `INSERT`. +- You can specify `` to a bind marker (positional `?` and named `:`). See the [Literal](#literal) section for the literal syntax. + +#### Examples + +Examples of `INSERT` are as follows: + +```sql +-- Insert a record without specifying column names +INSERT INTO ns.tbl VALUES (10, 'aaa', 1.23, 100, true); + +-- Insert a record with column names +INSERT INTO ns.tbl (c1, c2, c3, c4) VALUES (10, 'aaa', 1.23, 100); + +-- With positional bind markers +INSERT INTO tbl VALUES (?, ?, ?, ?, ?); + +-- Insert multiple records +INSERT INTO ns.tbl (c1, c2, c3, c4) VALUES (10, 'aaa', 1.23, 100), (20, 'bbb', 4.56, 200); +``` + +Examples of building statement objects for `INSERT` are as follows: + +```java +// Insert a record without specifying column names. +InsertStatement statement1 = StatementBuilder.insertInto("ns", "tbl") + .values(Value.ofInt(10), Value.ofText("aaa"), Value.of(1.23F), Value.of(100L), Value.of(true)) + .build(); + +// Insert a record with column names. +InsertStatement statement2 = StatementBuilder.insertInto("ns", "tbl") + .columns("c1", "c2", "c3", "c4", "c5") + .values(Value.ofInt(10), Value.ofText("aaa"), Value.of(1.23F), Value.of(100L), Value.of(true)) + .build(); + +// With positional bind markers +InsertStatement statement3 = + StatementBuilder.insertInto("tbl") + .columns("c1", "c2", "c3", "c4", "c5") + .values( + BindMarker.positional(), + BindMarker.positional(), + BindMarker.positional(), + BindMarker.positional(), + BindMarker.positional()) + .build(); + +// Insert multiple records. +InsertStatement statement4 = StatementBuilder.insertInto("ns", "tbl") + .columns("c1", "c2", "c3", "c4", "c5") + .values(Value.ofInt(10), Value.ofText("aaa"), Value.of(1.23F), Value.of(100L), Value.of(true)) + .values(Value.ofInt(20), Value.ofText("bbb"), Value.of(2.46F), Value.of(200L), Value.of(false)) + .build(); +``` + +### UPSERT + +The `UPSERT` command inserts new records into the database if they don't exist or updates the target records if they already exist. + +This command returns the following column: + +- `updateCount`: `INT` - the number of inserted or updated records + +#### Grammar + +```sql +UPSERT INTO [.]
[( [, ] ...)] + VALUES ( [, ] ...) [, ( [, ] ...)] ... + [WITH operation_attributes] + +operation_attributes: operation_attribute [AND operation_attribute] ... +operation_attribute: = +``` + +- You must specify a full primary key in `UPSERT`. +- You can specify `` to a bind marker (positional `?` and named `:`). See the [Literal](#literal) section for the literal syntax. + +#### Examples + +Examples of `UPSERT` are as follows: + +```sql +-- Upsert a record without specifying column names. +UPSERT INTO ns.tbl VALUES (10, 'aaa', 1.23, 100, true); + +-- Upsert a record with column names. +UPSERT INTO ns.tbl (c1, c2, c3, c4) VALUES (10, 'aaa', 1.23, 100); + +-- With positional bind markers +UPSERT INTO tbl VALUES (?, ?, ?, ?, ?); + +-- Upsert multiple records. +UPSERT INTO ns.tbl (c1, c2, c3, c4) VALUES (10, 'aaa', 1.23, 100), (20, 'bbb', 4.56, 200); + +-- With operations attributes +UPSERT INTO ns.tbl VALUES (10, 'aaa', 1.23, 100, true) WITH 'attribute1' = 'value1' AND 'attribute2' = 'value2'; +``` + +Examples of building statement objects for `UPSERT` are as follows: + +```java +// Upsert a record without specifying column names. +UpsertStatement statement1 = + StatementBuilder.upsertInto("ns", "tbl") + .values( + Value.ofInt(10), + Value.ofText("aaa"), + Value.of(1.23F), + Value.of(100L), + Value.of(true)) + .build(); + +// Upsert a record with column names. +UpsertStatement statement2 = + StatementBuilder.upsertInto("ns", "tbl") + .columns("c1", "c2", "c3", "c4", "c5") + .values( + Value.ofInt(10), + Value.ofText("aaa"), + Value.of(1.23F), + Value.of(100L), + Value.of(true)) + .build(); + +// With positional bind markers +UpsertStatement statement3 = + StatementBuilder.upsertInto("tbl") + .columns("c1", "c2", "c3", "c4", "c5") + .values( + BindMarker.positional(), + BindMarker.positional(), + BindMarker.positional(), + BindMarker.positional(), + BindMarker.positional()) + .build(); + +// Upsert multiple records. +UpsertStatement statement4 = + StatementBuilder.upsertInto("ns", "tbl") + .columns("c1", "c2", "c3", "c4", "c5") + .values( + Value.ofInt(10), + Value.ofText("aaa"), + Value.of(1.23F), + Value.of(100L), + Value.of(true)) + .values( + Value.ofInt(20), + Value.ofText("bbb"), + Value.of(2.46F), + Value.of(200L), + Value.of(false)) + .build(); + +// With operations attributes +UpsertStatement statement5 = + StatementBuilder.upsertInto("ns", "tbl") + .values( + Value.ofInt(10), + Value.ofText("aaa"), + Value.of(1.23F), + Value.of(100L), + Value.of(true)) + .withAttribute("attribute1", "value1") + .withAttribute("attribute2", "value2") + .build(); +``` + +### UPDATE + +The `UPDATE` command updates existing records in the database. You can specify any conditions in the `WHERE` clause to filter records. However, specifying a primary key uniquely as much as possible is recommended to avoid the cross-partition operation since it might cause performance and consistency issues, especially in non-JDBC databases. Because ScalarDB SQL creates an execution plan for an `UPDATE` command that uses a `Get` or `Scan` operation to identify the target records, the same rule is applied for the selection of records. To understand what kinds of `WHERE` clauses cause cross-partition operations and to avoid such operations, see [SELECT](#select). + +You need to enable the cross-partition scan option if you want to update all records across partitions without specifying the `WHERE` clause. You also need to enable the cross-partition scan with filtering option if you want to flexibly update records across partitions with arbitrary conditions in the `WHERE` clause. For details about configurations, see [Cross-partition scan configurations](../configurations.mdx#cross-partition-scan-configurations) and [ScalarDB Cluster SQL configurations](../scalardb-cluster/developer-guide-for-scalardb-cluster-with-java-api.mdx#scalardb-cluster-sql-client-configurations). + +:::warning + +For non-JDBC databases, transactions could be executed at read-committed snapshot isolation (`SNAPSHOT`), which is a lower isolation level, even if you enable cross-partition scan with the `SERIALIZABLE` isolation level. When using non-JDBC databases, use cross-partition scan only if consistency does not matter for your transactions. + +::: + +This command returns the following column: + +- `updateCount`: `INT` - the number of updated records + +#### Grammar + +```sql +UPDATE [.]
[AS ] + SET = [, = ] ... + [WHERE and_predicates [OR and_predicates ...] | or_predicates [AND or_predicates ...]] + [WITH operation_attributes] + +identifier: [[.]
.] | [alias.] +and_predicates: predicate | (predicate [AND predicate ...]) +or_predicates: predicate | (predicate [OR predicate ...]) +predicate: operator | BETWEEN AND | [NOT] LIKE [ESCAPE ] | IS [NOT] NULL +operator: = | <> | != | > | >= | < | <= +operation_attributes: operation_attribute [AND operation_attribute] ... +operation_attribute: = +``` + +##### Note + +`WHERE` clause: + +- You can use arbitrary predicates for any columns in the `WHERE` clause. +- In the `WHERE` clause, predicates must be an OR-wise of `and_predicates` (known as disjunctive normal form) or an AND-wise of `or_predicates` (known as conjunctive normal form). +- When connecting multiple `and_predicates` or `or_predicates`, which have more than one predicate, you need to put parentheses around `and_predicates` and `or_predicates`. +- You can specify `` to a bind marker (positional `?` and named `:`). See the [Literal](#literal) section for the literal syntax. +- You cannot specify encrypted columns in the `WHERE` clause. + +`LIKE` predicate: + +- `_` in `` matches any single character. +- `%` in `` matches any sequence of zero or more characters. +- `\` in `` works as the escape character by default. + - You can change the escape character by specifying the `ESCAPE` clause. + - You can disable the escape character by specifying an empty escape character, `ESCAPE ''`. + +#### Examples with the full primary key specified + +If you have the following table, for example: + +```sql +CREATE TABLE ns.tbl ( + c1 INT, + c2 TEXT, + c3 FLOAT, + c4 BIGINT, + c5 BOOLEAN, + PRIMARY KEY (c1, c2, c3) +) WITH CLUSTERING ORDER BY (c2 DESC, c3 ASC); +``` + +Examples of `UPDATE` with the full primary key specified are as follows: + +```sql +-- Update a record +UPDATE ns.tbl SET c4 = 200, c5 = false WHERE c1 = 10 AND c2 = 'aaa' AND c3 = 1.23; + +-- With positional bind markers +UPDATE ns.tbl SET c4 = ?, c5 = ? WHERE c1 = ? AND c2 = ? AND c3 = ?; + +-- With operations attributes +UPDATE ns.tbl SET c4 = 200, c5 = false WHERE c1 = 10 AND c2 = 'aaa' AND c3 = 1.23 WITH 'attribute1' = 'value1' AND 'attribute2' = 'value2'; +``` + +Examples of building statement objects for `UPDATE` are as follows: + +```java +// Update a record +UpdateStatement statement1 = + StatementBuilder.update("ns", "tbl") + .set( + Assignment.column("c4").value(Value.of(200L)), + Assignment.column("c5").value(Value.of(false))) + .where(Predicate.column("c1").isEqualTo(Value.of(10))) + .and(Predicate.column("c2").isEqualTo(Value.of("aaa"))) + .and(Predicate.column("c3").isEqualTo(Value.of(1.23F))) + .build(); + +// With positional bind markers +UpdateStatement statement2 = + StatementBuilder.update("tbl") + .set( + Assignment.column("c4").value(BindMarker.positional()), + Assignment.column("c5").value(BindMarker.positional())) + .where(Predicate.column("c1").isEqualTo(BindMarker.positional())) + .and(Predicate.column("c2").isEqualTo(BindMarker.positional())) + .and(Predicate.column("c3").isEqualTo(BindMarker.positional())) + .build(); + +// With operations attributes +UpdateStatement statement3 = + StatementBuilder.update("ns", "tbl") + .set( + Assignment.column("c4").value(Value.of(200L)), + Assignment.column("c5").value(Value.of(false))) + .where(Predicate.column("c1").isEqualTo(Value.of(10))) + .and(Predicate.column("c2").isEqualTo(Value.of("aaa"))) + .and(Predicate.column("c3").isEqualTo(Value.of(1.23F))) + .withAttribute("attribute1", "value1") + .withAttribute("attribute2", "value2") + .build(); +``` + +#### Examples without the full primary key specified + +If you have the following table, for example: + +```sql +CREATE TABLE ns.user ( + id INT, + name TEXT, + age INT, + height FLOAT, + PRIMARY KEY (id) +) +``` + +Examples of `UPDATE` without the full primary key specified are as follows: + +```sql +-- Without the WHERE clause to update all the records of a table +UPDATE ns.user SET age = 31; + +-- With AND predicates for non-primary key columns +UPDATE ns.user SET age = 31 WHERE age > 10 AND height > 140; + +-- With OR predicates for non-primary key columns +UPDATE ns.user SET age = 31 WHERE age > 10 OR height > 140; + +-- With OR-wise of AND predicates +UPDATE ns.user SET age = 31 WHERE (age > 10 AND height > 150) OR (age > 15 AND height > 145); + +-- With AND-wise of OR predicates +UPDATE ns.user SET age = 31 WHERE (age < 10 OR age > 65) AND (height < 145 OR height > 175); + +-- With LIKE predicates +UPDATE ns.user SET age = 31 WHERE name LIKE 'A%' OR name NOT LIKE 'B_b'; + +-- With LIKE predicates with an escape character +UPDATE ns.user SET age = 31 WHERE name LIKE '+%Alice' ESCAPE '+'; + +-- With IS NULL predicates +UPDATE ns.user SET age = 31 WHERE name IS NOT NULL AND age IS NULL; + +-- With positional bind markers +UPDATE ns.user SET age = ? WHERE age < ?; +``` + +Examples of building statement objects for `UPDATE` are as follows: + +```java +// Without the WHERE clause to update all the records of a table +UpdateStatement statement1 = + StatementBuilder.update("ns", "user") + .set(Assignment.column("age").value(Value.of(30))) + .build(); + +// With AND predicates for non-primary key columns +UpdateStatement statement2 = + StatementBuilder.update("ns", "user") + .set(Assignment.column("age").value(Value.of(30))) + .where(Predicate.column("age").isGreaterThan(Value.of(10))) + .and(Predicate.column("height").isGreaterThan(Value.of(140.0F))) + .build(); + +// With OR predicates for non-primary key columns +UpdateStatement statement3 = + StatementBuilder.update("ns", "user") + .set(Assignment.column("age").value(Value.of(30))) + .where(Predicate.column("age").isGreaterThan(Value.of(10))) + .or(Predicate.column("height").isGreaterThan(Value.of(140.0F))) + .build(); + +// With OR-wise of AND predicates +UpdateStatement statement4 = + StatementBuilder.update("ns", "user") + .set(Assignment.column("age").value(Value.of(30))) + .where( + AndPredicateList.predicate(Predicate.column("age").isGreaterThan(Value.of(10))) + .and(Predicate.column("height").isGreaterThan(Value.of(140.0F))) + .build()) + .or( + AndPredicateList.predicate(Predicate.column("age").isGreaterThan(Value.of(15))) + .and(Predicate.column("height").isGreaterThan(Value.of(145.0F))) + .build()) + .build(); + +// With AND-wise of OR predicates +UpdateStatement statement5 = + StatementBuilder.update("ns", "user") + .set(Assignment.column("age").value(Value.of(30))) + .where( + OrPredicateList.predicate(Predicate.column("age").isLessThan(Value.of(10))) + .or(Predicate.column("age").isGreaterThan(Value.of(65))) + .build()) + .and( + OrPredicateList.predicate(Predicate.column("height").isLessThan(Value.of(145.0F))) + .or(Predicate.column("height").isGreaterThan(Value.of(175.0F))) + .build()) + .build(); + +// With LIKE predicates +UpdateStatement statement6 = + StatementBuilder.update("ns", "user") + .set(Assignment.column("age").value(Value.of(30))) + .where(Predicate.column("name").isLike(Value.of("A%"))) + .or(Predicate.column("name").isNotLike(Value.of("B_b"))) + .build(); + +// With LIKE predicates with an escape character +UpdateStatement statement7 = + StatementBuilder.update("ns", "user") + .set(Assignment.column("age").value(Value.of(30))) + .where(Predicate.column("name").isLike(Value.of("+%Alice"), Value.of("+"))) + .build(); + +// With IS NULL predicates +UpdateStatement statement8 = + StatementBuilder.update("ns", "user") + .set(Assignment.column("age").value(Value.of(30))) + .where(Predicate.column("name").isNotNull()) + .and(Predicate.column("age").isNull()) + .build(); + +// With positional bind markers +UpdateStatement statement9 = + StatementBuilder.update("ns", "user") + .set(Assignment.column("age").value(BindMarker.positional())) + .where(Predicate.column("age").isLessThan(BindMarker.positional())) + .build(); +``` + +### DELETE + +The `DELETE` command deletes records in the database. You can specify any conditions in the `WHERE` clause to filter records. However, specifying a primary key uniquely is recommended as much as possible to avoid the cross-partition operation since it might cause performance and consistency issues, especially in non-JDBC databases. Because ScalarDB SQL creates an execution plan for a `DELETE` command that uses a `Get` or `Scan` operation to identify the target records, the same rule is applied for the selection of records. To understand what kinds of `WHERE` clauses cause cross-partition operations and to avoid such operations, see [SELECT](#select). + +You need to enable the cross-partition scan option if you want to delete all records across partitions without specifying the `WHERE` clause. You also need to enable the cross-partition scan with filtering option if you want to flexibly delete records across partitions with arbitrary conditions in the `WHERE` clause. For details about configurations, see [Cross-partition scan configurations](../configurations.mdx#cross-partition-scan-configurations) and [ScalarDB Cluster SQL configurations](../scalardb-cluster/developer-guide-for-scalardb-cluster-with-java-api.mdx#scalardb-cluster-sql-client-configurations). + +:::warning + +For non-JDBC databases, transactions could be executed at read-committed snapshot isolation (`SNAPSHOT`), which is a lower isolation level, even if you enable cross-partition scan with the `SERIALIZABLE` isolation level. When using non-JDBC databases, use cross-partition scan only if consistency does not matter for your transactions. + +::: + +This command returns the following column: + +- `updateCount`: `INT` - the number of deleted records + +#### Grammar + +```sql +DELETE FROM [.]
[AS ] + [WHERE and_predicates [OR and_predicates ...] | or_predicates [AND or_predicates ...]] + [WITH operation_attributes] + +identifier: [[.]
.] | [alias.] +and_predicates: predicate | (predicate [AND predicate ...]) +or_predicates: predicate | (predicate [OR predicate ...]) +predicate: operator | BETWEEN AND | [NOT] LIKE [ESCAPE ] | IS [NOT] NULL +operator: = | <> | != | > | >= | < | <= +operation_attributes: operation_attribute [AND operation_attribute] ... +operation_attribute: = +``` + +##### Note + +`WHERE` clause: + +- You can use arbitrary predicates for any columns in the `WHERE` clause. +- In the `WHERE` clause, predicates must be an OR-wise of `and_predicates` (known as disjunctive normal form) or an AND-wise of `or_predicates` (known as conjunctive normal form). +- When connecting multiple `and_predicates` or `or_predicates`, which have more than one predicate, you need to put parentheses around `and_predicates` and `or_predicates`. +- You can specify `` to a bind marker (positional `?` and named `:`). See the [Literal](#literal) section for the literal syntax. +- You cannot specify encrypted columns in the `WHERE` clause. + +`LIKE` predicate: + +- `_` in `` matches any single character. +- `%` in `` matches any sequence of zero or more characters. +- `\` in `` works as the escape character by default. + - You can change the escape character by specifying the `ESCAPE` clause. + - You can disable the escape character by specifying an empty escape character, `ESCAPE ''`. + +#### Examples with the full primary key specified + +If you have the following table, for example: + +```sql +CREATE TABLE ns.tbl ( + c1 INT, + c2 TEXT, + c3 FLOAT, + c4 BIGINT, + c5 BOOLEAN, + PRIMARY KEY (c1, c2, c3) +) WITH CLUSTERING ORDER BY (c2 DESC, c3 ASC); +``` + +Examples of `DELETE` are as follows: + +```sql +-- Delete a record +DELETE FROM ns.tbl WHERE c1 = 10 AND c2 = 'aaa' AND c3 = 1.23; + +-- With positional bind markers +DELETE FROM tbl WHERE c1 = ? AND c2 = ? AND c3 = ?; + +-- With operations attributes +DELETE FROM ns.tbl WHERE c1 = 10 AND c2 = 'aaa' AND c3 = 1.23 WITH 'attribute1' = 'value1' AND 'attribute2' = 'value2'; +``` + +Examples of building statement objects for `DELETE` are as follows: + +```java +// Delete a record +DeleteStatement statement1 = + StatementBuilder.deleteFrom("ns", "tbl") + .where(Predicate.column("c1").isEqualTo(Value.of(10))) + .and(Predicate.column("c2").isEqualTo(Value.of("aaa"))) + .and(Predicate.column("c3").isEqualTo(Value.of(1.23F))) + .build(); + +// With positional bind markers +DeleteStatement statement2 = + StatementBuilder.deleteFrom("tbl") + .where(Predicate.column("c1").isEqualTo(BindMarker.positional())) + .and(Predicate.column("c2").isEqualTo(BindMarker.positional())) + .and(Predicate.column("c3").isEqualTo(BindMarker.positional())) + .build(); + +// With operations attributes +DeleteStatement statement3 = + StatementBuilder.deleteFrom("ns", "tbl") + .where(Predicate.column("c1").isEqualTo(Value.of(10))) + .and(Predicate.column("c2").isEqualTo(Value.of("aaa"))) + .and(Predicate.column("c3").isEqualTo(Value.of(1.23F))) + .withAttribute("attribute1", "value1") + .withAttribute("attribute2", "value2") + .build(); +``` + +#### Examples without the full primary key specified + +If you have the following table, for example: + +```sql +CREATE TABLE ns.user ( + id INT, + name TEXT, + age INT, + height FLOAT, + PRIMARY KEY (id) +) +``` + +Examples of `DELETE` with cross-partition scan filtering are as follows: + +```sql +-- Without the WHERE clause to delete all the records of a table +DELETE FROM ns.user; + +-- With AND predicates for non-primary key columns +DELETE FROM ns.user WHERE age > 10 AND height > 140; + +-- With OR predicates for non-primary key columns +DELETE FROM ns.user WHERE age > 10 OR height > 140; + +-- With OR-wise of AND predicates +DELETE FROM ns.user WHERE (age > 10 AND height > 150) OR (age > 15 AND height > 145); + +-- With AND-wise of OR predicates +DELETE FROM ns.user WHERE (age < 10 OR age > 65) AND (height < 145 OR height > 175); + +-- With LIKE predicates +DELETE FROM ns.user WHERE name LIKE 'A%' OR name NOT LIKE 'B_b'; + +-- With LIKE predicates with an escape character +DELETE FROM ns.user WHERE name LIKE '+%Alice' ESCAPE '+'; + +-- With IS NULL predicates +DELETE FROM ns.user WHERE name IS NOT NULL AND age IS NULL; + +-- With positional bind markers +DELETE FROM ns.user WHERE age < ?; +``` + +Examples of building statement objects for `DELETE` are as follows: + +```java +// Without the WHERE clause to delete all the records of a table +DeleteStatement statement1 = StatementBuilder.deleteFrom("ns", "tbl").build(); + +// With AND predicates for non-primary key columns +DeleteStatement statement2 = + StatementBuilder.deleteFrom("ns", "tbl") + .where(Predicate.column("age").isGreaterThan(Value.of(10))) + .and(Predicate.column("height").isGreaterThan(Value.of(140.0F))) + .build(); + +// With OR predicates for non-primary key columns +DeleteStatement statement3 = + StatementBuilder.deleteFrom("ns", "tbl") + .where(Predicate.column("age").isGreaterThan(Value.of(10))) + .or(Predicate.column("height").isGreaterThan(Value.of(140.0F))) + .build(); + +// With OR-wise of AND predicates +DeleteStatement statement4 = + StatementBuilder.deleteFrom("ns", "tbl") + .where( + AndPredicateList.predicate(Predicate.column("age").isGreaterThan(Value.of(10))) + .and(Predicate.column("height").isGreaterThan(Value.of(140.0F))) + .build()) + .or( + AndPredicateList.predicate(Predicate.column("age").isGreaterThan(Value.of(15))) + .and(Predicate.column("height").isGreaterThan(Value.of(145.0F))) + .build()) + .build(); + +// With AND-wise of OR predicates +DeleteStatement statement5 = + StatementBuilder.deleteFrom("ns", "tbl") + .where( + OrPredicateList.predicate(Predicate.column("age").isLessThan(Value.of(10))) + .or(Predicate.column("age").isGreaterThan(Value.of(65))) + .build()) + .and( + OrPredicateList.predicate(Predicate.column("height").isLessThan(Value.of(145.0F))) + .or(Predicate.column("height").isGreaterThan(Value.of(175.0F))) + .build()) + .build(); + +// With LIKE predicates +DeleteStatement statement6 = + StatementBuilder.deleteFrom("ns", "tbl") + .where(Predicate.column("name").isLike(Value.of("A%"))) + .or(Predicate.column("name").isNotLike(Value.of("B_b"))) + .build(); + +// With LIKE predicates with an escape character +DeleteStatement statement7 = + StatementBuilder.deleteFrom("ns", "tbl") + .where(Predicate.column("name").isLike(Value.of("+%Alice"), Value.of("+"))) + .build(); + +// With IS NULL predicates +DeleteStatement statement8 = + StatementBuilder.deleteFrom("ns", "tbl") + .where(Predicate.column("name").isNotNull()) + .and(Predicate.column("age").isNull()) + .build(); + +// With positional bind markers +DeleteStatement statement9 = + StatementBuilder.deleteFrom("ns", "tbl") + .where(Predicate.column("age").isLessThan(BindMarker.positional())) + .build(); +``` + +## DCL + +### CREATE USER + +The `CREATE USER` command creates a new user with the specified username, password, and user attributes. + +#### Grammar + +```sql +CREATE USER [WITH] {PASSWORD | SUPERUSER | NO_SUPERUSER} ... +``` + +- If you don't specify a password for a user, the user is created without a password. +- If you specify the `SUPERUSER` attribute for a user, the user becomes a superuser. +- If you specify the `NO_SUPERUSER` attribute for a user, the user becomes a normal user. +- If you don't specify either `SUPERUSER` or `NO_SUPERUSER` for a user, the user becomes a normal user. + +#### Examples + +Examples of `CREATE USER` are as follows: + +```sql +-- Create a user with a password as a superuser. +CREATE USER user1 WITH PASSWORD 'password1' SUPERUSER; + +-- Create a user with a password. +CREATE USER user2 WITH PASSWORD 'password2'; + +-- Create a user without a password. +CREATE USER user3; +``` + +Examples of building statement objects for `CREATE USER` are as follows: + +```java +// Create a user with a password and the `SUPERUSER` attribute. +CreateUserStatement statement1 = + StatementBuilder.createUser("user1").with("password", UserOption.SUPERUSER).build(); + +// Create a user with a password. +CreateUserStatement statement2 = StatementBuilder.createUser("user1").with("password").build(); + +// Create a user without a password. +CreateUserStatement statement3 = StatementBuilder.createUser("user1").build(); +``` + +### ALTER USER + +The `ALTER USER` command changes the password and user attributes of the specified user. + +#### Grammar + +```sql +ALTER USER [WITH] {PASSWORD | SUPERUSER | NO_SUPERUSER} ... +``` + +- If you specify the `SUPERUSER` attribute for a user, the user becomes a superuser. +- If you specify the `NO_SUPERUSER` attribute for a user, the user becomes a normal user. + +#### Examples + +Examples of `ALTER USER` are as follows: + +```sql +-- Change the password of a user. +ALTER USER user1 WITH PASSWORD 'password1'; + +-- Change a user to a superuser. +ALTER USER user1 WITH SUPERUSER; +``` + +Examples of building statement objects for `ALTER USER` are as follows: + +```java +// Change the password of a user. +AlterUserStatement statement1 = StatementBuilder.alterUser("user1").with("password2").build(); + +// Change a user to a superuser. +AlterUserStatement statement2 = + StatementBuilder.alterUser("user1").with(UserOption.SUPERUSER).build(); +``` + +### DROP USER + +The `DROP USER` command deletes the specified user. + +#### Grammar + +```sql +DROP USER +``` + +#### Examples + +An example of `DROP USER` is as follows: + +```sql +-- Delete a user. +DROP USER user1; +``` + +An example of building statement objects for `DROP USER` is as follows: + +```java +// Delete a user. +DropUserStatement statement = StatementBuilder.dropUser("user1").build(); +``` + +### GRANT + +The `GRANT` command grants privileges to the specified user. + +#### Grammar + +```sql +GRANT {privilege [, privilege] ... | ALL [PRIVILEGES]} ON [TABLE]
[,
] ... TO [USER] [, ] ... [WITH GRANT OPTION] +GRANT {privilege [, privilege] ... | ALL [PRIVILEGES]} ON NAMESPACE [, ] ... TO [USER] [, ] ... [WITH GRANT OPTION] + +privilege: SELECT | INSERT | UPDATE | DELETE | CREATE | DROP | TRUNCATE | ALTER +``` + +- You must grant `INSERT` and `UPDATE` privileges together. +- To grant a user the `UPDATE` or `DELETE` privilege, the target user must have the `SELECT` privilege. +- If you specify the `WITH GRANT OPTION` option, the user can grant privileges to other users. + +#### Examples + +Examples of `GRANT` are as follows: + +```sql +-- Grant the SELECT privilege on a table to a user. +GRANT SELECT ON ns.tbl TO user1; + +-- Grant the SELECT privilege on tables to users. +GRANT SELECT ON ns.tbl1, ns.tbl2 TO user1, user2; + +-- Grant the SELECT privilege on all tables in a namespace to a user. +GRANT SELECT ON NAMESPACE ns TO user1; + +-- Grant all privileges and GRANT OPTION on a table to a user. +GRANT ALL ON ns.tbl TO user1 WITH GRANT OPTION; + +-- Grant all privileges and GRANT OPTION on all tables in a namespace to a user. +GRANT ALL ON NAMESPACE ns TO user1 WITH GRANT OPTION; +``` + +Examples of building statement objects for `GRANT` are as follows: + +```java +// Grant the SELECT privilege on a table to a user. +GrantStatement statement1 = + StatementBuilder.grant(Privilege.SELECT).onTable("ns", "tbl").toUser("user1").build(); + +// Grant the SELECT privilege on tables to users. +GrantStatement statement2 = + StatementBuilder.grant(Privilege.SELECT) + .onTable("ns", "tbl1", "ns", "tbl2") + .toUser("user1", "user2") + .build(); + +// Grant the SELECT privilege on all tables in a namespace to a user. +GrantStatement statement3 = + StatementBuilder.grant(Privilege.SELECT).onNamespace("ns").toUser("user1").build(); + +// Grant all privileges and GRANT OPTION on a table to a user. +GrantStatement statement4 = + StatementBuilder.grant(Privilege.values()) + .onTable("ns", "tbl") + .toUser("user1") + .withGrantOption() + .build(); + +// Grant all privileges and GRANT OPTION on all tables in a namespace to a user. +GrantStatement statement5 = + StatementBuilder.grant(Privilege.values()) + .onNamespace("ns") + .toUser("user1") + .withGrantOption() + .build(); +``` + +### REVOKE + +The `REVOKE` command revokes privileges from the specified user. + +#### Grammar + +```sql +REVOKE {privilege [, privilege] ... | ALL [PRIVILEGES]} [, GRANT OPTION] ON [TABLE]
[,
] ... FROM [USER] [, ] ... +REVOKE GRANT OPTION ON [TABLE]
[,
] ... FROM [USER] [, ] ... +REVOKE {privilege [, privilege] ... | ALL [PRIVILEGES]} [, GRANT OPTION] ON NAMESPACE [, ] ... FROM [USER] [, ] ... +REVOKE GRANT OPTION ON NAMESPACE [, ] ... FROM [USER] [, ] ... + +privilege: SELECT | INSERT | UPDATE | DELETE | CREATE | DROP | TRUNCATE | ALTER +``` + +- You must revoke `INSERT` and `UPDATE` privileges together. +- If the target user has the `INSERT` or `UPDATE` privilege, you cannot revoke the `SELECT` privilege from them. + +#### Examples + +Examples of `REVOKE` are as follows: + +```sql +-- Revoke the SELECT privilege on a table from a user. +REVOKE SELECT ON ns.tbl FROM user1; + +-- Revoke the SELECT privilege on tables from users. +REVOKE SELECT ON ns.tbl1, ns.tbl2 FROM user1, user2; + +-- Revoke the SELECT privilege on all tables in a namespace from a user. +REVOKE SELECT ON NAMESPACE ns FROM user1; + +-- Revoke all privileges and GRANT OPTION on a table from a user. +REVOKE ALL, GRANT OPTION ON ns.tbl FROM user1; + +-- Revoke all privileges and GRANT OPTION on all tables in a namespace from a user. +REVOKE ALL, GRANT OPTION ON NAMESPACE ns FROM user1; +``` + +Examples of building statement objects for `REVOKE` are as follows: + +```java +// Revoke the SELECT privilege on a table from a user. +RevokeStatement statement1 = + StatementBuilder.revoke(Privilege.SELECT).onTable("ns", "tbl").fromUser("user1").build(); + +// Revoke the SELECT privilege on tables from users. +RevokeStatement statement2 = + StatementBuilder.revoke(Privilege.SELECT) + .onTable("ns", "tbl1", "ns", "tbl2") + .fromUser("user1", "user2") + .build(); + +// Revoke the SELECT privilege on all tables in a namespace from a user. +RevokeStatement statement3 = + StatementBuilder.revoke(Privilege.SELECT).onNamespace("ns").fromUser("user1").build(); + +// Revoke all privileges and GRANT OPTION on a table from a user. +RevokeStatement statement4 = + StatementBuilder.revoke(Privilege.values()) + .onTable("ns", "tbl") + .fromUser("user1") + .build(); + +// Revoke all privileges and GRANT OPTION on all tables in a namespace from a user. +RevokeStatement statement5 = + StatementBuilder.revoke(Privilege.values()) + .onNamespace("ns") + .fromUser("user1") + .build(); +``` + +## Others + +### USE + +The `USE` command specifies a default namespace. If a namespace name is omitted in a SQL statement, the default namespace will be used. + +#### Grammar + +```sql +USE +``` + +#### Examples + +An example of `USE` is as follows: + +```sql +-- Specify a default namespace name "ns" +USE ns; +``` + +An example of building statement objects for `USE` is as follows: + +```java +// Specify a default namespace name "ns" +UseStatement statement = StatementBuilder.use("ns").build(); +``` + +### BEGIN + +The `BEGIN` command begins a transaction. + +This command returns the following column: + +- `transactionId`: `TEXT` - a transaction ID associated with the transaction you have begun + +#### Grammar + +```sql +BEGIN [READ ONLY | READ WRITE] +``` + +- If you specify `READ ONLY`, the transaction will be started in read-only mode. +- If you specify `READ WRITE`, the transaction will be started in read-write mode. +- If you omit the `READ ONLY` or `READ WRITE` option, the transaction will be started as a read-write transaction by default. + +#### Examples + +An example of building statement objects for `BEGIN` is as follows: + +```java +// Begin a transaction. +BeginStatement statement1 = StatementBuilder.begin().build(); + +// Begin a transaction in read-only mode. +BeginStatement statement2 = StatementBuilder.begin().readOnly().build(); + +// Begin a transaction in read-write mode. +BeginStatement statement3 = StatementBuilder.begin().readWrite().build(); +``` + +### START TRANSACTION + +The `START TRANSACTION` command starts a transaction. This command is an alias of `BEGIN`. + +This command returns the following column: + +- `transactionId`: `TEXT` - the transaction ID associated with the transaction you have started + +#### Grammar + +```sql +START TRANSACTION [READ ONLY | READ WRITE] +``` + +- If you specify `READ ONLY`, the transaction will be started in read-only mode. +- If you specify `READ WRITE`, the transaction will be started in read-write mode. +- If you omit the `READ ONLY` or `READ WRITE` option, the transaction will be started as a read-write transaction by default. + +#### Examples + +An example of building statement objects for `START TRANSACTION` is as follows: + +```java +// Start a transaction. +StartTransactionStatement statement1 = StatementBuilder.startTransaction().build(); + +// Start a transaction in read-only mode. +StartTransactionStatement statement2 = StatementBuilder.startTransaction().readOnly().build(); + +// Start a transaction in read-write mode. +StartTransactionStatement statement3 = StatementBuilder.startTransaction().readWrite().build(); +``` + +### JOIN + +The `JOIN` command joins a transaction associated with the specified transaction ID. + +#### Grammar + +```sql +JOIN +``` + +#### Examples + +An example of `JOIN` is as follows: + +```sql +-- Join a transaction +JOIN 'id'; +``` + +An example of building statement objects for `JOIN` is as follows: + +```java +// Join a transaction +JoinStatement statement = StatementBuilder.join("id").build(); +``` + +### PREPARE + +The `PREPARE` command prepares the current transaction. + +#### Grammar + +```sql +PREPARE +``` + +#### Examples + +An example of building statement objects for `PREPARE` is as follows: + +```java +// Prepare the current transaction +PrepareStatement statement = StatementBuilder.prepare().build(); +``` + +### VALIDATE + +The `VALIDATE` command validates the current transaction. + +#### Grammar + +```sql +VALIDATE +``` + +#### Examples + +An example of building statement objects for `VALIDATE` is as follows: + +```java +// Validate the current transaction +ValidateStatement statement = StatementBuilder.validate().build(); +``` + +### COMMIT + +The `COMMIT` command commits the current transaction. + +#### Grammar + +```sql +COMMIT +``` + +#### Examples + +An example of building statement objects for `COMMIT` is as follows: + +```java +// Commit the current transaction +CommitStatement statement = StatementBuilder.commit().build(); +``` + +### ROLLBACK + +The `ROLLBACK` command rolls back the current transaction. + +#### Grammar + +```sql +ROLLBACK +``` + +#### Examples + +An example of building statement objects for `ROLLBACK` is as follows: + +```java +// Rollback the current transaction +RollbackStatement statement = StatementBuilder.rollback().build(); +``` + +### ABORT + +The `ABORT` command rolls back the current transaction. This command is an alias of `ROLLBACK`. + +#### Grammar + +```sql +ABORT +``` + +#### Examples + +An example of building statement objects for `ABORT` is as follows: + +```java +// Abort the current transaction. +AbortStatement statement = StatementBuilder.abort().build(); +``` + +### SET MODE + +The `SET MODE` command switches the current transaction mode. + +#### Grammar + +```sql +SET MODE transaction_mode + +transaction_mode: TRANSACTION | TWO_PHASE_COMMIT_TRANSACTION +``` + +#### Examples + +An example of `SET MODE` is as follows: + +```sql +-- Switch the current transaction mode to Two-phase Commit Transaction +SET MODE TWO_PHASE_COMMIT_TRANSACTION; +``` + +An example of building statement objects for `SET MODE` is as follows: + +```java +// Switch the current transaction mode to Two-phase Commit Transaction +SetModeStatement statement = + StatementBuilder.setMode(TransactionMode.TWO_PHASE_COMMIT_TRANSACTION).build(); +``` + +### SHOW NAMESPACES + +The `SHOW NAMESPACES` command shows namespace names. + +:::note + +This command extracts the namespace names of user tables dynamically. As a result, only namespaces that contain tables are returned. Starting from ScalarDB 4.0, we plan to improve the design to remove this limitation. + +::: + +This command returns the following column: + +- `namespaceName`: `TEXT` - the namespace name + +#### Grammar + +```sql +SHOW NAMESPACES +``` + +#### Examples + +An example of building statement objects for `SHOW NAMESPACES` is as follows: + +```java +// Show namespace names. +ShowNamespacesStatement statement = StatementBuilder.showNamespaces().build(); +``` + +### SHOW TABLES + +The `SHOW TABLES` command shows table names in a namespace. If a namespace name is omitted, the default namespace will be used. + +This command returns the following column: + +- `tableName`: `TEXT` - a table name + +#### Grammar + +```sql +SHOW TABLES [FROM ] +``` + +#### Examples + +Examples of `SHOW TABLES` is as follows: + +```sql +-- Show table names in the default namespace +SHOW TABLES; + +-- Show table names in a namespace "ns" +SHOW TABLES FROM ns; +``` + +Examples of building statement objects for `SHOW TABLES` is as follows: + +```java +// Show table names in the default namespace +ShowTablesStatement statement1 = StatementBuilder.showTables().build(); + +// Show table names in a namespace "ns" +ShowTablesStatement statement2 = StatementBuilder.showTables().from("ns").build(); +``` + +### DESCRIBE + +The `DESCRIBE` command returns column metadata for the specified table. + +This command returns the following columns: + +- `columnName`: `TEXT` - a table name +- `type`: `TEXT` - a type name +- `isPrimaryKey`: `BOOLEAN` - whether it's a column part of primary key +- `isPartitionKey`: `BOOLEAN` - whether it's a column part of partition key +- `isClusteringKey`: `BOOLEAN` - whether it's a column part of clustering key +- `clusteringOrder`: `TEXT` - a clustering order +- `isIndexed`: `BOOLEAN` - whether it's an indexed column + +#### Grammar + +```sql +DESCRIBE [.]
+ +DESC [.]
+``` + +#### Examples + +Examples of `DESCRIBE` is as follows: + +```sql +-- Returns column metadata for "ns.tbl" +DESCRIBE ns.tbl; + +-- Returns column metadata for "tbl" +DESC tbl; +``` + +Examples of building statement objects for `DESCRIBE` is as follows: + +```java +// Returns column metadata for "ns.tbl" +DescribeStatement statement1 = StatementBuilder.describe("ns", "tbl").build(); + +// Returns column metadata for "tbl" +DescribeStatement statement2 = StatementBuilder.describe("tbl").build(); +``` + +### SUSPEND + +The `SUSPEND` command suspends the ongoing transaction in the current session. + +#### Grammar + +```sql +SUSPEND +``` + +#### Examples + +Examples of building statement objects for `SUSPEND` is as follows: + +```java +// Suspend the ongonig transaction in the current session +SuspendStatement statement = StatementBuilder.suspend().build(); +``` + +### RESUME + +The `RESUME` command resumes the transaction associated with the specified transaction ID in the current session. + +#### Grammar + +```sql +RESUME +``` + +#### Examples + +An example of `RESUME` is as follows: + +```sql +-- Resume a transaction +RESUME 'id'; +``` + +An example of building statement objects for `RESUME` is as follows: + +```java +// Resume a transaction +ResumeStatement statement = StatementBuilder.resume("id").build(); +``` + +### SHOW USERS + +The `SHOW USERS` command shows usernames and user attributes. If you are a superuser, you can see all users. If you are a normal user, you can see only your own username and attributes. + +This command returns the following columns: + +- `username`: `TEXT` - a username +- `isSuperuser`: `BOOLEAN` - whether the user is a superuser + +#### Grammar + +```sql +SHOW USERS +``` + +#### Examples + +An example of `SHOW USERS` is as follows: + +```sql +-- Show usernames and user attributes +SHOW USERS; +``` + +An example of building statement objects for `SHOW USERS` is as follows: + +```java +// Show usernames and user attributes +ShowUsersStatement statement = StatementBuilder.showUsers().build(); +``` + +### SHOW GRANTS + +The `SHOW GRANTS` command shows the privileges granted to the current user or the specified user. + +This command returns the following columns: + +- `name`: `TEXT` - a namespace name or a table name, depending on the type +- `type`: `TEXT` - the type of the object (either `NAMESPACE` or `TABLE`) +- `privilege`: `TEXT` - a privilege + +#### Grammar + +```sql +-- Show privileges granted to the current user +SHOW GRANTS + +-- Show privileges granted to the specified user +SHOW GRANTS FOR [USER] +``` + +#### Examples + +Examples of `SHOW GRANTS` are as follows: + +```sql +-- Show privileges granted to the current user +SHOW GRANTS; + +-- Show privileges granted to user1 +SHOW GRANTS FOR user1; +``` + +Examples of building statement objects for `SHOW GRANTS` are as follows: + +```java +// Show privileges granted to the current user +ShowGrantsStatement statement1 = StatementBuilder.showGrants().build(); + +// Show privileges granted to user1 +ShowGrantsStatement statement2 = StatementBuilder.showGrants().forUser("user1").build(); +``` + +## Literal + +Literal and column values are synonymous and refer to a fixed data value used when writing SQL statements. For example, `1`, `'abc'`, `1.23`, and `'2024-05-19'` are literals. + +### Text + +A text literal is a sequence of characters enclosed in single quotes `'`, such as `'abc'` and `'abc def'`. + +### Numeric + +Numeric literals include exact-value (INTEGER and BIGINT) and approximate-value (FLOAT and DOUBLE) literals. + +An exact-value literal is a sequence of digits, such as `123` and `-5`. + +An approximate-value literal is a sequence of digits with a decimal point, such as `4.754` and `-1.2`. + +### Date and time + +Date and time literals are text literals that follow a specific format to represents DATE, TIME, TIMESTAMP, and TIMESTAMPTZ values. + +| ScalarDB type | Format | Note | Example | +|---------------|-------------------------------------|------------------------------------------------------------------|----------------------------------------------------------------------------------| +| DATE | **'YYYY-MM-DD'** | | `'2024-05-19'` | +| TIME | **'HH:MM:[SS[.FFFFFF]]'** | Second and fractional second (up to 1 microsecond) are optional. | `'12:34'`, `'12:34:56'`, `'12:34:56.789123'` | +| TIMESTAMP | **'YYYY-MM-DD HH:MM:[SS[.FFF]]'** | Second and fractional second (up to 1 millisecond) are optional. | `'2024-05-19 12:34'`, `'2024-05-19 12:34:56'`, `'2024-05-19 12:34:56.789'` | +| TIMESTAMPTZ | **'YYYY-MM-DD HH:MM:[SS[.FFF]] Z'** | Second and fractional second (up to 1 millisecond) are optional. | '`2024-05-19 12:34 Z'`, `'2024-05-19 12:34:56 Z'`, `'2024-05-19 12:34:56.789 Z'` | diff --git a/versioned_docs/version-3.X/scalardb-sql/images/spring_data_ingegration_overall_arch.png b/versioned_docs/version-3.X/scalardb-sql/images/spring_data_ingegration_overall_arch.png new file mode 100644 index 00000000..67b52a44 Binary files /dev/null and b/versioned_docs/version-3.X/scalardb-sql/images/spring_data_ingegration_overall_arch.png differ diff --git a/versioned_docs/version-3.X/scalardb-sql/index.mdx b/versioned_docs/version-3.X/scalardb-sql/index.mdx new file mode 100644 index 00000000..4de31391 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-sql/index.mdx @@ -0,0 +1,37 @@ +--- +tags: + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# ScalarDB SQL Overview + +ScalarDB SQL is an interface layer that allows client applications to communicate with ScalarDB Cluster by using SQL. + +:::note + +ScalarDB SQL is not fully compatible with standard SQL, but it offers a large subset of the SQL language. + +::: + +## Types of SQL interfaces + +ScalarDB SQL has three types of SQL interfaces. + +### JDBC + +The JDBC interface lets you connect to ScalarDB Cluster by using the standard JDBC API. This is useful for applications that already use JDBC. + +For details on how to set up and use the JDBC interface, see the [ScalarDB JDBC Guide](./jdbc-guide.mdx). + +### SQL API + +The SQL API lets you connect to ScalarDB Cluster by using the proprietary and modern Java SQL API. This is useful for applications that do not need to rely on the JDBC interface. + +For details on how to set up and use the SQL API, see the [ScalarDB SQL API Guide](./sql-api-guide.mdx). + +### Spring Data JDBC + +The Spring Data JDBC interface lets you interact with ScalarDB Cluster via Spring Data JDBC repositories and entities. This is useful for applications that already use Spring Data or when you want to integrate ScalarDB Cluster into Spring applications. + +For details on how to set up and use the Sprign Data JDBC interface, see the [Guide of Spring Data JDBC for ScalarDB](./spring-data-guide.mdx). diff --git a/versioned_docs/version-3.X/scalardb-sql/jdbc-guide.mdx b/versioned_docs/version-3.X/scalardb-sql/jdbc-guide.mdx new file mode 100644 index 00000000..4e285c32 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-sql/jdbc-guide.mdx @@ -0,0 +1,224 @@ +--- +tags: + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# ScalarDB JDBC Guide + +The usage of ScalarDB JDBC basically follows [Java JDBC API](https://docs.oracle.com/javase/8/docs/technotes/guides/jdbc/). +This guide describes several important topics that are specific to ScalarDB JDBC. + +## Add ScalarDB JDBC driver to your project + +To add the dependencies for the ScalarDB JDBC driver by using Gradle, use the following, replacing `` with the versions of the ScalarDB JDBC driver and the related library, respectively, that you are using: + +```gradle +dependencies { + implementation 'com.scalar-labs:scalardb-sql-jdbc:' + implementation 'com.scalar-labs:scalardb-cluster-java-client-sdk:' +} +``` + +To add the dependencies by using Maven, use the following, replacing `...` with the version of the ScalarDB JDBC driver that you are using: + +```xml + + + com.scalar-labs + scalardb-sql-jdbc + ... + + + com.scalar-labs + scalardb-cluster-java-client-sdk + ... + + +``` + +## JDBC connection URL + +The JDBC connection URL format of ScalarDB JDBC is as follows: + +```console +jdbc:scalardb:?=&=&... +``` + +For example: + +Only specify configuration file path: + +```console +jdbc:scalardb:/path/to/database.properties +``` + +Only specify properties: + +```console +jdbc:scalardb:?scalar.db.contact_points=localhost&scalar.db.username=cassandra&scalar.db.password=cassandra&scalar.db.storage=cassandra +``` + +Specify configuration file path and properties: + +```console +jdbc:scalardb:/path/to/database.properties?scalar.db.metadata.cache_expiration_time_secs=0 +``` + +## Configurations for ScalarDB JDBC + +Please see [ScalarDB Cluster SQL client configurations](../scalardb-cluster/developer-guide-for-scalardb-cluster-with-java-api.mdx#scalardb-cluster-sql-client-configurations) for details on the configurations. + +In addition, the ScalarDB JDBC specific configurations are as follows: + +| name | description | default | +|---------------------------------------------------------------------|-----------------------------------------------------------------------------|---------| +| scalar.db.sql.jdbc.default_auto_commit | The default auto-commit mode for connections. | true | +| scalar.db.sql.jdbc.default_read_only | The default read-only state for connections. | false | +| scalar.db.sql.jdbc.sql_session_factory_cache.expiration_time_millis | The expiration time in milliseconds for the cache of SQL session factories. | 10000 | + +## Data type mapping between ScalarDB and JDBC + +Since ScalarDB doesn't support all the data types defined in JDBC, the following explains the data type mapping between ScalarDB and JDBC. + +The data type mapping between ScalarDB and JDBC is as follows: + +| ScalarDB Type | JDBC (Java) Type | +|---------------|-------------------------| +| BOOLEAN | boolean or Boolean | +| INT | int or Integer | +| BIGINT | long or Long | +| FLOAT | float or Float | +| DOUBLE | double or Double | +| TEXT | String | +| BLOB | byte[] | +| DATE | java.time.LocalDate | +| TIME | java.time.LocalTime | +| TIMESTAMP | java.time.LocalDateTime | +| TIMESTAMPTZ | java.time.Instant | + +How to get the data from a `java.sql.ResultSet` object for each data type is as follows: + +```java +try (ResultSet resultSet = ...) { + resultSet.next(); + + // Get a BOOLEAN value of a column + boolean booleanValue = resultSet.getBoolean(""); + + // Get an INT value of a column + int intValue = resultSet.getInt(""); + + // Get a BIGINT value of a column + long bigIntValue = resultSet.getLong(""); + + // Get a FLOAT value of a column + float floatValue = resultSet.getFloat(""); + + // Get a DOUBLE value of a column + double doubleValue = resultSet.getDouble(""); + + // Get a TEXT value of a column + String textValue = resultSet.getString(""); + + // Get a BLOB value of a column + byte[] blobValue = resultSet.getBytes(""); + + // Get a DATE value of a column + LocalDate dateValue = resultSet.getObject("", LocalDate.class); + + // Get a TIME value of a column + LocalTime timeValue = resultSet.getObject("", LocalTime.class); + + // Get a TIMESTAMP value of a column + LocalDateTime timestampValue = resultSet.getObject("", LocalDateTime.class); + + // Get a TIMESTAMPTZ value of a column + Instant timestampTZValue = resultSet.getObject("", Instant.class); +} +``` + +How to set the data as a parameter for each data type for a `java.sql.PreparedStatement` object is as follows: + +```java +try (PreparedStatement preparedStatement = ...) { + // Set a BOOLEAN value as parameter + preparedStatement.setBoolean(1, ); + + // Set an INT value as parameter + preparedStatement.setInt(2, ); + + // Set a BIGINT value as parameter + preparedStatement.setLong(3, ); + + // Set a FLOAT value as parameter + preparedStatement.setFloat(4, ); + + // Set a DOUBLE value as parameter + preparedStatement.setDouble(5, ); + + // Set a TEXT value as parameter + preparedStatement.setString(7, ""); + + // Set a BLOB value as parameter + preparedStatement.setBytes(8, ); + + //Set a DATE value as parameter + preparedStatement.setObject(9, ); + + //Set a TIME value as parameter + preparedStatement.setObject(10, ); + + //Set a TIMESTAMP value as parameter + preparedStatement.setObject(11, ); + + //Set a TIMESTAMPTZ value as parameter + preparedStatement.setObject(12, ); + + preparedStatement.execute(); +} +``` + +## Handle SQLException + +The exception handling is basically the same as ScalarDB SQL API as follows: + +```java +// If you execute multiple statements in a transaction, you need to set auto-commit to false. +connection.setAutoCommit(false); + +try { + // Execute statements (SELECT/INSERT/UPDATE/DELETE) in the transaction + ... + + // Commit the transaction + connection.commit(); +} catch (SQLException e) { + if (e.getErrorCode() == 301) { + // The error code 301 indicates that you catch `UnknownTransactionStatusException`. + // If you catch `UnknownTransactionStatusException`, it indicates that the status of the + // transaction, whether it has succeeded or not, is unknown. In such a case, you need to check + // if the transaction is committed successfully or not and retry it if it failed. How to + // identify a transaction status is delegated to users + } else { + // For other cases, you can try retrying the transaction + + // Rollback the transaction + connection.rollback(); + + // The cause of the exception can be `TransactionRetryableException` or the other + // exceptions. For `TransactionRetryableException`, you can basically retry the transaction. + // However, for the other exceptions, the transaction may still fail if the cause of the + // exception is nontransient. For such a case, you need to limit the number of retries and + // give up retrying + } +} +``` + +Please see also [ScalarDB SQL API Guide](sql-api-guide.mdx) for more details on exception handling. + +## References + +- [Java JDBC API](https://docs.oracle.com/javase/8/docs/technotes/guides/jdbc/) +- [ScalarDB SQL API Guide](sql-api-guide.mdx) +- [Javadoc for ScalarDB JDBC](https://javadoc.io/doc/com.scalar-labs/scalardb-sql-jdbc/3.16.0/index.html) diff --git a/versioned_docs/version-3.X/scalardb-sql/migration-guide.mdx b/versioned_docs/version-3.X/scalardb-sql/migration-guide.mdx new file mode 100644 index 00000000..b16ebe4b --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-sql/migration-guide.mdx @@ -0,0 +1,115 @@ +--- +tags: + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# How to Migrate Your Applications and Databases into a ScalarDB-Based Environment + +This guide describes how to migrate your existing applications and relational databases into ScalarDB-based applications and ScalarDB-managed databases, respectively. + +## Target audience + +The target audiences for this guide are application developers and database administrators. The purpose of this guide is to help you understand how to migrate your existing applications and databases and in what conditions. + +## What you will learn + +- Requirements for the migration +- Steps to migrate your application +- Changes to make in your application for the migration + +## Steps to migrate your application + +```mermaid +flowchart LR +subgraph AM[After migration] + direction TB + C[Application - updated] -->|ScalarDB SQL| D[ScalarDB] --> E[Relational/JDBC databases - schema updated] +end +subgraph BM[Before migration] + direction TB + A[Application] ---->|SQL| B[Relational/JDBC databases] +end +BM===>AM +``` + +1. Verify the items in the checklist. + - See [Migration checklist](#migration-checklist) to confirm your database is migratable. +2. Migrate your application (if necessary). + - ScalarDB provides selection, projection, and join operations with dedicated SQL grammar. Thus, some SQL statements in your application might have to be changed for ScalarDB SQL, for example, at a grammar level or a logic level like aggregation processing. + - For details, see [How to migrate your application](#how-to-migrate-your-application). +3. Back up your database. + - Although ScalarDB Schema Loader, which you will use to import your database, only changes the metadata of your database when importing it to ScalarDB, you must back it up to avoid unexpected accidents. + - Follow the administration guide of your database. +4. Set up a ScalarDB environment. + - Prepare a configuration file so that ScalarDB can access target databases. + - For details about ScalarDB configurations, see [ScalarDB Configurations](../configurations.mdx). +5. Import your database to ScalarDB. + - Prepare an import schema file that defines target schemas and tables. The schemas and tables will be mapped to ScalarDB namespaces and tables, respectively. Note that "schema" is a synonym for "database" in some database systems. + - Run the ScalarDB Schema Loader with the import option, the ScalarDB configuration file that you created, and the schema file that you prepared. + - For details on how to use Schema Loader, see [Run Schema Loader for importing existing tables](../schema-loader-import.mdx#run-schema-loader-for-importing-existing-tables). +6. Switch your application and check the behavior. + - Now, you can switch your application to the ScalarDB-based application. + +## Migration checklist + +Before starting the migration, check the following questions. If the answer to any of these questions is "No", you must address them before proceeding with the migration. + +- Are your target database and version one of the [supported relational databases (called JDBC databases in ScalarDB) and versions](../requirements.mdx#relational-databases)? +- Do you have a fully privileged account that can manage the target database? For details, see [the general requirements](../database-configurations.mdx#general-requirements). +- Do all target tables have primary keys? +- Is the data type of each column supported in ScalarDB? For supported data types and how they are mapped to ScalarDB data types, see [Data-type mapping from JDBC databases to ScalarDB](../schema-loader-import.mdx#data-type-mapping-from-jdbc-databases-to-scalardb). +- Do the functionality and grammar of the queries in your application comply with the [ScalarDB SQL specifications](./grammar.mdx)? Or, for non-compliant queries, can you re-write them for ScalarDB? For examples of re-writes, see [How to migrate your application](#how-to-migrate-your-application). +- After migrating your applications and databases into ScalarDB applications and ScalarDB-managed databases, respectively, can you stop accessing the databases directly? In other words, is it acceptable for you to always access the databases through ScalarDB? + +## How to migrate your application + +Depending on your application environment, you may need to migrate your application in the following three aspects: + +- Change connection settings. +- Modify SQL statements based on the ScalarDB SQL grammar. +- Modify application logic if there is no available SQL modification workaround. + +### Change connection settings + +If your application is based on Java, you can use the ScalarDB JDBC driver when migrating. For details on how to add dependencies for the ScalarDB JDBC driver and rewrite the connection URL, see the [ScalarDB JDBC Guide](./jdbc-guide.mdx). + +If your application is not based on Java, you can connect ScalarDB and issue SQL via gRPC. For details, see [ScalarDB Cluster SQL gRPC API Guide](../scalardb-cluster/scalardb-cluster-sql-grpc-api-guide.mdx). + +### Modify SQL statements + +You may need to change the SQL statements in your application due to the differences in SQL grammar. Typical examples are as follows. For more details, see [ScalarDB SQL Grammar](./grammar.mdx). + +- `JOIN` queries + - ScalarDB supports only `JOIN` queries in the style of writing the table to be joined and the condition in the `FROM` clause. + - The `JOIN` condition and filtering also have a few limitations. + - You may need to rewrite the queries based on the above. You can choose application-level modification if your SQL queries are not compliant with the ScalarDB specifications. +- `WHERE` clause + - In ScalarDB, predicates must be an OR-wise of `AND` predicate lists (known as disjunctive normal form or DNF) or an AND-wise of `OR` predicate lists (known as conjunctive normal form or CNF). Thus, you may have to change the `WHERE` clause, but note that an arbitrary form of predicates can be changed to either DNF or CNF. + - Similarly, if you use `IN` clauses, you will need to change them to either DNF or CNF. For `IN` clauses with sub-queries, see [Modify application logic](#modify-application-logic). + - ScalarDB adopts a specification similar to that of the `LIKE` operator and the escape sequence of PostgreSQL and MySQL. If your database is neither PostgreSQL nor MySQL, you may need to change predicates with the `LIKE` operator. + +### Modify application logic + +Although ScalarDB SQL does not provide some functionalities, such as aggregate queries and sub-queries, those queries can be modified to application-level implementations. Typical modification techniques are as follows: + +- Aggregate queries + - For simple aggregate queries such as `count()` and `sum()` without the `GROUP BY` clause, you can use `SELECT` for the target records and then count the number of records or calculate the sum by using the results. + - For `GROUP BY` aggregate queries, first use `SELECT` for all target records without the `GROUP BY` clause. Then, put result records into a multi-map data structure while categorizing them based on the columns specified in the `GROUP BY` clause, which should be used as keys of the multi-map. Finally, aggregate records for each key in the multi-map. For the multi-map, you can use libraries such as [Guava](https://github.com/google/guava). +- Sub-queries + - For sub-queries in the `IN` clause, first use `SELECT` for the records specified in the sub-queries, then add result values as `OR` predicates in the `WHERE` clause. + - For other sub-queries, basically, you need to use `SELECT` for the records for each query, then join or filter results records in your application. +- Read-modify-write by using a single update query + - `UPDATE` queries may often have an expression like an increment or a decrement, for example, `UPDATE table SET a = a + 1 WHERE ...`. In ScalarDB, you need to use `SELECT` for a target record and then set the incremented value as a constant in a single transaction, just like `UPDATE table SET a = 5 WHERE ...`. + +## Limitations + +Due to the difference in data types, ScalarDB will throw an error when writing data larger than the maximum size of the column in the underlying database, even if the size is acceptable for the ScalarDB data type. Conversely, in a few types, the data in the underlying database may be larger than the maximum size in ScalarDB. For details, see [Data-type mapping from JDBC databases to ScalarDB](../schema-loader-import.mdx#data-type-mapping-from-jdbc-databases-to-scalardb). + +## References + +- [Supported Databases](../requirements.mdx#databases) +- [ScalarDB SQL API Guide](./sql-api-guide.mdx) +- [ScalarDB JDBC Guide](./jdbc-guide.mdx) +- [ScalarDB SQL Grammar](./grammar.mdx) +- [Importing Existing Tables to ScalarDB by Using ScalarDB Schema Loader](../schema-loader-import.mdx) diff --git a/versioned_docs/version-3.X/scalardb-sql/scalardb-sql-status-codes.mdx b/versioned_docs/version-3.X/scalardb-sql/scalardb-sql-status-codes.mdx new file mode 100644 index 00000000..62cfd0f6 --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-sql/scalardb-sql-status-codes.mdx @@ -0,0 +1,667 @@ +--- +tags: + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# ScalarDB SQL Error Codes + +This page provides a list of error codes in ScalarDB SQL. + +## Error code classes and descriptions + +| Class | Description | +|:---------------|:-----------------------------------| +| `DB-SQL-1xxxx` | Errors for the user error category | + +## `DB-SQL-1xxxx` status codes + +The following are status codes and messages for the user error category. + +### `DB-SQL-10000` + +**Message** + +```markdown +The namespace does not exist. Namespace: %s +``` + +### `DB-SQL-10001` + +**Message** + +```markdown +The table does not exist. Table: %s +``` + +### `DB-SQL-10002` + +**Message** + +```markdown +The column %s does not exist +``` + +### `DB-SQL-10003` + +**Message** + +```markdown +The column does not exist. Table: %s; Column: %s +``` + +### `DB-SQL-10004` + +**Message** + +```markdown +The column index is out of bounds. Index: %d; Size: %d +``` + +### `DB-SQL-10005` + +**Message** + +```markdown +A positional bind marker is not allowed when binding named values +``` + +### `DB-SQL-10006` + +**Message** + +```markdown +A named bind marker is not allowed when binding positional values +``` + +### `DB-SQL-10007` + +**Message** + +```markdown +Cannot convert BLOB values to SQL. Please use a bind marker for a BLOB value and bind it separately +``` + +### `DB-SQL-10008` + +**Message** + +```markdown +No namespace name has been specified. Set a default namespace name, or explicitly specify a namespace name +``` + +### `DB-SQL-10009` + +**Message** + +```markdown +Zero bytes may not occur in string parameters +``` + +### `DB-SQL-10010` + +**Message** + +```markdown +Mixing positional values and named values is not allowed +``` + +### `DB-SQL-10011` + +**Message** + +```markdown +Preparing a transaction is supported only in two-phase commit transaction mode +``` + +### `DB-SQL-10012` + +**Message** + +```markdown +Validating a transaction is supported only in two-phase commit transaction mode +``` + +### `DB-SQL-10013` + +**Message** + +```markdown +The previous transaction is still in progress. Commit, roll back, or suspend the previous transaction first +``` + +### `DB-SQL-10014` + +**Message** + +```markdown +This SQL session has already been closed +``` + +### `DB-SQL-10015` + +**Message** + +```markdown +A transaction has not begun +``` + +### `DB-SQL-10016` + +**Message** + +```markdown +The type %s is not supported +``` + +### `DB-SQL-10017` + +**Message** + +```markdown +No connection mode implementations are found. Please add a connection mode implementation to the classpath +``` + +### `DB-SQL-10018` + +**Message** + +```markdown +The connection mode is not specified, but multiple connection mode implementations are found. Specify one of the following connection modes: %s +``` + +### `DB-SQL-10019` + +**Message** + +```markdown +The connection mode '%s' is not found. Specify one of the following connection modes: %s +``` + +### `DB-SQL-10020` + +**Message** + +```markdown +Access denied: You need the %s privilege on the namespace %s to execute this operation +``` + +### `DB-SQL-10021` + +**Message** + +```markdown +Access denied: You need the %s privilege on the table %s to execute this operation +``` + +### `DB-SQL-10022` + +**Message** + +```markdown +Access denied: You can't grant the %s privilege because you don't have the same privilege on the table %s +``` + +### `DB-SQL-10023` + +**Message** + +```markdown +Access denied: You can't grant the %s privilege because you don't have the same privilege on the namespace %s +``` + +### `DB-SQL-10024` + +**Message** + +```markdown +Access denied: You can't revoke the %s privilege because you don't have the same privilege on the table %s +``` + +### `DB-SQL-10025` + +**Message** + +```markdown +Access denied: You can't revoke the %s privilege because you don't have the same privilege on the namespace %s +``` + +### `DB-SQL-10026` + +**Message** + +```markdown +Syntax error. Line %d:%d %s +``` + +### `DB-SQL-10027` + +**Message** + +```markdown +Syntax error. Multiple PRIMARY KEYs specified (exactly one required) +``` + +### `DB-SQL-10028` + +**Message** + +```markdown +Cannot grant the INSERT privilege if the user doesn't have the UPDATE privilege +``` + +### `DB-SQL-10029` + +**Message** + +```markdown +Cannot grant the UPDATE privilege if the user doesn't have the INSERT privilege +``` + +### `DB-SQL-10030` + +**Message** + +```markdown +Cannot grant the UPDATE privilege if the user doesn't have the SELECT privilege +``` + +### `DB-SQL-10031` + +**Message** + +```markdown +Cannot grant the DELETE privilege if the user doesn't have the SELECT privilege +``` + +### `DB-SQL-10032` + +**Message** + +```markdown +Cannot revoke the SELECT privilege if the user has the UPDATE privilege +``` + +### `DB-SQL-10033` + +**Message** + +```markdown +Cannot revoke the SELECT privilege if the user has the DELETE privilege +``` + +### `DB-SQL-10034` + +**Message** + +```markdown +Cannot revoke the INSERT privilege if the user has the UPDATE privilege +``` + +### `DB-SQL-10035` + +**Message** + +```markdown +Cannot revoke the UPDATE privilege if the user has the INSERT privilege +``` + +### `DB-SQL-10036` + +**Message** + +```markdown +A non-clustering-key column is specified in the CLUSTERING ORDER directive. Column: %s +``` + +### `DB-SQL-10037` + +**Message** + +```markdown +The order of the columns in the CLUSTERING ORDER directive must match the order for the clustering key (%s must appear before %s) +``` + +### `DB-SQL-10038` + +**Message** + +```markdown +The CLUSTERING ORDER is missing for the column %s +``` + +### `DB-SQL-10039` + +**Message** + +```markdown +Empty SQL is specified +``` + +### `DB-SQL-10040` + +**Message** + +```markdown +Multiple SQLs are not allowed +``` + +### `DB-SQL-10041` + +**Message** + +```markdown +The column %s is ambiguous +``` + +### `DB-SQL-10042` + +**Message** + +```markdown +The column %s cannot be specified in the %s clause. Only the columns in the table %s can be specified in the %s clause +``` + +### `DB-SQL-10043` + +**Message** + +```markdown +An unbound bind marker is still in the escape character of the LIKE predicate for the column %s +``` + +### `DB-SQL-10044` + +**Message** + +```markdown +The escape character must be a TEXT value for the LIKE predicate for the column %s +``` + +### `DB-SQL-10045` + +**Message** + +```markdown +The value of the predicate must not be null unless the operator is 'IS NULL' or 'IS NOT NULL'. Predicate: %s +``` + +### `DB-SQL-10046` + +**Message** + +```markdown +An unbound bind marker is still in the LIMIT clause +``` + +### `DB-SQL-10047` + +**Message** + +```markdown +The LIMIT must be an INT value +``` + +### `DB-SQL-10048` + +**Message** + +```markdown +Unmatched column names or values +``` + +### `DB-SQL-10049` + +**Message** + +```markdown +The column %s is specified twice +``` + +### `DB-SQL-10050` + +**Message** + +```markdown +All primary key columns must be specified in the INSERT or UPSERT statement +``` + +### `DB-SQL-10051` + +**Message** + +```markdown +An unbound bind marker is still in the value of the column %s +``` + +### `DB-SQL-10052` + +**Message** + +```markdown +Unmatched column type. The type of the column %s should be %s, but a boolean value (BOOLEAN) is specified +``` + +### `DB-SQL-10053` + +**Message** + +```markdown +Unmatched column type. The type of the column %s should be %s, but a decimal number (INT or BIGINT) is specified +``` + +### `DB-SQL-10054` + +**Message** + +```markdown +Unmatched column type. The type of the column %s should be %s, but a floating point number (FLOAT or DOUBLE) is specified +``` + +### `DB-SQL-10055` + +**Message** + +```markdown +Unmatched column type. The type of the column %s should be %s, but a string (TEXT, DATE, TIME, TIMESTAMP, or TIMESTAMPTZ) is specified +``` + +### `DB-SQL-10056` + +**Message** + +```markdown +Unmatched column type. The type of the column %s should be %s, but a BOOLEAN value is specified +``` + +### `DB-SQL-10057` + +**Message** + +```markdown +Unmatched column type. The type of the column %s should be %s, but an INT value is specified +``` + +### `DB-SQL-10058` + +**Message** + +```markdown +Unmatched column type. The type of the column %s should be %s, but a BIGINT value is specified +``` + +### `DB-SQL-10059` + +**Message** + +```markdown +Unmatched column type. The type of the column %s should be %s, but a FLOAT value is specified +``` + +### `DB-SQL-10060` + +**Message** + +```markdown +Unmatched column type. The type of the column %s should be %s, but a DOUBLE value is specified +``` + +### `DB-SQL-10061` + +**Message** + +```markdown +Unmatched column type. The type of the column %s should be %s, but a TEXT value is specified +``` + +### `DB-SQL-10062` + +**Message** + +```markdown +Unmatched column type. The type of the column %s should be %s, but a BLOB value is specified +``` + +### `DB-SQL-10063` + +**Message** + +```markdown +RIGHT OUTER JOIN can only be specified as the first join +``` + +### `DB-SQL-10064` + +**Message** + +```markdown +The JOIN predicate is not specified properly. Predicate: %s +``` + +### `DB-SQL-10065` + +**Message** + +```markdown +The data types of the columns in the JOIN predicate do not match. Predicate: %s +``` + +### `DB-SQL-10066` + +**Message** + +```markdown +The column %s is specified twice in the JOIN predicates. Predicates: %s +``` + +### `DB-SQL-10067` + +**Message** + +```markdown +Either all primary key columns or an indexed column for the table %s must be specified in the JOIN predicates. Predicates: %s +``` + +### `DB-SQL-10068` + +**Message** + +```markdown +Cannot issue mutation DML SQLs such as INSERT, UPDATE or DELETE with executeQuery() +``` + +### `DB-SQL-10069` + +**Message** + +```markdown +Cannot issue SELECT SQLs with executeUpdate() +``` + +### `DB-SQL-10070` + +**Message** + +```markdown +The TWO_PHASE_COMMIT_TRANSACTION mode is not supported in the current transaction manager +``` + +### `DB-SQL-10071` + +**Message** + +```markdown +The encrypted column %s is not allowed in the %s clause +``` + +### `DB-SQL-10072` + +**Message** + +```markdown +The user %s does not exist +``` + +### `DB-SQL-10073` + +**Message** + +```markdown +Unmatched column type. The type of the column %s should be %s, but a DATE value is specified +``` + +### `DB-SQL-10074` + +**Message** + +```markdown +Unmatched column type. The type of the column %s should be %s, but a TIME value is specified +``` + +### `DB-SQL-10075` + +**Message** + +```markdown +Unmatched column type. The type of the column %s should be %s, but a TIMESTAMP value is specified +``` + +### `DB-SQL-10076` + +**Message** + +```markdown +Unmatched column type. The type of the column %s should be %s, but a TIMESTAMPTZ value is specified +``` + +### `DB-SQL-10077` + +**Message** + +```markdown +The policy %s does not exist +``` + +### `DB-SQL-10078` + +**Message** + +```markdown +Beginning a transaction in read-only mode is not supported in two-phase commit transaction mode +``` + +### `DB-SQL-10079` + +**Message** + +```markdown +Starting a transaction in read-only mode is not supported in two-phase commit transaction mode +``` + +### `DB-SQL-10080` + +**Message** + +```markdown +Cannot change read-only mode while a transaction is in progress +``` diff --git a/versioned_docs/version-3.X/scalardb-sql/spring-data-guide.mdx b/versioned_docs/version-3.X/scalardb-sql/spring-data-guide.mdx new file mode 100644 index 00000000..040e412a --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-sql/spring-data-guide.mdx @@ -0,0 +1,823 @@ +--- +tags: + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Guide of Spring Data JDBC for ScalarDB + +Directly using the ScalarDB API may be difficult because you need to write a lot of code and consider how and when to call the APIs (e.g., `rollback()` and `commit()`) for transactions. Since we assume most ScalarDB users develop their applications in Java, you can take advantage of the Spring Framework, which is one of the most popular application frameworks for developing in Java. By using Spring Data JDBC for ScalarDB, you can streamline development by using a familiar framework. + +![Rough overall architecture of Spring Data JDBC for ScalarDB](images/spring_data_ingegration_overall_arch.png) + +The usage of Spring Data JDBC for ScalarDB basically follows [Spring Data JDBC - Reference Documentation](https://docs.spring.io/spring-data/jdbc/docs/3.0.x/reference/html/). +This guide describes several important topics to use Spring Data JDBC for ScalarDB and its limitations. + +:::warning + +Spring Data JDBC for ScalarDB extends Spring Data JDBC, but full compatibility is not guaranteed. Only the features listed on this page are officially tested and supported. + +::: + +## Add Spring Data JDBC for ScalarDB to your project + +To add the dependencies on Spring Data JDBC for ScalarDB by using Gradle, use the following, replacing `` with the versions of Spring Data JDBC for ScalarDB and the related library, respectively, that you are using: + +```gradle +dependencies { + implementation 'com.scalar-labs:scalardb-sql-spring-data:' + implementation 'com.scalar-labs:scalardb-cluster-java-client-sdk:' +} +``` + +To add the dependencies by using Maven, use the following, replacing `...` with the version of Spring Data JDBC for ScalarDB that you are using: + +```xml + + + com.scalar-labs + scalardb-sql-spring-data + ... + + + com.scalar-labs + scalardb-cluster-java-client-sdk + ... + + +``` + +## Configurations + +Spring Data JDBC for ScalarDB is supposed to be used as a part of Spring application. The following properties are needed at least. + +### spring.datasource.driver-class-name + +This needs to be set to fixed value `com.scalar.db.sql.jdbc.SqlJdbcDriver`. + +```console +spring.datasource.driver-class-name=com.scalar.db.sql.jdbc.SqlJdbcDriver +``` + +### spring.datasource.url + +This value follows the ScalarDB JDBC connection URL configuration. For more information, see [ScalarDB JDBC Guide](jdbc-guide.mdx) and [ScalarDB Cluster SQL client configurations](../scalardb-cluster/developer-guide-for-scalardb-cluster-with-java-api.mdx#scalardb-cluster-sql-client-configurations). + +```console +spring.datasource.url=jdbc:scalardb:\ +?scalar.db.sql.connection_mode=direct\ +&scalar.db.contact_points=jdbc:mysql://localhost:3306/my_app_ns\ +&scalar.db.username=root\ +&scalar.db.password=mysql\ +&scalar.db.storage=jdbc\ +&scalar.db.consensus_commit.isolation_level=SERIALIZABLE +``` + +## Annotations + +`@EnableScalarDbRepositories` annotation is needed on the JVM application to use Spring Data JDBC for ScalarDB as follows. + +```java +@SpringBootApplication +@EnableScalarDbRepositories +public class MyApplication { + // These repositories are described in the next section in details + @Autowired private GroupRepository groupRepository; + @Autowired private UserRepository userRepository; +``` + +## Persistent entity model + +The users of Spring Data JDBC for ScalarDB needs to write classes for object mapping to ScalarDB tables. How to write those classes are written in [Persisting Entities](https://docs.spring.io/spring-data/jdbc/docs/3.0.x/reference/html/#jdbc.entity-persistence), so this section describes some limitations on the integration. + +These are example model classes: + +### domain/model/User + +```java +// This model class corresponds to the following table schema: +// +// create table my_app_ns.user (id bigint, group_id bigint, name text, primary key (id)); +// +// -- UserRepository can use `name` column as a condition in SELECT statement +// -- as the column is a ScalarDB secondary index. +// create index on my_app_ns.user (name); + +// Set `schema` parameter in @Table annotation if you don't use `scalar.db.sql.default_namespace_name` property. +// +// Spring Data automatically decides the target table name based on a model class name. +// You can also specify a table name by setting `value` parameter. +// +// @Table(schema = "my_app_ns", value = "user") +@Table +public class User { + @Id + public final Long id; + + public final Long groupId; + + // Spring Data automatically decides the target column name based on an instance variable name. + // You can also specify a column name by setting `value` parameter in @Column annotation. + // @Column("name") + public final String name; + + public User(Long id, Long groupId, String name) { + this.id = id; + this.groupId = groupId; + this.name = name; + } +} +``` + +### domain/model/Group + +```java +// This model class corresponds to the following table schema: +// +// create table my_app_ns.group (account_id int, group_type int, balance int, primary key (account_id, group_type)); + +@Table +public class Group { + // This column `account_id` is a part of PRIMARY KEY in ScalarDB SQL + // + // Spring Data JDBC always requires a single @Id annotation while it doesn't allow multiple @Id annotations. + // The corresponding ScalarDB SQL table `group` has a primary key consisting of multiple columns. + // So, Spring Data @Id annotation can't be used in this case, but @Id annotation must be put on any instance variable + // (@Id annotation can be put on `balance` as well.) + @Id + public final Integer accountId; + + // This column `group_type` is also a part of PRIMARY KEY in ScalarDB SQL + public final Integer groupType; + + public final Integer balance; + + public Group(Integer accountId, Integer groupType, Integer balance) { + this.accountId = accountId; + this.groupType = groupType; + this.balance = balance; + } +} +``` + +[This sample implementation](https://github.com/scalar-labs/scalardb-samples/tree/main/spring-data-sample/src/main/java/sample/domain/model) can be used as a reference as well. + +### domain/repository/UserRepository + +```java +@Transactional +@Repository +public interface UserRepository extends ScalarDbRepository { + + // `insert()` and `update()` are automatically enabled with `ScalarDbRepository` (or `ScalarDbTwoPcRepository`). + + // Many APIs of `CrudRepository` and `PagingAndSortingRepository` are automatically enabled. + // https://docs.spring.io/spring-data/commons/docs/3.0.x/api/org/springframework/data/repository/CrudRepository.html + // https://docs.spring.io/spring-data/commons/docs/3.0.x/api/org/springframework/data/repository/PagingAndSortingRepository.html + + // Also, you can prepare complicated APIs with the combination of the method naming conventions. + // https://docs.spring.io/spring-data/jdbc/docs/3.0.x/reference/html/#repositories.definition-tuning + + // These APIs use the ScalarDB secondary index + List findByName(String name); + List findTop2ByName(String name); + // Current ScalarDB SQL doesn't support range scan or order using secondary indexes + // List findByNameBetween(String name); + // List findByGroupIdOrderByName(long groupId); + + default void reverseName(long id) { + Optional model = findById(id); + if (model.isPresent()) { + User existing = model.get(); + User updated = + new User( + existing.id, + existing.groupId, + existing.name.reverse()); + update(updated); + } + } + + default void deleteAfterSelect(long id) { + Optional existing = findById(id); + existing.ifPresent(this::delete); + } +} +``` + +### domain/repository/GroupRepository + +```java +@Transactional +@Repository +public interface GroupRepository extends ScalarDbRepository { + + // @Id annotation is put only on Group.accountId, but ScalarDB SQL expects the combination of + // `account_id` and `group_type` columns as the table uses them as a primary key. So `findById()` can't be used. + Optional findFirstByAccountIdAndGroupType(int accountId, int groupType); + + List findByAccountIdAndGroupTypeBetweenOrderByGroupTypeDesc( + int accountId, int groupTypeFrom, int groupTypeTo); + + List findTop2ByAccountIdAndGroupTypeBetween( + int accountId, int groupTypeFrom, int groupTypeTo); + + // `update()` method also depends on @Id annotation as well as `findById()`, + // so users need to write ScalarDB SQL in @Query annotation. + @Modifying + @Query( + "UPDATE \"my_app_ns\".\"group\" SET \"balance\" = :balance \n" + + " WHERE \"my_app_ns\".\"group\".\"account_id\" = :accountId \n" + + " AND \"my_app_ns\".\"group\".\"group_type\" = :groupType \n") + int updateWithAttributes( + @Param("accountId") int accountId, + @Param("groupType") int groupType, + @Param("balance") int balance); + + default void incrementBalance(int accountId, int groupType, int value) { + Optional model = findFirstByAccountIdAndGroupType(accountId, groupType); + model.ifPresent( + found -> + updateWithAttributes( + found.accountId, found.groupType, found.balance + value)); + } + + default void transfer( + int accountIdFrom, int groupTypeFrom, int accountIdTo, int groupTypeTo, int value) { + incrementBalance(accountIdFrom, groupTypeFrom, -value); + incrementBalance(accountIdTo, groupTypeTo, value); + } + + // This method name and signature results in issuing an unexpected SELECT statement and + // results in query failure. It looks a bug of Spring Data... + // + // void deleteByAccountIdAndGroupType(int accountId, int groupType); + + @Modifying + @Query( + "DELETE FROM \"my_app_ns\".\"group\" \n" + + " WHERE \"my_app_ns\".\"group\".\"account_id\" = :accountId \n" + + " AND \"my_app_ns\".\"group\".\"group_type\" = :groupType \n") + int deleteByAccountIdAndGroupType( + @Param("accountId") int accountId, @Param("groupType") int groupType); + + default void deleteByAccountIdAndGroupTypeAfterSelect(int accountId, int groupType) { + Optional entity = findFirstByAccountIdAndGroupType(accountId, groupType); + entity.ifPresent(found -> deleteByAccountIdAndGroupType(accountId, groupType)); + } +} +``` + +[This sample implementation](https://github.com/scalar-labs/scalardb-samples/tree/main/spring-data-sample/src/main/java/sample/domain/repository) can be used as a reference as well. + +## Error handling + +Spring Data JDBC for ScalarDB can throw the following exceptions. + +- com.scalar.db.sql.springdata.exception.ScalarDbTransientException + - This is thrown when a transaction fails due to a transient error + - The transaction should be retried + - This is a subclass of `org.springframework.dao.TransientDataAccessException` and catching the superclass is safer to handle other type of transient errors thrown from Spring Data +- com.scalar.db.sql.springdata.exception.ScalarDbNonTransientException + - This is thrown when a transaction fails due to a non-transient error + - The transaction should not be retried + - This is a subclass of `org.springframework.dao.NonTransientDataAccessException` and catching the superclass is safer to handle other type of non-transient errors thrown from Spring Data +- com.scalar.db.sql.springdata.exception.ScalarDbUnknownTransactionStateException + - This is a subclass of `ScalarDbNonTransientException` and the transaction should not be retried as well + - This is thrown when a transaction commit fails and the final state is unknown + - Whether the transaction is actually committed or not needs to be decided by the application side (e.g. check if the target record is expectedly updated) + +These exceptions include the transaction ID, which can be useful for troubleshooting purposes. + +## Limitations + +### Multiple column PRIMARY KEY + +As you see in the above example, Spring Data JDBC's `@Id` annotation doesn't support multiple columns. So, if a table has a primary key consisting of multiple columns, users can't use the following APIs and may need to write Scalar SQL DB query in `@Query` annotation. + +- `findById()` +- `existsById()` +- `update(T entity)` +- `delete(T entity)` +- `deleteById(ID id)` +- `deleteAllById(Iterable ids)` + +### One-to-many relationships between two entities + +Spring Data JDBC supports one-to-many relationships. But it implicitly deletes and re-creates all the associated child records even if only parent's attributes are changed. This behavior would result in a performance penalty. Additionally, certain use cases of the one-to-many relationship in Spring Data JDBC for ScalarDB fail because of the combination with some limitations of ScalarDB SQL. Considering those concerns and limitations, it's not recommended to use the feature in Spring Data JDBC for ScalarDB. + +For instance, assuming a Bank record contains many Account records, the following implementation fails when calling `BankRepository#update()` + +```java +@Autowired BankRepository bankRepository; + +... + +bankRepository.insert(new Bank(42, "My bank", ImmutableSet.of( + new Account(1, "Alice"), + new Account(2, "Bob"), + new Account(3, "Carol") +))); + +Bank bank = bankRepository.findById(42).get(); +System.out.printf("Bank: " + bank); + +// Fails here as `DELETE FROM "account" WHERE "account"."bank_id" = ?` is implicitly issued by Spring Data JDBC +// while ScalarDB SQL doesn't support DELETE with a secondary index +// (Spring Data JDBC's custom query might avoid these limitations) +bankRepository.update(new Bank(bank.bankId, bank.name + " 2", bank.accounts)); +``` + +## Advanced features + +### Multi-storage transaction + +ScalarDB supports [Multi-storage Transaction](../multi-storage-transactions.mdx), and users can use the feature via Spring Data JDBC for ScalarDB. The following needs to be configured to use the feature. + +#### spring.datasource.url +Here is a sample datasource URL assuming there are two namespaces "north" and "south" that manage data with MySQL and PostgreSQL respectively. +``` +spring.datasource.url=jdbc:scalardb:\ +?scalar.db.sql.connection_mode=direct\ +&scalar.db.storage=multi-storage\ +&scalar.db.multi_storage.storages=mysql,postgresql\ +&scalar.db.multi_storage.namespace_mapping=north:mysql,south:postgresql\ +&scalar.db.multi_storage.default_storage=postgresql\ +&scalar.db.multi_storage.storages.mysql.storage=jdbc\ +&... +``` + +#### @Table annotation on model classes + +- `schema` parameter: multi-storage mapping key name (`scalar.db.multi_storage.namespace_mapping`) +- `value` parameter: actual table name + +```java + @Table(schema = "north", value = "account") + public class NorthAccount { +``` + +### Retry + +#### Retry with Spring Retry + +Spring Data JDBC for ScalarDB could throw exceptions when concurrent transactions conflict. Users need to take care of those exceptions by retrying the operations. [Spring Retry](https://github.com/spring-projects/spring-retry) provides some functionalities for retry. Also in Spring Data JDBC for ScalarDB, Spring Retry would be helpful to make retry handling simpler and easier. This section introduces how to use Spring Retry. + +##### Dependencies + +The following dependencies need to be added to your project. + +```gradle +dependencies { + implementation "org.springframework.boot:spring-boot-starter:${springBootVersion}" + implementation "org.springframework.boot:spring-boot-starter-aop:${springBootVersion}" + implementation "org.springframework.retry:spring-retry:${springRetryVersion}" +} +``` + +##### Annotation + +`@EnableRetry` annotation needs to be added in the JVM application. +```java +@SpringBootApplication +@EnableScalarDbRepositories +@EnableRetry +public class MyApp { +``` + +`@Retryable` annotation makes Spring Data repository class or method automatically retry a failed operation. Spring Data JDBC for ScalarDB can throw a transient error exception, so it's highly recommended to specify `org.springframework.dao.TransientDataAccessException` as a target class in the annotation. Also, backoff and max attempts can be configured in the annotation like this: + +```java + @Transactional + @Retryable( + include = TransientDataAccessException.class, + maxAttempts = 4, + backoff = @Backoff(delay = 500, maxDelay = 2000, multiplier = 2)) + default void insertWithRetry(Player player) { + insert(player); + } +``` + +With `@Recover` annotation, retry-exhausted failure will be automatically recovered by a specified method. + +```java + @Transactional + @Retryable(include = TransientDataAccessException.class, + recover = "recoverInsert") + default void insertWithRetryAndRecover(Player player) { + insert(player); + } + + @Transactional + @Recover + default void recoverInsert(Throwable throwable, Player player) throws Throwable { + Optional existing = findById(player.id); + if (!existing.isPresent()) { + throw throwable; + } + logger.info( + "Found an existing record {}. Updating it with a new record {}", existing.get(), player); + + update(player); + } +``` + +#### Retry with other retry library + +There are other options available for retrying transactions, such as Spring Retry's RetryTemplate or Resilience4j. Feel free to choose and use your preferred retry library. + +### Two-phase commit transaction + +ScalarDB supports [Two-phase commit transaction](../two-phase-commit-transactions.mdx), and users can use the feature via Spring Data JDBC for ScalarDB. The following configurations are needed. + +#### spring.datasource.url + +- `scalar.db.sql.default_transaction_mode` property: `two_phase_commit_transaction` + +```console +spring.datasource.url=jdbc:scalardb:\ +?scalar.db.sql.connection_mode=direct\ +&scalar.db.contact_points=jdbc:mysql://localhost:3306/my_app_ns\ +&...\ +&scalar.db.sql.default_transaction_mode=two_phase_commit_transaction +``` + +#### Configuration of Spring Data transaction manager + +Spring Data JDBC for ScalarDB provides a custom Spring Data transaction manager to achieve 2PC transactions. You need to configure either of the following annotations to enable the custom transaction manager. + +- Set `transactionManager` parameter of all the `@Transactional` to `scalarDbSuspendableTransactionManager` +- Set `transactionManagerRef` parameter of the `@EnableScalarDbRepositories` to `scalarDbSuspendableTransactionManager` + +#### Repository classes + +##### APIs + +Spring Data JDBC for ScalarDB supports 2 types of APIs for 2PC transaction. One is primitive APIs and the other is high level API. + +###### Primitive 2PC APIs + +`ScalarDbTwoPcRepository` is an extension of `ScalarDbRepository` and it has the following APIs that correspond to the same name methods in ScalarDB and users can use them to build custom repository methods for 2PC transaction. + +- begin() + - returns an auto-generated transaction ID +- prepare() +- validate() +- suspend() +- commit() +- join(`transactionId`) +- resume(`transactionId`) + +All in-flight operations are rolled back when any exception is thrown from Spring Data repository method. + +See [How to execute Two-phase Commit Transactions](../two-phase-commit-transactions.mdx#how-to-execute-two-phase-commit-transactions) for details. + +```java +@Transactional(transactionManager = "scalarDbSuspendableTransactionManager") +@Repository +public interface TwoPcPlayerRepository extends ScalarDbTwoPcRepository { + + Logger logger = LoggerFactory.getLogger(TwoPcPlayerRepository.class); + + // Either of twoPcJoinAndInsert() or twoPcBeginAndInsert() can be used to start a transaction + default void twoPcJoinAndInsert(String txId, Player player) throws SQLException { + join(txId); + insert(player); + suspend(); + } + + default String twoPcBeginAndInsert(String id, Player player) throws SQLException { + String txId = begin(); + insert(player); + suspend(); + return txId; + } + + default void twoPcPrepare(String txId) throws SQLException { + resume(txId); + prepare(); + suspend(); + } + + default void twoPcValidate(String txId) throws SQLException { + resume(txId); + validate(); + suspend(); + } + + default void twoPcCommit(String txId) throws SQLException { + resume(txId); + commit(); + } +``` + +###### High level 2PC API + +The above primitive APIs are powerful and make it possible to explicitly control 2PC transaction operations in flexible and fine-grained ways. On the other hand, users need to consider which APIs to call in a proper order when using the APIs. Especially coordinator side operations for local state and remote service calls would be easily complicated. + +`ScalarDbTwoPcRepository` also provides some user-friendly APIs called high-level APIs to cover common use cases. With these APIs, you can develop your microservice applications more easily and securely. + +For the development of coordinator service in a microservice, `ScalarDbTwoPcRepository` provides `executeTwoPcTransaction` API that implicitly executes 2PC related operations in the following order. By using the API, you don’t need to think about how and when to execute transactional operations. + +- Start a local transaction with a global transaction ID +- Execution phase: Local and remote CRUD operations (*) +- Prepare phase: Local and remote prepare operations (**) in parallel +- Validation phase: Local and remote validation operations (**) in parallel + - This is needed only if `scalar.db.consensus_commit.isolation_level` is `SERIALIZABLE` and `scalar.db.consensus_commit.serializable_strategy` is `EXTRA_READ` +- Commit phase: Local commit operation is first executed. Remote commit operations are executed (**) in parallel after the local commit operation succeeded +- (If any operation except for remote commit operation fails) rollback phase: Local and remote rollback operations (**) in parallel + +(* This implementation of local and remote operation callbacks is injected by users)\ +(** This implementation of remote operation callbacks is injected by users) + +Rollback operations for local and remote participants will be executed when an exception is thrown from any operation. + +As for the error handling of `executeTwoPcTransaction()`, + +- The following exceptions can be thrown from the API + - `ScalarDbTransientException` + - Users should retry the 2PC transaction operations from the beginning when this exception is thrown + - `ScalarDbNonTransientException` + - `ScalarDbUnknownTransactionStateException` + - Whether the 2PC transaction is actually committed or not needs to be decided by the application side +- The exceptions contain the 2PC global transaction ID. It should be useful for trouble shootings + +As for the implementations of Execution phase operations (in local and remote participants) and remote operations of Prepare/Validation/Commit/Rollback phases that are passed by users, those callbacks need to throw either of the exceptions when it fails: + +- `ScalarDbTransientException` when any transient issue happens including network disconnection and database transaction conflict +- `ScalarDbNonTransientException` when any non-transient issue happens including authentication error and permission error +- `ScalarDbUnknownTransactionStateException` when any exception that contains `UnknownTransactionStatusException` as a cause +- Other exceptions thrown from the callbacks are treated as `ScalarDbTransientException` + +For the development of participant service in a microservice, `ScalarDbTwoPcRepository` provides the following APIs. By using the API, you don’t need to think about how and when to join, resume and suspend a transaction in details. + +- `joinTransactionOnParticipant` + - Join the transaction, execute the CRUD operations and suspend the transaction on the participant service. This API should be called first, and then `prepareTransactionOnParticipant` and following APIs are supposed to be called. +- `resumeTransactionOnParticipant` + - Resume the transaction, execute the CRUD operations and suspend the transaction on the participant service. This API can be called after `joinTransactionOnParticipant` before `prepareTransactionOnParticipant` if needed. +- `prepareTransactionOnParticipant` + - Prepare the transaction and suspend the transaction on the participant service. This API should be called after `joinTransactionOnParticipant`, and then `validateTransactionOnParticipant` and following APIs are supposed to be called. +- `validateTransactionOnParticipant` + - Validate the transaction and suspend the transaction on the participant service. This API should be called after `prepareTransactionOnParticipant`, and then `commitTransactionOnParticipant` or `rollbackTransactionOnParticipant` is supposed to be called. + - This is needed only if `scalar.db.consensus_commit.isolation_level` is `SERIALIZABLE` and `scalar.db.consensus_commit.serializable_strategy` is `EXTRA_READ` +- `commitTransactionOnParticipant` + - Commit the transaction on the participant service. This API should be called after `prepareTransactionOnParticipant` or `validateTransactionOnParticipant, depending on the transaction manager configurations. +- `rollbackTransactionOnParticipant` + - Rollback the transaction on the participant service. This API should be called after `prepareTransactionOnParticipant` or `validateTransactionOnParticipant, depending on the transaction manager configurations. + +With the high-level 2PC APIs of Spring Data JDBC for ScalarDB, you can focus on the business logic by hiding complicated transaction operations inside the APIs as follows: + +**Coordinator service** + +```java + @Autowired private AccountRepository accountRepository; + private final StockService stockService = ...; + private final NotificationService notificationService = ...; + private final List remotePrepareCommitOpsList = + Arrays.asList( + RemotePrepareCommitPhaseOperations.createSerializable( + stockService::prepareTransaction, + stockService::validateTransaction, + stockService::commitTransaction, + stockService::rollbackTransaction), + RemotePrepareCommitPhaseOperations.createSerializable( + notificationService::prepareTxn, + notificationService::validateTxn, + notificationService::commitTxn, + notificationService::rollbackTxn)); +``` + +```java + private Result> executeTwoPcTransactionUsingHighLevelApi( + Account account, String itemName, int itemPrice, String notificationEventName) { + return accountRepository.executeTwoPcTransaction( + // CRUD operations for local and remote participants in execution phase. + txId -> { + // [local] Read the account's balance + Optional stored = accountRepository.findById(account.id); + if (!stored.isPresent()) { + // Cancel the transaction if the account doesn't exist. + // No need to retry. + throw new ScalarDbNonTransientException( + "The local state doesn't meet the condition. Aborting this transaction"); + } + // [remote] Start a transaction with the transaction ID, + // read the item information and decrement the count + Optional price = stockService.purchaseItem(txId, account.id, itemName); + // [remote] Start a transaction with the transaction ID, + // read the notification and remove it + Optional notification = + notificationService.getNotification(txId, account.id, notificationEventName); + if (price.isPresent() && notification.isPresent()) { + int currentBalance = stored.get().balance - price.get(); + if (currentBalance < 0) { + // Cancel the transaction if the global state doesn't meet the condition. + // No need to retry. + throw new ScalarDbNonTransientException( + "The state of local and remote participants doesn't meet the condition. Aborting this transaction"); + } + // [local] Decrease the account's balance for the item + accountRepository.update(new Account(account.id, currentBalance)); + return Pair.of(currentBalance, notification.get()); + } + // Cancel the transaction if the global state doesn't meet the condition. + // No need to retry. + throw new ScalarDbNonTransientException( + "The remote state doesn't meet the condition. Aborting this transaction"); + }, + // Remote operations for Prepare/Validate/Commit/Rollback + remotePrepareCommitOpsList); + } +``` + +```java + RetryTemplate retryTemplate = + new RetryTemplateBuilder() + .retryOn(TransientDataAccessException.class) + .exponentialBackoff(500, 2.0, 8000) + .maxAttempts(8) + .withListener( + new RetryListenerSupport() { + @Override + public void onError(RetryContext context, RetryCallback callback, Throwable throwable) { + if (throwable instanceof ScalarDbUnknownTransactionStateException) { + // Report an exception occurred that requires special treatments + reportToDevelopers( + String.format("Failed to process a 2PC transaction (%s). The final transaction status is unknown. Please check current application status", + ((ScalarDbUnknownTransactionStateException) throwable).getTransactionId()), throwable); + } + }}) + .build(); + + Result> result = + retryTemplate.execute(context -> + executeTwoPcTransactionUsingHighLevelApi(account, itemName, itemPrice, notificationEventName)); +``` + +[This sample implementation](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/order-service/src/main/java/sample/order/OrderService.java) can be used as a reference as well. + +**Participant service** + +```java +@RestController +public class StockController { + @Autowired private StockRepository stockRepository; + + @PostMapping("/purchaseItem") + public Optional purchaseItem( + @RequestParam("transactionId") String transactionId, + @RequestParam("accountId") int accountId, + @RequestParam("itemName") String itemName) { + return stockRepository.joinTransactionOnParticipant(txId, () -> { + Optional item = stockRepository.findById(itemName); + + ... + + return Optional.of(item.price); + }); + } + + @PostMapping("/prepareTransaction") + public void prepareTransaction(@RequestParam("transactionId") String transactionId) { + return stockRepository.prepareTransactionOnParticipant(txId); + } + + @PostMapping("/validateTransaction") + public void validateTransaction(@RequestParam("transactionId") String transactionId) { + return stockRepository.validateTransactionOnParticipant(txId); + } + + @PostMapping("/commitTransaction") + public void commitTransaction(@RequestParam("transactionId") String transactionId) { + return stockRepository.commitTransactionOnParticipant(txId); + } + + @PostMapping("/rollbackTransaction") + public void rollbackTransaction(@RequestParam("transactionId") String transactionId) { + return stockRepository.rollbackTransactionOnParticipant(txId); + } +} +``` + +[This sample implementation](https://github.com/scalar-labs/scalardb-samples/blob/main/spring-data-microservice-transaction-sample/customer-service/src/main/java/sample/customer/CustomerService.java) uses gRPC not REST API, but it can be used as a reference as well. + +#### How to use both 2PC and normal transaction modes in a JVM application + +In most cases, only one of the 2PC and normal transaction modes is supposed to be used in an application. But there might be some use cases for using both transaction modes. For instance, assuming a service that is used as a participant in 2PC also has some APIs that are directly called by other services or clients without 2PC protocol. In this case, developers would want to simply use normal transaction mode for the APIs not used in 2PC. + +To achieve this use case, different `scalar.db.sql.default_transaction_mode` parameters for 2PC and normal transaction modes need to be passed to Spring Data JDBC framework via `spring.datasource.url` property. Spring Data JDBC doesn't provide a simple way to use multiple datasource configurations, though. But with some custom configuration classes, users can use both 2PC and normal transaction modes in a JVM application using multiple datasource configurations. + +#### Limitations + +##### `@Transactional` methods don't implicitly call `commit()` + +In microservice applications with ScalarDB, commits must be explicitly invoked by a coordinator service, not be locally triggered by the Spring Data transaction framework when exiting `@Transactional` methods. The `@Transactional(transactionManager = "scalarDbSuspendableTransactionManager")` annotation prevents such local commits. + +This extended behavior may confuse developers who expect `@Transactional` methods to implicitly commit transactions. + +For instance, assuming you want to use the `@Transactional` annotation on methods of a service class, the following code works in the **normal** transaction mode. + +```java +@Service +public class SampleService { + + ... + + // For the normal transaction mode + @Transactional + // For the 2PC transaction mode + // @Transactional(transactionManager = "scalarDbSuspendableTransactionManager") + public void repayment(int customerId, int amount) { + Customer customer = customerRepository.getById(customerId); + + int updatedCreditTotal = customer.creditTotal - amount; + + // Check if over repayment or not + if (updatedCreditTotal < 0) { + throw new RuntimeException( + String.format( + "Over repayment. creditTotal:%d, payment:%d", customer.creditTotal, amount)); + } + + // Reduce credit_total for the customer + customerRepository.update(customer.withCreditTotal(updatedCreditTotal)); + } +} +``` + +However, that code doesn't work in the 2PC transaction mode even with `transactionManager = "scalarDbSuspendableTransactionManager"`. Instead, use `ScalarDbTwoPcRepository.executeOneshotOperations()` as follows. + +```java +@Service +public class SampleService { + + ... + + public void repayment(int customerId, int amount) { + customerRepository.executeOneshotOperations(() -> { + Customer customer = customerRepository.getById(customerId); + + int updatedCreditTotal = customer.creditTotal - amount; + + // Check if over repayment or not + if (updatedCreditTotal < 0) { + throw new RuntimeException( + String.format( + "Over repayment. creditTotal:%d, payment:%d", customer.creditTotal, amount)); + } + + // Reduce credit_total for the customer + customerRepository.update(customer.withCreditTotal(updatedCreditTotal)); + + return null; + }); + } +} +``` + +## Troubleshooting + +This section describes methods to troubleshoot errors that may occur when using Spring Data JDBC. + +### `A constructor parameter name must not be null to be used with Spring Data JDBC` runtime error + +The runtime error `A constructor parameter name must not be null to be used with Spring Data JDBC` may occur when using Spring Boot 3. To work around this issue, you can pass the `-parameters` option to `javac` as follows: + +```gradle + compileJava { + options.compilerArgs << '-parameters' + } +``` + +## Sample application + +You can see the following sample applications that use Spring Data JDBC for ScalarDB. It only serves as a reference and does not necessarily meet production code standards. + +- [Getting Started with ScalarDB Cluster SQL via Spring Data JDBC for ScalarDB](../scalardb-cluster/getting-started-with-scalardb-cluster-sql-spring-data-jdbc.mdx) +- [Sample application of Spring Data JDBC for ScalarDB with Multi-storage Transactions](../scalardb-samples/spring-data-multi-storage-transaction-sample/README.mdx) +- [Sample application of Spring Data JDBC for ScalarDB with Microservice Transactions](../scalardb-samples/spring-data-microservice-transaction-sample/README.mdx) + +## How it works + +In order to use Spring Data JDBC for ScalarDB, the following features are implemented in the integration + +- Map `jdbc:scalardb` protocol in JDBC Connection URL to a Spring Data JDBC dialect class for ScalarDB SQL + - This feature is handled by ScalarDbDialect and ScalarDbDialectProvider +- Prevent users from using some APIs of Spring Data Repository classes (CrudRepository and PagingAndSortingRepository) unsupported in ScalarDB SQL + - This feature is handled by ScalarDbJdbcAggregateTemplate which is a bit lower layer Spring Data JDBC component used by Repository classes +- Make Spring Data Repository classes implicitly use the custom JdbcAggregateTemplate (ScalarDbJdbcAggregateTemplate) + - This feature is handled by ScalarDbJdbcRepositoryFactory and ScalarDbJdbcRepositoryFactoryBean +- Add explicit `insert()` and `update()` APIs to Spring Data Repository classes instead of bundled `save()` which depends on autoincrement ID feature in underlying databases while ScalarDB SQL doesn't support it + - This feature is handled by ScalarDbRepository (or ScalarDbTwoPcRepository) and ScalarDbRepositoryImpl +- Enable all the above features in Spring framework manner + - This configuration is handled by + - some Java classes in `com.scalar.db.sql.springdata` + - `@EnableScalarDbRepositories` annotation + - `resources/META-INF/spring/org.springframework.boot.autoconfigure.AutoConfiguration.imports` + - `resources/META-INF/spring.factories` + +## References + +- [Spring Data JDBC - Reference Documentation](https://docs.spring.io/spring-data/jdbc/docs/3.0.x/reference/html/) +- [ScalarDB JDBC Guide](jdbc-guide.mdx) +- [Javadoc for Spring Data JDBC for ScalarDB](https://javadoc.io/doc/com.scalar-labs/scalardb-sql-spring-data/3.16.0/index.html) diff --git a/versioned_docs/version-3.X/scalardb-sql/sql-api-guide.mdx b/versioned_docs/version-3.X/scalardb-sql/sql-api-guide.mdx new file mode 100644 index 00000000..9301191a --- /dev/null +++ b/versioned_docs/version-3.X/scalardb-sql/sql-api-guide.mdx @@ -0,0 +1,377 @@ +--- +tags: + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# ScalarDB SQL API Guide + +import JavadocLink from '/src/theme/JavadocLink.js'; + +This guide describes how to use ScalarDB SQL API. + +## Add ScalarDB SQL API to your project + +To add the dependencies on ScalarDB SQL API by using Gradle, use the following, replacing `` with the versions of ScalarDB SQL API and the related library, respectively, that you are using: + +```gradle +dependencies { + implementation 'com.scalar-labs:scalardb-sql:' + implementation 'com.scalar-labs:scalardb-cluster-java-client-sdk:' +} +``` + +To add the dependencies by using Maven, use the following, replacing `...` with the version of ScalarDB SQL API that you are using: + +```xml + + + com.scalar-labs + scalardb-sql + ... + + + com.scalar-labs + scalardb-cluster-java-client-sdk + ... + + +``` + +## SqlSessionFactory + +In ScalarDB SQL API, you execute all operations through a `SqlSession` instance, which is instantiated with `SqlSessionFactory`. +This section explains how to use them. + +Before explaining `SqlSessionFactory`, we start with the explanation for Connection mode and Transaction mode. + +### Transaction mode + +Also, ScalarDB SQL offers two transaction modes: *Transaction* mode and *Two-phase Commit Transaction* mode. +Transaction mode exposes only `commit` interface to users and runs two-phase commit behind the scene, while Two-phase Commit Transaction mode exposes two-phase commit style interfaces (`prepare` and `commit`) to users. + +You can specify the default transaction mode in your configuration file or when you build `SqlSessionFactory`. +And you also can change it with the `setTransactionMode()` method of `SqlSession`. + +### Build SqlSessionFactory + +You can build `SqlSessionFactory` with a properties file as follows: + +```java +SqlSessionFactory sqlSessionFactory = SqlSessionFactory.builder() + .withPropertiesFile("") + // If you need to set custom properties, you can specify them with withProperty() or withProperties() + .withProperty("", "") + .build(); +``` + +Please see [ScalarDB Cluster SQL client configurations](../scalardb-cluster/developer-guide-for-scalardb-cluster-with-java-api.mdx#scalardb-cluster-sql-client-configurations) for the details of the configurations. + +### Get a SqlSession instance + +You can get a `SqlSession` instance with `SqlSessionFactory` as follows: + +```java +SqlSession sqlSession = sqlSessionFactory.createSqlSession(); +``` + +Note that `SqlSession` is not thread-safe. +Please don't use it from multiple threads at the same time. + +#### Close a SqlSession instance + +Once all operations are done with a `SqlSession` instance, you should close the SqlSession instance: + +```java +sqlSession.close(); +``` + +### Close a SqlSessionFactory instance + +`sqlSessionFactory` should also be closed once it's no longer needed: + +```java +sqlSessionFactory.close(); +``` + +## Execute SQLs + +You can execute a SQL with `SqlSession` as follows: + +```java +ResultSet resultSet = sqlSession.execute(""); +``` + +You can also execute a `Statement` object with `SqlSession` as follows: + +```java +// Build a statement +Statement statement = StatementBuilder....; + +// Execute the statement +ResultSet resultSet = sqlSession.execute(statement); +``` + +`Statement` objects can be built by `StatementBuilder` that has factory methods for corresponding SQLs. For more details, see the page in the Javadoc and [ScalarDB SQL Grammar](grammar.mdx). + +### Handle ResultSet objects + +As the result of the SQL execution, `SqlSession` returns a `ResultSet` object. +Here, we describe how to handle `ResultSet` objects. + +If you want to get results one by one from the `ResultSet` object, you can use the `one()` method as follows: +```java +Optional record = resultSet.one(); +``` + +Or, if you want to get results all at once as a `List`, you can use the `all()` method as follows: +```java +List records = resultSet.all(); +``` + +Also, as `ResultSet` implements `Iterable`, you can use it in a for-each loop as follows: + +```java +for (Record record : resultSet) { + ... +} +``` + +If you want to get the metadata of the `ResultSet` object, you can use the `getColumnDefinitions()` method as follows: + +```java +ColumnDefinitions columnDefinitions = resultSet.getColumnDefinitions(); +``` + +For more details, see the page in the Javadoc. + +### Handle Record objects + +As mentioned, a `ResultSet` object returns `Record` objects that represent records of the database. + +You can get a column value of a result with `getXXX("")` or `getXXX()` methods (XXX is a type name) as follows: + +```java +// Get a BOOLEAN value of a column +boolean booleanValueGottenByName = record.getBoolean(""); +boolean booleanValueGottenByIndex = record.getBoolean(); + +// Get an INT value of a column +int intValueGottenByName = record.getInt(""); +int intValueGottenByIndex = record.getInt(); + +// Get a BIGINT value of a column +long bigIntValueGottenByName = record.getBigInt(""); +long bigIntValueGottenByIndex = record.getBigInt(); + +// Get a FLOAT value of a column +float floatValueGottenByName = record.getFloat(""); +float floatValueGottenByIndex = record.getFloat(); + +// Get a DOUBLE value of a column +double doubleValueGottenByName = record.getDouble(""); +double doubleValueGottenByIndex = record.getDouble(); + +// Get a TEXT value of a column +String textValueGottenByName = record.getText(""); +String textValueGottenByIndex = record.getText(); + +// Get a BLOB value of a column (as a ByteBuffer) +ByteBuffer blobValueGottenByName = record.getBlob(""); +ByteBuffer blobValueGottenByIndex = record.getBlob(); + +// Get a BLOB value of a column as a byte array +byte[] blobValueAsBytesGottenByName = record.getBlobAsBytes(""); +byte[] blobValueAsBytesGottenByIndex = record.getBlobAsBytes(); + +// Get a DATE value of a column as a LocalDate +LocalDate dateValueGottenByName = record.getDate(""); +LocalDate dateValueGottenByName = record.getDate(); + +// Get a TIME value of a column as a LocalTime +LocalTime timeValueGottenByName = record.getTime(""); +LocalTime timeValueGottenByName = record.getTime(); + +// Get a TIMESTAMP value of a column as a LocalDateTime +LocalDateTime timestampValueGottenByName = record.getTimestamp(""); +LocalDateTime timestampValueGottenByName = record.getTimestamp(); + +// Get a TIMESTAMPTZ value of a column as an Instant +Instant timestampTZValueGottenByName = record.getTimestampTZ(""); +Instant timestampTZValueGottenByName = record.getTimestampTZ(); +``` + +And if you need to check if a value of a column is null, you can use the `isNull("")` or `isNull()` method. + +``` java +// Check if a value of a column is null +boolean isNullGottenByName = record.isNull(""); +boolean isNullGottenByIndex = record.isNull(); +``` + +For more details, see the page of the Javadoc. + +### Prepared Statements + +You can use `PreparedStatement` for queries that are executed multiple times in your application: + +```java +PreparedStatement preparedStatement = sqlSession.prepareStatement(""); +ResultSet result = preparedStatement.execute(); +``` + +If you execute the same query a second time or later, the cached pre-parsed statement object is used. +Thus, you can gain a performance advantage with `PreparedStatement` when you execute the query multiple times. +If you execute a query only once, a prepared statement is inefficient because it requires extra processing. +Consider using the `sqlSession.execute()` method instead in that case. + +Also, you can use `PreparedStatement` with bind parameters. +Parameters can be either positional or named: + +```java +// Positional parameters +PreparedStatement preparedStatement1 = + sqlSession.prepareStatement("INSERT INTO tbl (c1, c2) VALUES (?, ?)"); + +// Named parameters +PreparedStatement preparedStatement2 = + sqlSession.prepareStatement("INSERT INTO tbl (c1, c2) VALUES (:a, :b)"); +``` + +You can set parameters first and execute it: + +```java +// Positional setters +preparedStatement1 + .setInt(0, 10) + .setText(1, "value") + .execute(); + +// Named setters +preparedStatement2 + .setInt("a", 10) + .setText("b", "value") + .execute(); +``` + +For more details, see the page of the Javadoc. + +## Execute transactions + +In ScalarDB SQL, you can execute DML statements (SELECT/INSERT/UPDATE/DELETE) only in transactions. +So before executing DML statements, you must begin a transaction. + +Note that you cannot execute statements other than DML statements transactionally. +So even if you execute a non-DML statement after beginning a transaction, it is executed immediately, and it doesn't affect the transaction you have begun. + +This section describes how to execute a transaction for each transaction mode: Transaction mode and Two-phase Commit Transaction mode. + +### Transaction Mode + +An example code for Transaction mode is as follows: + +```java +try { + // Begin a transaction + sqlSession.begin(); + + // Execute statements (SELECT/INSERT/UPDATE/DELETE) in the transaction + ... + + // Commit the transaction + sqlSession.commit(); +} catch (UnknownTransactionStatusException e) { + // If you catch `UnknownTransactionStatusException`, it indicates that the status of the + // transaction, whether it has succeeded or not, is unknown. In such a case, you need to check if + // the transaction is committed successfully or not and retry it if it failed. How to identify a + // transaction status is delegated to users +} catch (SqlException e) { + // For other exceptions, you can try retrying the transaction + + // Rollback the transaction + sqlSession.rollback(); + + // For `TransactionRetryableException`, you can basically retry the transaction. However, for + // the other exceptions, the transaction may still fail if the cause of the exception is + // nontransient. For such a case, you need to limit the number of retries and give up retrying +} +``` + +If you catch `UnknownTransactionStatusException`, it indicates that the status of the transaction, whether it has succeeded or not, is unknown. +In such a case, you need to check if the transaction is committed successfully or not and retry it if it fails. +How to identify a transaction status is delegated to users. +You may want to create a transaction status table and update it transactionally with other application data so that you can get the status of a transaction from the status table. + +If you catch another exception, you can try retrying the transaction. +For `TransactionRetryableException`, you can basically retry the transaction. +However, for the other exceptions, the transaction may still fail if the cause of the exception is nontransient. +For such a case, you need to limit the number of retries and give up retrying. + +### Two-phase Commit Transaction Mode + +Before reading this, please read [this document](../two-phase-commit-transactions.mdx) to learn the concept of Two-phase commit transactions. + +To begin a transaction for a coordinator, you can do as follows: + +```java +String transactionId = sqlSession.begin(); +``` + +And to join a transaction for participants, you can do as follows: + +```java +sqlSession.join(transactionId); +``` + +An example code of Two-phase Commit Transaction mode is as follows: + +```java +try { + // Begin a transaction + sqlSession.begin(); + + // Execute statements (SELECT/INSERT/UPDATE/DELETE) in the transaction + ... + + // Prepare the transaction + sqlSession.prepare(); + + // Validate the transaction + sqlSession.validate(); + + // Commit the transaction + sqlSession.commit(); +} catch (UnknownTransactionStatusException e) { + // If you catch `UnknownTransactionStatusException` when committing the transaction, it + // indicates that the status of the transaction, whether it has succeeded or not, is unknown. + // In such a case, you need to check if the transaction is committed successfully or not and + // retry it if it failed. How to identify a transaction status is delegated to users +} catch (SqlException e) { + // For other exceptions, you can try retrying the transaction + + // Rollback the transaction + sqlSession.rollback(); + + // For `TransactionRetryableException`, you can basically retry the transaction. However, for + // the other exceptions, the transaction may still fail if the cause of the exception is + // nontransient. For that case, you need to limit the number of retries and give up retrying +} +``` + +The exception handling is the same as Transaction mode. + +## Get Metadata + +You can get metadata with the `SqlSession.getMetadata()` method as follows: + +```java +Metadata metadata = sqlSession.getMetadata(); +``` + +For more details, see the page of the Javadoc. + +## References + +- [ScalarDB SQL Grammar](grammar.mdx) +- [Two-phase Commit Transactions](../two-phase-commit-transactions.mdx) +- [Javadoc for ScalarDB SQL](https://javadoc.io/doc/com.scalar-labs/scalardb-sql/3.16.0/index.html) diff --git a/versioned_docs/version-3.X/schema-loader-import.mdx b/versioned_docs/version-3.X/schema-loader-import.mdx new file mode 100644 index 00000000..836f204f --- /dev/null +++ b/versioned_docs/version-3.X/schema-loader-import.mdx @@ -0,0 +1,323 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Importing Existing Tables to ScalarDB by Using ScalarDB Schema Loader + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +You might want to use ScalarDB (e.g., for database-spanning transactions) with your existing databases. In that case, you can import those databases under the ScalarDB control using ScalarDB Schema Loader. ScalarDB Schema Loader automatically adds ScalarDB-internal metadata columns in each existing table and metadata tables to enable various ScalarDB functionalities including transaction management across multiple databases. + +## Before you begin + +:::warning + +You should carefully plan to import a table to ScalarDB in production because it will add transaction metadata columns to your database tables and the ScalarDB metadata tables. In this case, there would also be several differences between your database and ScalarDB, as well as some limitations. + +::: + +### What will be added to your databases + +- **ScalarDB metadata tables:** ScalarDB manages namespace names and table metadata in a namespace (schema or database in underlying databases) called 'scalardb'. +- **Transaction metadata columns:** The Consensus Commit transaction manager requires metadata (for example, transaction ID, record version, and transaction status) stored along with the actual records to handle transactions properly. Thus, this tool adds the metadata columns if you use the Consensus Commit transaction manager. + +:::note + +This tool only changes database metadata. Thus, the processing time does not increase in proportion to the database size and usually takes only several seconds. + +::: + +### Requirements + +- [JDBC databases](./requirements.mdx#relational-databases), except for SQLite, can be imported. +- Each table must have primary key columns. (Composite primary keys can be available.) +- Target tables must only have columns with supported data types. For details, see [Data-type mapping from JDBC databases to ScalarDB](#data-type-mapping-from-jdbc-databases-to-scalardb)). + +### Set up Schema Loader + +To set up Schema Loader for importing existing tables, see [Set up Schema Loader](./schema-loader.mdx#set-up-schema-loader). + +## Run Schema Loader for importing existing tables + +You can import an existing table in JDBC databases to ScalarDB by using the `--import` option and an import-specific schema file. To import tables, run the following command, replacing the contents in the angle brackets as described: + +```console +java -jar scalardb-schema-loader-.jar --config -f --import +``` + +- ``: Version of ScalarDB Schema Loader that you set up. +- ``: Path to a properties file for ScalarDB. For a sample properties file, see [`database.properties`](https://github.com/scalar-labs/scalardb/blob/master/conf/database.properties). +- ``: Path to an import schema file. For a sample, see [Sample import schema file](#sample-import-schema-file). + +If you use the Consensus Commit transaction manager after importing existing tables, run the following command separately, replacing the contents in the angle brackets as described: + +```console +java -jar scalardb-schema-loader-.jar --config --coordinator +``` + +## Sample import schema file + +The following is a sample schema for importing tables. For the sample schema file, see [`import_schema_sample.json`](https://github.com/scalar-labs/scalardb/blob/master/schema-loader/sample/import_schema_sample.json). + +```json +{ + "sample_namespace1.sample_table1": { + "transaction": true, + "override-columns-type": { + "c3": "TIME", + "c5": "TIMESTAMP" + } + }, + "sample_namespace1.sample_table2": { + "transaction": true + }, + "sample_namespace2.sample_table3": { + "transaction": false + } +} +``` + +The import table schema consists of a namespace name, a table name, a `transaction` field, and an optional `override-columns-type` field: +- The `transaction` field indicates whether or not the table will be imported for transactions. If you set the `transaction` field to `true` or don't specify the `transaction` field, this tool will create a table with transaction metadata, if needed. If you set the `transaction` field to `false`, this tool will import a table without adding transaction metadata (that is, for a table using the [Storage API](run-non-transactional-storage-operations-through-primitive-crud-interface.mdx)). +- The `override-columns-type` field indicates the columns for which you wish to override the default data-type mapping. This field is optional and only needs to be set with the columns requiring a type override. + +## Data-type mapping from JDBC databases to ScalarDB + +The following table shows the supported data types in each JDBC database and their mapping to the ScalarDB data types. Select your database and check if your existing tables can be imported. + + + + | MySQL | ScalarDB | Notes | + |--------------|-------------------------------------|---------------------------------------------------------------------------------------------------------------------| + | bigint | BIGINT | See warning [1](#1) below. | + | binary | BLOB | | + | bit | BOOLEAN | | + | blob | BLOB | See warning [2](#2) below. | + | char | TEXT | See warning [2](#2) below. | + | date | DATE | | + | datetime | TIMESTAMP (default) and TIMESTAMPTZ | When importing as TIMESTAMPTZ, ScalarDB will assume the data to be on the UTC time zone. See warning [6](#6) below. | + | double | DOUBLE | | + | float | FLOAT | | + | int | INT | | + | int unsigned | BIGINT | See warning [2](#2) below. | + | integer | INT | | + | longblob | BLOB | | + | longtext | TEXT | | + | mediumblob | BLOB | See warning [2](#2) below. | + | mediumint | INT | See warning [2](#2) below. | + | mediumtext | TEXT | See warning [2](#2) below. | + | smallint | INT | See warning [2](#2) below. | + | text | TEXT | See warning [2](#2) below. | + | time | TIME | | + | timestamp | TIMESTAMPTZ | | + | tinyblob | BLOB | See warning [2](#2) below. | + | tinyint | INT | See warning [2](#2) below. | + | tinyint(1) | BOOLEAN | | + | tinytext | TEXT | See warning [2](#2) below. | + | varbinary | BLOB | See warning [2](#2) below. | + | varchar | TEXT | See warning [2](#2) below. | + + Data types not listed above are not supported. The following are some common data types that are not supported: + + - bigint unsigned + - bit(n) (n > 1) + - decimal + - enum + - geometry + - json + - numeric + - set + - year + + + | PostgreSQL/YugabyteDB | ScalarDB | Notes | + |--------------------------|-------------|----------------------------| + | bigint | BIGINT | See warning [1](#1) below. | + | boolean | BOOLEAN | | + | bytea | BLOB | | + | character | TEXT | See warning [2](#2) below. | + | character varying | TEXT | See warning [2](#2) below. | + | date | DATE | | + | double precision | DOUBLE | | + | integer | INT | | + | real | FLOAT | | + | smallint | INT | See warning [2](#2) below. | + | text | TEXT | | + | time | TIME | | + | timestamp | TIMESTAMP | | + | timestamp with time zone | TIMESTAMPTZ | | + + Data types not listed above are not supported. The following are some common data types that are not supported: + + - bigserial + - bit + - box + - cidr + - circle + - inet + - interval + - json + - jsonb + - line + - lseg + - macaddr + - macaddr8 + - money + - numeric + - path + - pg_lsn + - pg_snapshot + - point + - polygon + - serial + - smallserial + - time with time zone + - tsquery + - tsvector + - txid_snapshot + - uuid + - xml + + + | Oracle | ScalarDB | Notes | + |--------------------------------|-------------------------------------|----------------------------| + | binary_double | DOUBLE | | + | binary_float | FLOAT | | + | blob | BLOB | See warning [3](#3) below. | + | char | TEXT | See warning [2](#2) below. | + | clob | TEXT | | + | date | DATE (default), TIME, and TIMESTAMP | See warning [6](#6) below. | + | float | DOUBLE | See warning [4](#4) below. | + | long | TEXT | | + | long raw | BLOB | | + | nchar | TEXT | See warning [2](#2) below. | + | nclob | TEXT | | + | number | BIGINT / DOUBLE | See warning [5](#5) below. | + | nvarchar2 | TEXT | See warning [2](#2) below. | + | raw | BLOB | See warning [2](#2) below. | + | timestamp | TIMESTAMP (default) and TIME | See warning [6](#6) below. | + | timestamp with time zone | TIMESTAMPTZ | | + | timestamp with local time zone | TIMESTAMPTZ | | + | varchar2 | TEXT | See warning [2](#2) below. | + + Data types not listed above are not supported. The following are some common data types that are not supported: + + - interval + - rowid + - urowid + - bfile + - json + + + | SQL Server | ScalarDB | Notes | + |----------------|-------------|----------------------------| + | bigint | BIGINT | See warning [1](#1) below. | + | binary | BLOB | See warning [2](#2) below. | + | bit | BOOLEAN | | + | char | TEXT | See warning [2](#2) below. | + | date | DATE | | + | datetime | TIMESTAMP | + | datetime2 | TIMESTAMP | | + | float | DOUBLE | | + | image | BLOB | | + | int | INT | | + | nchar | TEXT | See warning [2](#2) below. | + | ntext | TEXT | | + | nvarchar | TEXT | See warning [2](#2) below. | + | offsetdatetime | TIMESTAMPTZ | | + | real | FLOAT | | + | smalldatetime | TIMESTAMP | | + | smallint | INT | See warning [2](#2) below. | + | text | TEXT | | + | time | TIME | | + | tinyint | INT | See warning [2](#2) below. | + | varbinary | BLOB | See warning [2](#2) below. | + | varchar | TEXT | See warning [2](#2) below. | + + Data types not listed above are not supported. The following are some common data types that are not supported: + + - cursor + - decimal + - geography + - geometry + - hierarchyid + - money + - numeric + - rowversion + - smallmoney + - sql_variant + - uniqueidentifier + - xml + + + | Db2 | ScalarDB | Notes | + |-----------------------|----------------------------------------|----------------------------| + | BIGINT | BIGINT | See warning [1](#1) below. | + | BINARY | BLOB | | + | BLOB | BLOB | | + | BOOLEAN | BOOLEAN | | + | CHAR | TEXT | | + | CHAR FOR BIT DATA | BLOB | | + | CLOB | TEXT | | + | DATE | DATE | | + | DOUBLE | DOUBLE | See warning [2](#2) below. | + | FLOAT(p), with p ≤ 24 | FLOAT | See warning [2](#2) below. | + | FLOAT(p), with p ≥ 25 | DOUBLE | See warning [2](#2) below. | + | GRAPHIC | TEXT | | + | INT | INT | | + | NCHAR | TEXT | | + | NCLOB | TEXT | | + | NVARCHAR | TEXT | | + | REAL | FLOAT | See warning [2](#2) below. | + | SMALLINT | INT | | + | TIME | TIME | | + | TIMESTAMP | TIMESTAMP (default), TIME, TIMESTAMPTZ | See warning [6](#6) below. | + | VARBINARY | BLOB | | + | VARCHAR | TEXT | | + | VARCHAR FOR BIT DATA | BLOB | | + | VARGRAPHIC | TEXT | | + + Data types not listed above are not supported. The following are some common data types that are not supported: + + - decimal + - decfloat + - xml + + + +:::warning + +
    +
  1. + The value range of `BIGINT` in ScalarDB is from -2^53 to 2^53, regardless of the size of `bigint` in the underlying database. Thus, if the data out of this range exists in the imported table, ScalarDB cannot read it. +
    2. +
  2. + For certain data types noted above, ScalarDB may map a data type larger than that of the underlying database. In that case, You will see errors when putting a value with a size larger than the size specified in the underlying database. +
    3. +
  3. + The maximum size of `BLOB` in ScalarDB is about 2GB (precisely 2^31-1 bytes). In contrast, Oracle `blob` can have (4GB-1)*(number of blocks). Thus, if data larger than 2GB exists in the imported table, ScalarDB cannot read it. +
    4. +
  4. + ScalarDB does not support Oracle `float` columns that have a higher precision than `DOUBLE` in ScalarDB. +
    5. +
  5. + ScalarDB does not support Oracle `numeric(p, s)` columns (`p` is precision and `s` is scale) when `p` is larger than 15 due to the maximum size of the data type in ScalarDB. Note that ScalarDB maps the column to `BIGINT` if `s` is zero; otherwise ScalarDB will map the column to `DOUBLE`. For the latter case, be aware that round-up or round-off can happen in the underlying database since the floating-point value will be cast to a fixed-point value. +
    6. +
  6. + The underlying storage type can be mapped to several ScalarDB data types. To override the default mapping, use the `override-columns-type` field in the import schema file. For an example, see [Sample import schema file](#sample-import-schema-file). +
    7. +
+ +::: + +## Use import function in your application + +You can use the import function in your application by using the following interfaces: + +- [ScalarDB Admin API](./api-guide.mdx#import-a-table) +- [ScalarDB Schema Loader API](./schema-loader.mdx#use-schema-loader-in-your-application) diff --git a/versioned_docs/version-3.X/schema-loader.mdx b/versioned_docs/version-3.X/schema-loader.mdx new file mode 100644 index 00000000..5270bb5e --- /dev/null +++ b/versioned_docs/version-3.X/schema-loader.mdx @@ -0,0 +1,705 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# ScalarDB Schema Loader + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +ScalarDB has its own data model and schema that maps to the implementation-specific data model and schema. In addition, ScalarDB stores internal metadata, such as transaction IDs, record versions, and transaction statuses, to manage transaction logs and statuses when you use the Consensus Commit transaction manager. + +Since managing the schema mapping and metadata for transactions can be difficult, you can use ScalarDB Schema Loader, which is a tool to create schemas that doesn't require you to need in-depth knowledge about schema mapping or metadata. + +You have two options to specify general CLI options in Schema Loader: + +- Pass the ScalarDB properties file and database-specific or storage-specific options. +- Pass database-specific or storage-specific options without the ScalarDB properties file. (Deprecated) + +:::note + +This tool supports only basic options to create, delete, repair, or alter a table. If you want to use the advanced features of a database, you must alter your tables with a database-specific tool after creating the tables with this tool. + +::: + +## Set up Schema Loader + +Select your preferred method to set up Schema Loader, and follow the instructions. + + + + You can download the release versions of Schema Loader from the [ScalarDB Releases](https://github.com/scalar-labs/scalardb/releases) page. + + + You can pull the Docker image from the [Scalar container registry](https://github.com/orgs/scalar-labs/packages/container/package/scalardb-schema-loader) by running the following command, replacing the contents in the angle brackets as described: + + ```console + docker run --rm -v :/scalardb.properties -v :/schema.json ghcr.io/scalar-labs/scalardb-schema-loader: --config /scalardb.properties --schema-file /schema.json + ``` + +:::note + +You can specify the same command arguments even if you use the fat JAR or the container. In the [Available commands](#available-commands) section, the JAR is used, but you can run the commands by using the container in the same way by replacing `java -jar scalardb-schema-loader-.jar` with `docker run --rm -v : [-v :] ghcr.io/scalar-labs/scalardb-schema-loader:`. + +::: + + + +## Run Schema Loader + +This section explains how to run Schema Loader. + +### Available commands + +Select how you would like to configure Schema Loader for your database. The preferred method is to use the properties file since other, database-specific methods are deprecated. + +The following commands are available when using the properties file: + +```console +Usage: java -jar scalardb-schema-loader-.jar [-D] [--coordinator] + [--no-backup] [--no-scaling] -c= + [--compaction-strategy=] [-f=] + [--replication-factor=] + [--replication-strategy=] [--ru=] +Create/Delete schemas in the storage defined in the config file + -A, --alter Alter tables : it will add new columns and create/delete + secondary index for existing tables. It compares the + provided table schema to the existing schema to decide + which columns need to be added and which indexes need + to be created or deleted + -c, --config= + Path to the config file of ScalarDB + --compaction-strategy= + The compaction strategy, must be LCS, STCS or TWCS + (supported in Cassandra) + --coordinator Create/delete/repair Coordinator tables + -D, --delete-all Delete tables + -f, --schema-file= + -I, --import Import tables : it will import existing non-ScalarDB + tables to ScalarDB. + Path to the schema json file + --no-backup Disable continuous backup (supported in DynamoDB) + --no-scaling Disable auto-scaling (supported in DynamoDB, Cosmos DB) + --repair-all Repair tables : it repairs the table metadata of + existing tables. When using Cosmos DB, it + additionally repairs stored procedure attached + to each table + --replication-factor= + The replication factor (supported in Cassandra) + --replication-strategy= + The replication strategy, must be SimpleStrategy or + NetworkTopologyStrategy (supported in Cassandra) + --ru= Base resource unit (supported in DynamoDB, Cosmos DB) +``` + +For a sample properties file, see [`database.properties`](https://github.com/scalar-labs/scalardb/blob/master/conf/database.properties). + +:::note + +The following database-specific methods have been deprecated. Please use the [commands for configuring the properties file](#available-commands) instead. + + + + ```console + Usage: java -jar scalardb-schema-loader-.jar --jdbc [-D] + -f= -j= -p= -u= + Create/Delete JDBC schemas + -A, --alter Alter tables : it will add new columns and create/delete + secondary index for existing tables. It compares the + provided table schema to the existing schema to decide + which columns need to be added and which indexes need + to be created or deleted + -D, --delete-all Delete tables + -f, --schema-file= + Path to the schema json file + -j, --jdbc-url= JDBC URL + -p, --password= + JDBC password + --repair-all Repair tables : it repairs the table metadata of + existing tables + -u, --user= JDBC user + ``` + + + ```console + Usage: java -jar scalardb-schema-loader-.jar --dynamo [-D] + [--no-backup] [--no-scaling] [--endpoint-override=] + -f= -p= [-r=] --region= + -u= + Create/Delete DynamoDB schemas + -A, --alter Alter tables : it will add new columns and create/delete + secondary index for existing tables. It compares the + provided table schema to the existing schema to decide + which columns need to be added and which indexes need + to be created or deleted + -D, --delete-all Delete tables + --endpoint-override= + Endpoint with which the DynamoDB SDK should + communicate + -f, --schema-file= + Path to the schema json file + --no-backup Disable continuous backup for DynamoDB + --no-scaling Disable auto-scaling for DynamoDB + -p, --password= AWS access secret key + -r, --ru= Base resource unit + --region= AWS region + --repair-all Repair tables : it repairs the table metadata of + existing tables + -u, --user= AWS access key ID + ``` + + + ```console + Usage: java -jar scalardb-schema-loader-.jar --cosmos [-D] + [--no-scaling] -f= -h= -p= [-r=] + Create/Delete Cosmos DB schemas + -A, --alter Alter tables : it will add new columns and create/delete + secondary index for existing tables. It compares the + provided table schema to the existing schema to decide + which columns need to be added and which indexes need + to be created or deleted + -D, --delete-all Delete tables + -f, --schema-file= + Path to the schema json file + -h, --host= Cosmos DB account URI + --no-scaling Disable auto-scaling for Cosmos DB + -p, --password= Cosmos DB key + -r, --ru= Base resource unit + --repair-all Repair tables : it repairs the table metadata of + existing tables and repairs stored procedure + attached to each table + ``` + + + ```console + Usage: java -jar scalardb-schema-loader-.jar --cassandra [-D] + [-c=] -f= -h= + [-n=] [-p=] [-P=] + [-R=] [-u=] + Create/Delete Cassandra schemas + -A, --alter Alter tables : it will add new columns and create/delete + secondary index for existing tables. It compares the + provided table schema to the existing schema to decide + which columns need to be added and which indexes need + to be created or deleted + -c, --compaction-strategy= + Cassandra compaction strategy, must be LCS, STCS or TWCS + -D, --delete-all Delete tables + -f, --schema-file= + Path to the schema json file + -h, --host= Cassandra host IP + -n, --network-strategy= + Cassandra network strategy, must be SimpleStrategy or + NetworkTopologyStrategy + -p, --password= + Cassandra password + -P, --port= Cassandra Port + -R, --replication-factor= + Cassandra replication factor + --repair-all Repair tables : it repairs the table metadata of + existing tables + -u, --user= Cassandra user + ``` + + + +::: + +### Create namespaces and tables + +To create namespaces and tables by using a properties file, run the following command, replacing the contents in the angle brackets as described: + +```console +java -jar scalardb-schema-loader-.jar --config -f [--coordinator] +``` + +If `--coordinator` is specified, a [Coordinator table](api-guide.mdx#specify-operations-for-the-coordinator-table) will be created. When the [group commit feature for the Coordinator table](api-guide.mdx#group-commit-for-coordinator-table) is enabled, this command creates a Coordinator table with additional columns for the group commit feature. + +:::note + +The following database-specific CLI arguments have been deprecated. Please use the CLI arguments for configuring the properties file instead. + + + + ```console + java -jar scalardb-schema-loader-.jar --jdbc -j -u -p -f + ``` + + + ```console + java -jar scalardb-schema-loader-.jar --dynamo -u -p --region -f [-r BASE_RESOURCE_UNIT] + ``` + + - `` should be a string to specify an AWS region like `ap-northeast-1`. + - `-r` option is almost the same as Cosmos DB for NoSQL option. However, the unit means DynamoDB capacity unit. The read and write capacity units are set the same value. + + + ```console + java -jar scalardb-schema-loader-.jar --cosmos -h -p -f [-r BASE_RESOURCE_UNIT] + ``` + + - `` you can use a primary key or a secondary key. + - `-r BASE_RESOURCE_UNIT` is an option. You can specify the RU of each database. The maximum RU in tables in the database will be set. If you don't specify RU of tables, the database RU will be set with this option. By default, it's 400. + + + ```console + java -jar scalardb-schema-loader-.jar --cassandra -h [-P ] [-u ] [-p ] -f [-n ] [-R ] + ``` + + - If `-P ` is not supplied, it defaults to `9042`. + - If `-u ` is not supplied, it defaults to `cassandra`. + - If `-p ` is not supplied, it defaults to `cassandra`. + - `` should be `SimpleStrategy` or `NetworkTopologyStrategy`. + + + +::: + +### Alter tables + +You can use a command to add new columns to and create or delete a secondary index for existing tables. This command compares the provided table schema to the existing schema to decide which columns need to be added and which indexes need to be created or deleted. + +To add new columns to and create or delete a secondary index for existing tables, run the following command, replacing the contents in the angle brackets as described: + +```console +java -jar scalardb-schema-loader-.jar --config -f --alter +``` + +:::note + +The following database-specific CLI arguments have been deprecated. Please use the CLI arguments for configuring the properties file instead. + + + + ```console + java -jar scalardb-schema-loader-.jar --jdbc -j -u -p -f --alter + ``` + + + ```console + java -jar scalardb-schema-loader-.jar --dynamo -u -p --region -f --alter + ``` + + + ```console + java -jar scalardb-schema-loader-.jar --cosmos -h -p -f --alter + ``` + + + ```console + java -jar scalardb-schema-loader-.jar --cassandra -h [-P ] [-u ] [-p ] -f --alter + ``` + + + +::: + +### Delete tables + +You can delete tables by using the properties file. To delete tables, run the following command, replacing the contents in the angle brackets as described: + +```console +java -jar scalardb-schema-loader-.jar --config -f [--coordinator] -D +``` + +If `--coordinator` is specified, the Coordinator table will be deleted as well. + +:::note + +The following database-specific CLI arguments have been deprecated. Please use the CLI arguments for configuring the properties file instead. + + + + ```console + java -jar scalardb-schema-loader-.jar --jdbc -j -u -p -f -D + ``` + + + ```console + java -jar scalardb-schema-loader-.jar --dynamo -u -p --region -f -D + ``` + + + ```console + java -jar scalardb-schema-loader-.jar --cosmos -h -p -f -D + ``` + + + ```console + java -jar scalardb-schema-loader-.jar --cassandra -h [-P ] [-u ] [-p ] -f -D + ``` + + + +::: + +### Repair tables + +You can repair the table metadata of existing tables by using the properties file. To repair table metadata of existing tables, run the following command, replacing the contents in the angle brackets as described: + +```console +java -jar scalardb-schema-loader-.jar --config -f [--coordinator] --repair-all +``` + +:::warning + +Before executing this command, you should confirm the following configurations are the same as those that were last applied: + +- The schema configuration +- Whether the [group commit feature for the Coordinator table](api-guide.mdx#group-commit-for-coordinator-table) is enabled or not, if the `--coordinator` option described below is specified + +::: + +If `--coordinator` is specified, the Coordinator table will be repaired as well. In addition, if you're using Cosmos DB for NoSQL, running this command will also repair stored procedures attached to each table. + +:::note + +The following database-specific CLI arguments have been deprecated. Please use the CLI arguments for configuring the properties file instead. + + + + ```console + java -jar scalardb-schema-loader-.jar --jdbc -j -u -p -f --repair-all + ``` + + + ```console + java -jar scalardb-schema-loader-.jar --dynamo -u -p --region [--no-backup] -f --repair-all + ``` + + + ```console + java -jar scalardb-schema-loader-.jar --cosmos -h -p -f --repair-all + ``` + + + ```console + java -jar scalardb-schema-loader-.jar --cassandra -h [-P ] [-u ] [-p ] -f --repair-all + ``` + + + +::: + +### Import tables + +You can import an existing table in JDBC databases to ScalarDB by using the `--import` option and an import-specific schema file. For details, see [Importing Existing Tables to ScalarDB by Using ScalarDB Schema Loader](./schema-loader-import.mdx). + +### Sample schema file + +The following is a sample schema. For a sample schema file, see [`schema_sample.json`](https://github.com/scalar-labs/scalardb/blob/master/schema-loader/sample/schema_sample.json). + +```json +{ + "sample_db.sample_table": { + "transaction": false, + "partition-key": [ + "c1" + ], + "clustering-key": [ + "c4 ASC", + "c6 DESC" + ], + "columns": { + "c1": "INT", + "c2": "TEXT", + "c3": "BLOB", + "c4": "INT", + "c5": "BOOLEAN", + "c6": "INT" + }, + "secondary-index": [ + "c2", + "c4" + ] + }, + + "sample_db.sample_table1": { + "transaction": true, + "partition-key": [ + "c1" + ], + "clustering-key": [ + "c4" + ], + "columns": { + "c1": "INT", + "c2": "TEXT", + "c3": "INT", + "c4": "INT", + "c5": "BOOLEAN" + } + }, + + "sample_db.sample_table2": { + "transaction": false, + "partition-key": [ + "c1" + ], + "clustering-key": [ + "c4", + "c3" + ], + "columns": { + "c1": "INT", + "c2": "TEXT", + "c3": "INT", + "c4": "INT", + "c5": "BOOLEAN" + } + } +} +``` + +The schema has table definitions that include `columns`, `partition-key`, `clustering-key`, `secondary-index`, and `transaction` fields. + +- The `columns` field defines columns of the table and their data types. +- The `partition-key` field defines which columns the partition key is composed of. +- The `clustering-key` field defines which columns the clustering key is composed of. +- The `secondary-index` field defines which columns are indexed. +- The `transaction` field indicates whether the table is for transactions or not. + - If you set the `transaction` field to `true` or don't specify the `transaction` field, this tool creates a table with transaction metadata if needed. + - If you set the `transaction` field to `false`, this tool creates a table without any transaction metadata (that is, for a table with [Storage API](run-non-transactional-storage-operations-through-primitive-crud-interface.mdx)). + +You can also specify database or storage-specific options in the table definition as follows: + +```json +{ + "sample_db.sample_table3": { + "partition-key": [ + "c1" + ], + "columns": { + "c1": "INT", + "c2": "TEXT", + "c3": "BLOB" + }, + "compaction-strategy": "LCS", + "ru": 5000 + } +} +``` + +The database or storage-specific options you can specify are as follows: + + + + No options are available for JDBC databases. + + + The `ru` option stands for Request Units. For details, see [RUs](#rus). + + + The `ru` option stands for Request Units. For details, see [RUs](#rus). + + + The `compaction-strategy` option is the compaction strategy used. This option should be `STCS` (SizeTieredCompaction), `LCS` (LeveledCompactionStrategy), or `TWCS` (TimeWindowCompactionStrategy). + + + +## Scale for performance when using Cosmos DB for NoSQL or DynamoDB + +When using Cosmos DB for NoSQL or DynamoDB, you can scale by using Request Units (RUs) or auto-scaling. + +### RUs + +You can scale the throughput of Cosmos DB for NoSQL and DynamoDB by specifying the `--ru` option. When specifying this option, scaling applies to all tables or the `ru` parameter for each table. + +If the `--ru` option is not set, the default values will be `400` for Cosmos DB for NoSQL and `10` for DynamoDB. + +:::note + +- Schema Loader abstracts [Request Units](https://docs.microsoft.com/azure/cosmos-db/request-units) for Cosmos DB for NoSQL and [Capacity Units](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.ReadWriteCapacityMode.html#HowItWorks.ProvisionedThroughput.Manual) for DynamoDB with `RU`. Therefore, be sure to set an appropriate value depending on the database implementation. +- Be aware that Schema Loader sets the same value to both read capacity unit and write capacity unit for DynamoDB. + +::: + +### Auto-scaling + +By default, Schema Loader enables auto-scaling of RUs for all tables: RUs scale between 10 percent and 100 percent of a specified RU depending on the workload. For example, if you specify `-r 10000`, the RUs of each table auto-scales between `1000` and `10000`. + +:::note + +Auto-scaling for Cosmos DB for NoSQL is enabled only when this option is set to `4000` or more. + +::: + +## Data-type mapping between ScalarDB and other databases + +The following table shows the supported data types in ScalarDB and their mapping to the data types of other databases. + +| ScalarDB | Cassandra | Cosmos DB for NoSQL | Db2 | DynamoDB | MySQL/MariaDB | PostgreSQL/YugabyteDB | Oracle | SQL Server | SQLite | +|-------------|----------------------|---------------------|------------------|----------|---------------|--------------------------|--------------------------|-----------------|---------| +| BOOLEAN | boolean | boolean (JSON) | BOOLEAN | BOOL | boolean | boolean | number(1) | bit | boolean | +| INT | int | number (JSON) | INT | N | int | int | number(10) | int | int | +| BIGINT | bigint | number (JSON) | BIGINT | N | bigint | bigint | number(16) | bigint | bigint | +| FLOAT | float | number (JSON) | REAL | N | real | real | binary_float | float(24) | float | +| DOUBLE | double | number (JSON) | DOUBLE | N | double | double precision | binary_double | float | double | +| TEXT | text | string (JSON) | VARCHAR(32672) | S | longtext | text | varchar2(4000) | varchar(8000) | text | +| BLOB | blob | string (JSON) | VARBINARY(32672) | B | longblob | bytea | RAW(2000) | varbinary(8000) | blob | +| DATE | date | number (JSON) | DATE | N | date | date | date | date | int | +| TIME | time | number (JSON) | TIMESTAMP | N | time | time | timestamp | time | bigint | +| TIMESTAMP | Unsupported | number (JSON) | TIMESTAMP | N | datetime | timestamp | timestamp | datetime2 | bigint | +| TIMESTAMPTZ | timestamp | number (JSON) | TIMESTAMP | N | datetime | timestamp with time zone | timestamp with time zone | datetimeoffset | bigint | + +:::note + +TIMESTAMP represents a date-time without time zone information, while TIMESTAMPTZ represents a date-time on the UTC time zone. + +::: + +However, the following data types in JDBC databases are converted differently when they are used as a primary key or a secondary index key. This is due to the limitations of RDB data types. For MySQL and Oracle, you can change the column size (minimum 64 bytes) as long as it meets the limitation of the total size of key columns. For details, see [Underlying storage or database configurations](configurations.mdx#underlying-storage-or-database-configurations). + +| ScalarDB | MySQL/MariaDB | PostgreSQL/YugabyteDB | Oracle | Db2 | +|----------|----------------|-----------------------|---------------|----------------| +| TEXT | VARCHAR(128) | VARCHAR(10485760) | VARCHAR2(128) | VARCHAR(128) | +| BLOB | VARBINARY(128) | | RAW(128) | VARBINARY(128) | + +The following data types have a value range and precision regardless of the underlying database. + +| Type | Minimum | Maximum | Precision | +|-------------|---------------------------|---------------------------|---------------| +| BIGINT | -2^53 | 2^53 | - | +| DATE | 1000-01-01 | 9999-12-31 | 1 day | +| TIME | 00:00:00.000000 | 23:59:59.999999 | 1 microsecond | +| TIMESTAMP | 1000-01-01 00:00:00.000 | 9999-12-31 23:59:59.999 | 1 millisecond | +| TIMESTAMPTZ | 1000-01-01 00:00:00.000 Z | 9999-12-31 23:59:59.999 Z | 1 millisecond | + +:::note + +YugabyteDB has limitations that prevent floating point types (FLOAT and DOUBLE) from functioning correctly as a primary key, clustering keys, or secondary index keys. + +::: + +If this data-type mapping doesn't match your application, please alter the tables to change the data types after creating them by using this tool. + +## Internal metadata for Consensus Commit + +The Consensus Commit transaction manager manages metadata (for example, transaction ID, record version, and transaction status) stored along with the actual records to handle transactions properly. + +Thus, along with any columns that the application requires, additional columns for the metadata need to be defined in the schema. Additionally, this tool creates a table with the metadata if you use the Consensus Commit transaction manager. + +## Use Schema Loader in your application + +You can check the version of Schema Loader from the [Maven Central Repository](https://mvnrepository.com/artifact/com.scalar-labs/scalardb-schema-loader). For example in Gradle, you can add the following dependency to your `build.gradle` file, replacing `` with the version of Schema Loader that you want to use: + +```gradle +dependencies { + implementation 'com.scalar-labs:scalardb-schema-loader:' +} +``` + +### Create, alter, repair, or delete tables + +You can create, alter, delete, or repair tables that are defined in the schema by using Schema Loader. To do this, you can pass a ScalarDB properties file, schema, and additional options, if needed, as shown below: + +```java +public class SchemaLoaderSample { + public static int main(String... args) throws SchemaLoaderException { + Path configFilePath = Paths.get("database.properties"); + // "sample_schema.json" and "altered_sample_schema.json" can be found in the "/sample" directory. + Path schemaFilePath = Paths.get("sample_schema.json"); + Path alteredSchemaFilePath = Paths.get("altered_sample_schema.json"); + boolean createCoordinatorTables = true; // whether to create the Coordinator table or not + boolean deleteCoordinatorTables = true; // whether to delete the Coordinator table or not + boolean repairCoordinatorTables = true; // whether to repair the Coordinator table or not + + Map tableCreationOptions = new HashMap<>(); + + tableCreationOptions.put( + CassandraAdmin.REPLICATION_STRATEGY, ReplicationStrategy.SIMPLE_STRATEGY.toString()); + tableCreationOptions.put(CassandraAdmin.COMPACTION_STRATEGY, CompactionStrategy.LCS.toString()); + tableCreationOptions.put(CassandraAdmin.REPLICATION_FACTOR, "1"); + + tableCreationOptions.put(DynamoAdmin.REQUEST_UNIT, "1"); + tableCreationOptions.put(DynamoAdmin.NO_SCALING, "true"); + tableCreationOptions.put(DynamoAdmin.NO_BACKUP, "true"); + + Map indexCreationOptions = new HashMap<>(); + indexCreationOptions.put(DynamoAdmin.NO_SCALING, "true"); + + Map tableReparationOptions = new HashMap<>(); + indexCreationOptions.put(DynamoAdmin.NO_BACKUP, "true"); + + // Create tables. + SchemaLoader.load(configFilePath, schemaFilePath, tableCreationOptions, createCoordinatorTables); + + // Alter tables. + SchemaLoader.alterTables(configFilePath, alteredSchemaFilePath, indexCreationOptions); + + // Repair tables. + SchemaLoader.repairTables(configFilePath, schemaFilePath, tableReparationOptions, repairCoordinatorTables); + + // Delete tables. + SchemaLoader.unload(configFilePath, schemaFilePath, deleteCoordinatorTables); + + return 0; + } +} +``` + +You can also create, delete, or repair a schema by passing a serialized-schema JSON string (the raw text of a schema file) as shown below: + +```java +// Create tables. +SchemaLoader.load(configFilePath, serializedSchemaJson, tableCreationOptions, createCoordinatorTables); + +// Alter tables. +SchemaLoader.alterTables(configFilePath, serializedAlteredSchemaFilePath, indexCreationOptions); + +// Repair tables. +SchemaLoader.repairTables(configFilePath, serializedSchemaJson, tableReparationOptions, repairCoordinatorTables); + +// Delete tables. +SchemaLoader.unload(configFilePath, serializedSchemaJson, deleteCoordinatorTables); +``` + +When configuring ScalarDB, you can use a `Properties` object as well, as shown below: + +```java +// Create tables. +SchemaLoader.load(properties, serializedSchemaJson, tableCreationOptions, createCoordinatorTables); + +// Alter tables. +SchemaLoader.alterTables(properties, serializedAlteredSchemaFilePath, indexCreationOptions); + +// Repair tables. +SchemaLoader.repairTables(properties, serializedSchemaJson, tableReparationOptions, repairCoordinatorTables); + +// Delete tables. +SchemaLoader.unload(properties, serializedSchemaJson, deleteCoordinatorTables); +``` + +### Import tables + +You can import an existing JDBC database table to ScalarDB by using the `--import` option and an import-specific schema file, in a similar manner as shown in [Sample schema file](#sample-schema-file). For details, see [Importing Existing Tables to ScalarDB by Using ScalarDB Schema Loader](./schema-loader-import.mdx). + +:::warning + +You should carefully plan to import a table to ScalarDB in production because it will add transaction metadata columns to your database tables and the ScalarDB metadata tables. In this case, there would also be several differences between your database and ScalarDB, as well as some limitations. + +::: + +The following is an import sample: + +```java +public class SchemaLoaderImportSample { + public static int main(String... args) throws SchemaLoaderException { + Path configFilePath = Paths.get("database.properties"); + // "import_sample_schema.json" can be found in the "/sample" directory. + Path schemaFilePath = Paths.get("import_sample_schema.json"); + Map tableImportOptions = new HashMap<>(); + + // Import tables. + // You can also use a Properties object instead of configFilePath and a serialized-schema JSON + // string instead of schemaFilePath. + SchemaLoader.importTables(configFilePath, schemaFilePath, tableImportOptions); + + return 0; + } +} +``` diff --git a/versioned_docs/version-3.X/slides/TransactionManagementOnCassandra.pdf b/versioned_docs/version-3.X/slides/TransactionManagementOnCassandra.pdf new file mode 100644 index 00000000..40e855bd Binary files /dev/null and b/versioned_docs/version-3.X/slides/TransactionManagementOnCassandra.pdf differ diff --git a/versioned_docs/version-3.X/two-phase-commit-transactions.mdx b/versioned_docs/version-3.X/two-phase-commit-transactions.mdx new file mode 100644 index 00000000..0f251d60 --- /dev/null +++ b/versioned_docs/version-3.X/two-phase-commit-transactions.mdx @@ -0,0 +1,745 @@ +--- +tags: + - Community + - Enterprise Standard + - Enterprise Premium +displayed_sidebar: docsEnglish +--- + +# Transactions with a Two-Phase Commit Interface + +ScalarDB supports executing transactions with a two-phase commit interface. With the two-phase commit interface, you can execute a transaction that spans multiple processes or applications, like in a microservice architecture. + +This page explains how transactions with a two-phase commit interface work in ScalarDB and how to configure and execute them in ScalarDB. + +## How transactions with a two-phase commit interface work in ScalarDB + +ScalarDB normally executes transactions in a single transaction manager instance with a one-phase commit interface. In transactions with a one-phase commit interface, you begin a transaction, execute CRUD operations, and commit the transaction in the same transaction manager instance. + +In ScalarDB, you can execute transactions with a two-phase commit interface that span multiple transaction manager instances. The transaction manager instances can be in the same process or application, or the instances can be in different processes or applications. For example, if you have transaction manager instances in multiple microservices, you can execute a transaction that spans multiple microservices. + +In transactions with a two-phase commit interface, there are two roles—Coordinator and a participant—that collaboratively execute a single transaction. + +The Coordinator process and the participant processes all have different transaction manager instances. The Coordinator process first begins or starts a transaction, and the participant processes join the transaction. After executing CRUD operations, the Coordinator process and the participant processes commit the transaction by using the two-phase interface. + +## How to execute transactions with a two-phase commit interface + +To execute a two-phase commit transaction, you must get the transaction manager instance. Then, the Coordinator process can begin or start the transaction, and the participant can process the transaction. + +### Get a `TwoPhaseCommitTransactionManager` instance + +You first need to get a `TwoPhaseCommitTransactionManager` instance to execute transactions with a two-phase commit interface. + +To get a `TwoPhaseCommitTransactionManager` instance, you can use `TransactionFactory` as follows: + +```java +TransactionFactory factory = TransactionFactory.create(""); +TwoPhaseCommitTransactionManager transactionManager = factory.getTwoPhaseCommitTransactionManager(); +``` + +### Begin or start a transaction (for Coordinator) + +For the process or application that begins the transaction to act as Coordinator, you should use the following `begin` method: + +```java +// Begin a transaction. +TwoPhaseCommitTransaction tx = transactionManager.begin(); +``` + +Or, for the process or application that begins the transaction to act as Coordinator, you should use the following `start` method: + +```java +// Start a transaction. +TwoPhaseCommitTransaction tx = transactionManager.start(); +``` + +Alternatively, you can use the `begin` method for a transaction by specifying a transaction ID as follows: + +```java +// Begin a transaction by specifying a transaction ID. +TwoPhaseCommitTransaction tx = transactionManager.begin(""); +``` + +Or, you can use the `start` method for a transaction by specifying a transaction ID as follows: + +```java +// Start a transaction by specifying a transaction ID. +TwoPhaseCommitTransaction tx = transactionManager.start(""); +``` + +### Join a transaction (for participants) + +For participants, you can join a transaction by specifying the transaction ID associated with the transaction that Coordinator has started or begun as follows: + +```java +TwoPhaseCommitTransaction tx = transactionManager.join(""); +``` + +:::note + +To get the transaction ID with `getId()`, you can specify the following: + +```java +tx.getId(); +``` + +::: + +### CRUD operations for the transaction + +The CRUD operations for `TwoPhaseCommitTransacton` are the same as the operations for `DistributedTransaction`. For details, see [CRUD operations](api-guide.mdx#crud-operations). + +The following is example code for CRUD operations in transactions with a two-phase commit interface: + +```java +TwoPhaseCommitTransaction tx = ... + +// Retrieve the current balances by ID. +Get fromGet = + Get.newBuilder() + .namespace(NAMESPACE) + .table(TABLE) + .partitionKey(new Key(ID, fromId)) + .build(); + +Get toGet = + Get.newBuilder() + .namespace(NAMESPACE) + .table(TABLE) + .partitionKey(new Key(ID, toId)) + .build(); + +Optional fromResult = tx.get(fromGet); +Optional toResult = tx.get(toGet); + +// Calculate the balances (assuming that both accounts exist). +int newFromBalance = fromResult.get().getInt(BALANCE) - amount; +int newToBalance = toResult.get().getInt(BALANCE) + amount; + +// Update the balances. +Put fromPut = + Put.newBuilder() + .namespace(NAMESPACE) + .table(TABLE) + .partitionKey(new Key(ID, fromId)) + .intValue(BALANCE, newFromBalance) + .build(); + +Put toPut = + Put.newBuilder() + .namespace(NAMESPACE) + .table(TABLE) + .partitionKey(new Key(ID, toId)) + .intValue(BALANCE, newToBalance) + .build(); + +tx.put(fromPut); +tx.put(toPut); +``` + +### Prepare, commit, or roll back a transaction + +After finishing CRUD operations, you need to commit the transaction. As with the standard two-phase commit protocol, there are two phases: prepare and commit. + +In all the Coordinator and participant processes, you need to prepare and then commit the transaction as follows: + +```java +TwoPhaseCommitTransaction tx = ... + +try { + // Execute CRUD operations in the Coordinator and participant processes. + ... + + // Prepare phase: Prepare the transaction in all the Coordinator and participant processes. + tx.prepare(); + ... + + // Commit phase: Commit the transaction in all the Coordinator and participant processes. + tx.commit(); + ... +} catch (TransactionException e) { + // If an error happens, you will need to roll back the transaction in all the Coordinator and participant processes. + tx.rollback(); + ... +} +``` + +For `prepare()`, if any of the Coordinator or participant processes fail to prepare the transaction, you will need to call `rollback()` (or `abort()`) in all the Coordinator and participant processes. + +For `commit()`, if any of the Coordinator or participant processes successfully commit the transaction, you can consider the transaction as committed. When a transaction has been committed, you can ignore any errors in the other Coordinator and participant processes. If all the Coordinator and participant processes fail to commit the transaction, you will need to call `rollback()` (or `abort()`) in all the Coordinator and participant processes. + +For better performance, you can call `prepare()`, `commit()`, and `rollback()` in the Coordinator and participant processes in parallel, respectively. + +#### Validate the transaction + +Depending on the concurrency control protocol, you need to call `validate()` in all the Coordinator and participant processes after `prepare()` and before `commit()`, as shown below: + +```java +// Prepare phase 1: Prepare the transaction in all the Coordinator and participant processes. +tx.prepare(); +... + +// Prepare phase 2: Validate the transaction in all the Coordinator and participant processes. +tx.validate(); +... + +// Commit phase: Commit the transaction in all the Coordinator and participant processes. +tx.commit(); +... +``` + +Similar to `prepare()`, if any of the Coordinator or participant processes fail to validate the transaction, you will need to call `rollback()` (or `abort()`) in all the Coordinator and participant processes. In addition, you can call `validate()` in the Coordinator and participant processes in parallel for better performance. + +:::note + +When using the [Consensus Commit](configurations.mdx#use-consensus-commit-directly) transaction manager with `EXTRA_READ` set as the value for `scalar.db.consensus_commit.serializable_strategy` and `SERIALIZABLE` set as the value for `scalar.db.consensus_commit.isolation_level`, you need to call `validate()`. However, if you are not using Consensus Commit, specifying `validate()` will not have any effect. + +::: + +### Execute a transaction by using multiple transaction manager instances + +By using the APIs described above, you can execute a transaction by using multiple transaction manager instances. The following is an example of how that works. + +:::warning + +The following sample code is a simplified version that solely illustrates how to use the two-phase commit interface and doesn't represent a real-world use case. In actual applications, it is assumed that a transaction manager instance is created per process or service. + +For a real-world example, see the [microservice transactions sample tutorial](scalardb-samples/microservice-transaction-sample/README.mdx). + +::: + +```java +TransactionFactory factory1 = + TransactionFactory.create(""); +TwoPhaseCommitTransactionManager transactionManager1 = + factory1.getTwoPhaseCommitTransactionManager(); + +TransactionFactory factory2 = + TransactionFactory.create(""); +TwoPhaseCommitTransactionManager transactionManager2 = + factory2.getTwoPhaseCommitTransactionManager(); + +TwoPhaseCommitTransaction transaction1 = null; +TwoPhaseCommitTransaction transaction2 = null; +try { + // Begin a transaction. + transaction1 = transactionManager1.begin(); + + // Join the transaction begun by `transactionManager1` by getting the transaction ID. + transaction2 = transactionManager2.join(transaction1.getId()); + + // Execute CRUD operations in the transaction. + Optional result = transaction1.get(...); + List results = transaction2.scan(...); + transaction1.put(...); + transaction2.delete(...); + + // Prepare the transaction. + transaction1.prepare(); + transaction2.prepare(); + + // Validate the transaction. + transaction1.validate(); + transaction2.validate(); + + // Commit the transaction. If any of the transactions successfully commit, + // you can regard the transaction as committed. + AtomicReference exception = new AtomicReference<>(); + boolean anyMatch = + Stream.of(transaction1, transaction2) + .anyMatch( + t -> { + try { + t.commit(); + return true; + } catch (TransactionException e) { + exception.set(e); + return false; + } + }); + + // If all the transactions fail to commit, throw the exception and roll back the transaction. + if (!anyMatch) { + throw exception.get(); + } +} catch (TransactionException e) { + // Roll back the transaction. + if (transaction1 != null) { + try { + transaction1.rollback(); + } catch (RollbackException e1) { + // Handle the exception. + } + } + if (transaction2 != null) { + try { + transaction2.rollback(); + } catch (RollbackException e1) { + // Handle the exception. + } + } +} +``` + +For simplicity, the above example code doesn't handle the exceptions that the APIs may throw. For details about handling exceptions, see [How to handle exceptions](#how-to-handle-exceptions). + +As previously mentioned, for `commit()`, if any of the Coordinator or participant processes succeed in committing the transaction, you can consider the transaction as committed. Also, for better performance, you can execute `prepare()`, `validate()`, and `commit()` in parallel, respectively. + +### Resume or re-join a transaction + +Given that processes or applications that use transactions with a two-phase commit interface usually involve multiple request and response exchanges, you might need to execute a transaction across various endpoints or APIs. For such scenarios, you can use `resume()` or `join()` to get a transaction object (an instance of `TwoPhaseCommitTransaction`) for the transaction that you previously joined. + +The following shows how `resume()` and `join()` work: + +```java +// Join (or begin) the transaction. +TwoPhaseCommitTransaction tx = transactionManager.join(""); + +... + +// Resume the transaction by using the transaction ID. +TwoPhaseCommitTransaction tx1 = transactionManager.resume(""); + +// Or you can re-join the transaction by using the transaction ID. +TwoPhaseCommitTransaction tx2 = transactionManager.join(""); +``` + +:::note + +To get the transaction ID with `getId()`, you can specify the following: + +```java +tx.getId(); +``` + +In addition, when using `join()` to re-join a transaction, if you have not joined the transaction before, a new transaction object will be returned. On the other hand, when using `resume()` to resume a transaction, if you have not joined the transaction before, `TransactionNotFoundException` will be thrown. + + +::: + +The following is an example of two services that have multiple endpoints: + +```java +interface ServiceA { + void facadeEndpoint() throws Exception; +} + +interface ServiceB { + void endpoint1(String txId) throws Exception; + + void endpoint2(String txId) throws Exception; + + void prepare(String txId) throws Exception; + + void commit(String txId) throws Exception; + + void rollback(String txId) throws Exception; +} +``` + +The following is an example of a client calling `ServiceA.facadeEndpoint()` that begins a transaction that spans the two services (`ServiceA` and `ServiceB`): + +```java +public class ServiceAImpl implements ServiceA { + + private TwoPhaseCommitTransactionManager transactionManager = ...; + private ServiceB serviceB = ...; + + ... + + @Override + public void facadeEndpoint() throws Exception { + TwoPhaseCommitTransaction tx = transactionManager.begin(); + + try { + ... + + // Call `ServiceB` `endpoint1`. + serviceB.endpoint1(tx.getId()); + + ... + + // Call `ServiceB` `endpoint2`. + serviceB.endpoint2(tx.getId()); + + ... + + // Prepare. + tx.prepare(); + serviceB.prepare(tx.getId()); + + // Commit. + tx.commit(); + serviceB.commit(tx.getId()); + } catch (Exception e) { + // Roll back. + tx.rollback(); + serviceB.rollback(tx.getId()); + } + } +} +``` + +As shown above, the facade endpoint in `ServiceA` calls multiple endpoints (`endpoint1()`, `endpoint2()`, `prepare()`, `commit()`, and `rollback()`) of `ServiceB`. In addition, in transactions with a two-phase commit interface, you need to use the same transaction object across the endpoints. + +In this situation, you can resume the transaction. The implementation of `ServiceB` is as follows: + +```java +public class ServiceBImpl implements ServiceB { + + private TwoPhaseCommitTransactionManager transactionManager = ...; + + ... + + @Override + public void endpoint1(String txId) throws Exception { + // Join the transaction. + TwoPhaseCommitTransaction tx = transactionManager.join(txId); + + ... + } + + @Override + public void endpoint2(String txId) throws Exception { + // Resume the transaction that you joined in `endpoint1()`. + TwoPhaseCommitTransaction tx = transactionManager.resume(txId); + + // Or re-join the transaction that you joined in `endpoint1()`. + // TwoPhaseCommitTransaction tx = transactionManager.join(txId); + + ... + } + + @Override + public void prepare(String txId) throws Exception { + // Resume the transaction. + TwoPhaseCommitTransaction tx = transactionManager.resume(txId); + + // Or re-join the transaction. + // TwoPhaseCommitTransaction tx = transactionManager.join(txId); + + ... + + // Prepare. + tx.prepare(); + } + + @Override + public void commit(String txId) throws Exception { + // Resume the transaction. + TwoPhaseCommitTransaction tx = transactionManager.resume(txId); + + // Or re-join the transaction. + // TwoPhaseCommitTransaction tx = transactionManager.join(txId); + + ... + + // Commit. + tx.commit(); + } + + @Override + public void rollback(String txId) throws Exception { + // Resume the transaction. + TwoPhaseCommitTransaction tx = transactionManager.resume(txId); + + // Or re-join the transaction. + // TwoPhaseCommitTransaction tx = transactionManager.join(txId); + + ... + + // Roll back. + tx.rollback(); + } +} +``` + +As shown above, by resuming or re-joining the transaction, you can share the same transaction object across multiple endpoints in `ServiceB`. + +## How to handle exceptions + +When executing a transaction by using multiple transaction manager instances, you will also need to handle exceptions properly. + +:::warning + +If you don't handle exceptions properly, you may face anomalies or data inconsistency. + +::: + +For instance, in the example code in [Execute a transaction by using multiple transaction manager instances](#execute-a-transaction-by-using-multiple-transaction-manager-instances), multiple transaction managers (`transactionManager1` and `transactionManager2`) are used in a single process for ease of explanation. However, that example code doesn't include a way to handle exceptions. + +The following example code shows how to handle exceptions in transactions with a two-phase commit interface: + +```java +public class Sample { + public static void main(String[] args) throws Exception { + TransactionFactory factory1 = + TransactionFactory.create(""); + TwoPhaseCommitTransactionManager transactionManager1 = + factory1.getTwoPhaseCommitTransactionManager(); + + TransactionFactory factory2 = + TransactionFactory.create(""); + TwoPhaseCommitTransactionManager transactionManager2 = + factory2.getTwoPhaseCommitTransactionManager(); + + int retryCount = 0; + TransactionException lastException = null; + + while (true) { + if (retryCount++ > 0) { + // Retry the transaction three times maximum in this sample code. + if (retryCount >= 3) { + // Throw the last exception if the number of retries exceeds the maximum. + throw lastException; + } + + // Sleep 100 milliseconds before retrying the transaction in this sample code. + TimeUnit.MILLISECONDS.sleep(100); + } + + TwoPhaseCommitTransaction transaction1 = null; + TwoPhaseCommitTransaction transaction2 = null; + try { + // Begin a transaction. + transaction1 = transactionManager1.begin(); + + // Join the transaction that `transactionManager1` begun by using the transaction ID. + transaction2 = transactionManager2.join(transaction1.getId()); + + // Execute CRUD operations in the transaction. + Optional result = transaction1.get(...); + List results = transaction2.scan(...); + transaction1.put(...); + transaction2.delete(...); + + // Prepare the transaction. + prepare(transaction1, transaction2); + + // Validate the transaction. + validate(transaction1, transaction2); + + // Commit the transaction. + commit(transaction1, transaction2); + } catch (UnknownTransactionStatusException e) { + // If you catch `UnknownTransactionStatusException` when committing the transaction, + // it indicates that the status of the transaction, whether it was successful or not, is unknown. + // In such a case, you need to check if the transaction is committed successfully or not and + // retry the transaction if it failed. How to identify a transaction status is delegated to users. + return; + } catch (TransactionException e) { + // For other exceptions, you can try retrying the transaction. + + // For `CrudConflictException`, `PreparationConflictException`, `ValidationConflictException`, + // `CommitConflictException`, and `TransactionNotFoundException`, you can basically retry the + // transaction. However, for the other exceptions, the transaction will still fail if the cause of + // the exception is non-transient. In such a case, you will exhaust the number of retries and + // throw the last exception. + + rollback(transaction1, transaction2); + + lastException = e; + } + } + } + + private static void prepare(TwoPhaseCommitTransaction... transactions) + throws TransactionException { + // You can execute `prepare()` in parallel. + List exceptions = + Stream.of(transactions) + .parallel() + .map( + t -> { + try { + t.prepare(); + return null; + } catch (TransactionException e) { + return e; + } + }) + .filter(Objects::nonNull) + .collect(Collectors.toList()); + + // If any of the transactions failed to prepare, throw the exception. + if (!exceptions.isEmpty()) { + throw exceptions.get(0); + } + } + + private static void validate(TwoPhaseCommitTransaction... transactions) + throws TransactionException { + // You can execute `validate()` in parallel. + List exceptions = + Stream.of(transactions) + .parallel() + .map( + t -> { + try { + t.validate(); + return null; + } catch (TransactionException e) { + return e; + } + }) + .filter(Objects::nonNull) + .collect(Collectors.toList()); + + // If any of the transactions failed to validate, throw the exception. + if (!exceptions.isEmpty()) { + throw exceptions.get(0); + } + } + + private static void commit(TwoPhaseCommitTransaction... transactions) + throws TransactionException { + // You can execute `commit()` in parallel. + List exceptions = + Stream.of(transactions) + .parallel() + .map( + t -> { + try { + t.commit(); + return null; + } catch (TransactionException e) { + return e; + } + }) + .filter(Objects::nonNull) + .collect(Collectors.toList()); + + // If any of the transactions successfully committed, you can regard the transaction as committed. + if (exceptions.size() < transactions.length) { + if (!exceptions.isEmpty()) { + // You can log the exceptions here if you want. + } + + return; // Commit was successful. + } + + // + // If all the transactions failed to commit: + // + + // If any of the transactions failed to commit due to `UnknownTransactionStatusException`, throw + // it because you should not retry the transaction in such a case. + Optional unknownTransactionStatusException = + exceptions.stream().filter(e -> e instanceof UnknownTransactionStatusException).findFirst(); + if (unknownTransactionStatusException.isPresent()) { + throw unknownTransactionStatusException.get(); + } + + // Otherwise, throw the first exception. + throw exceptions.get(0); + } + + private static void rollback(TwoPhaseCommitTransaction... transactions) { + Stream.of(transactions) + .parallel() + .filter(Objects::nonNull) + .forEach( + t -> { + try { + t.rollback(); + } catch (RollbackException e) { + // Rolling back the transaction failed. The transaction should eventually recover, + // so you don't need to do anything further. You can simply log the occurrence here. + } + }); + } +} +``` + +### `TransactionException` and `TransactionNotFoundException` + +The `begin()` API could throw `TransactionException` or `TransactionNotFoundException`: + +- If you catch `TransactionException`, this exception indicates that the transaction has failed to begin due to transient or non-transient faults. You can try retrying the transaction, but you may not be able to begin the transaction due to non-transient faults. +- If you catch `TransactionNotFoundException`, this exception indicates that the transaction has failed to begin due to transient faults. In this case, you can retry the transaction. + +The `join()` API could also throw `TransactionNotFoundException`. You can handle this exception in the same way that you handle the exceptions for the `begin()` API. + +### `CrudException` and `CrudConflictException` + +The APIs for CRUD operations (`get()`, `scan()`, `put()`, `delete()`, and `mutate()`) could throw `CrudException` or `CrudConflictException`: + +- If you catch `CrudException`, this exception indicates that the transaction CRUD operation has failed due to transient or non-transient faults. You can try retrying the transaction from the beginning, but the transaction will still fail if the cause is non-transient. +- If you catch `CrudConflictException`, this exception indicates that the transaction CRUD operation has failed due to transient faults (for example, a conflict error). In this case, you can retry the transaction from the beginning. + +### `PreparationException` and `PreparationConflictException` + +The `prepare()` API could throw `PreparationException` or `PreparationConflictException`: + +- If you catch `PreparationException`, this exception indicates that preparing the transaction fails due to transient or non-transient faults. You can try retrying the transaction from the beginning, but the transaction will still fail if the cause is non-transient. +- If you catch `PreparationConflictException`, this exception indicates that preparing the transaction has failed due to transient faults (for example, a conflict error). In this case, you can retry the transaction from the beginning. + +### `ValidationException` and `ValidationConflictException` + +The `validate()` API could throw `ValidationException` or `ValidationConflictException`: + +- If you catch `ValidationException`, this exception indicates that validating the transaction fails due to transient or non-transient faults. You can try retrying the transaction from the beginning, but the transaction will still fail if the cause is non-transient. +- If you catch `ValidationConflictException`, this exception indicates that validating the transaction has failed due to transient faults (for example, a conflict error). In this case, you can retry the transaction from the beginning. + +### `CommitException`, `CommitConflictException`, and `UnknownTransactionStatusException` + +The `commit()` API could throw `CommitException`, `CommitConflictException`, or `UnknownTransactionStatusException`: + +- If you catch `CommitException`, this exception indicates that committing the transaction fails due to transient or non-transient faults. You can try retrying the transaction from the beginning, but the transaction will still fail if the cause is non-transient. +- If you catch `CommitConflictException`, this exception indicates that committing the transaction has failed due to transient faults (for example, a conflict error). In this case, you can retry the transaction from the beginning. +- If you catch `UnknownTransactionStatusException`, this exception indicates that the status of the transaction, whether it was successful or not, is unknown. In this case, you need to check if the transaction is committed successfully and retry the transaction if it has failed. + +How to identify a transaction status is delegated to users. You may want to create a transaction status table and update it transactionally with other application data so that you can get the status of a transaction from the status table. + +### Notes about some exceptions + +Although not illustrated in the example code, the `resume()` API could also throw `TransactionNotFoundException`. This exception indicates that the transaction associated with the specified ID was not found and/or the transaction might have expired. In either case, you can retry the transaction from the beginning since the cause of this exception is basically transient. + +In the sample code, for `UnknownTransactionStatusException`, the transaction is not retried because the application must check if the transaction was successful to avoid potential duplicate operations. For other exceptions, the transaction is retried because the cause of the exception is transient or non-transient. If the cause of the exception is transient, the transaction may succeed if you retry it. However, if the cause of the exception is non-transient, the transaction will still fail even if you retry it. In such a case, you will exhaust the number of retries. + +:::note + +If you begin a transaction by specifying a transaction ID, you must use a different ID when you retry the transaction. + +In addition, in the sample code, the transaction is retried three times maximum and sleeps for 100 milliseconds before it is retried. But you can choose a retry policy, such as exponential backoff, according to your application requirements. + +::: + +## Request routing in transactions with a two-phase commit interface + +Services that use transactions with a two-phase commit interface usually execute a transaction by exchanging multiple requests and responses, as shown in the following diagram: + +![Sequence diagram for transactions with a two-phase commit interface](images/two_phase_commit_sequence_diagram.png) + +In addition, each service typically has multiple servers (or hosts) for scalability and availability and uses server-side (proxy) or client-side load balancing to distribute requests to the servers. In such a case, since transaction processing in transactions with a two-phase commit interface is stateful, requests in a transaction must be routed to the same servers while different transactions need to be distributed to balance the load, as shown in the following diagram: + +![Load balancing for transactions with a two-phase commit interface](images/two_phase_commit_load_balancing.png) + +There are several approaches to achieve load balancing for transactions with a two-phase commit interface depending on the protocol between the services. Some approaches for this include using gRPC, HTTP/1.1, and [ScalarDB Cluster](scalardb-cluster/index.mdx), which is a component that is available only in the ScalarDB Enterprise edition. + +### gRPC + +When you use a client-side load balancer, you can use the same gRPC connection to send requests in a transaction, which guarantees that the requests go to the same servers. + +When you use a server-side (proxy) load balancer, solutions are different between an L3/L4 (transport-level) load balancer and an L7 (application-level) load balancer: + +- When using an L3/L4 load balancer, you can use the same gRPC connection to send requests in a transaction, similar to when you use a client-side load balancer. In this case, requests in the same gRPC connection always go to the same server. +- When using an L7 load balancer, since requests in the same gRPC connection don't necessarily go to the same server, you need to use cookies or similar method to route requests to the correct server. + - For example, if you use [Envoy](https://www.envoyproxy.io/), you can use session affinity (sticky session) for gRPC. Alternatively, you can use [bidirectional streaming RPC in gRPC](https://grpc.io/docs/what-is-grpc/core-concepts/#bidirectional-streaming-rpc) since the L7 load balancer distributes requests in the same stream to the same server. + +For more details about load balancing in gRPC, see [gRPC Load Balancing](https://grpc.io/blog/grpc-load-balancing/). + +### HTTP/1.1 + +Typically, you use a server-side (proxy) load balancer with HTTP/1.1: + +- When using an L3/L4 load balancer, you can use the same HTTP connection to send requests in a transaction, which guarantees the requests go to the same server. +- When using an L7 load balancer, since requests in the same HTTP connection don't necessarily go to the same server, you need to use cookies or similar method to route requests to the correct server. You can use session affinity (sticky session) in that case. + +### ScalarDB Cluster + +ScalarDB Cluster addresses request routing by providing a routing mechanism that is capable of directing requests to the appropriate cluster node within the cluster. For details about ScalarDB Cluster, see [ScalarDB Cluster](scalardb-cluster/index.mdx). + +## Hands-on tutorial + +One of the use cases for transactions with a two-phase commit interface is microservice transactions. For a hands-on tutorial, see [Create a Sample Application That Supports Microservice Transactions](scalardb-samples/microservice-transaction-sample/README.mdx).