diff --git a/content/integrate/redis-data-integration/reference/_index.md b/content/integrate/redis-data-integration/reference/_index.md index 5d385b7135..897c34fe98 100644 --- a/content/integrate/redis-data-integration/reference/_index.md +++ b/content/integrate/redis-data-integration/reference/_index.md @@ -3,15 +3,16 @@ Title: Reference aliases: /integrate/redis-data-integration/ingest/reference/ alwaysopen: false categories: -- docs -- integrate -- rs -- rdi + - docs + - integrate + - rs + - rdi description: View reference material for Redis Data Integration group: di hideListLinks: false linkTitle: Reference -summary: Redis Data Integration keeps Redis in sync with the primary database in near +summary: + Redis Data Integration keeps Redis in sync with the primary database in near real time. type: integration weight: 60 diff --git a/content/integrate/redis-data-integration/reference/api-reference.md b/content/integrate/redis-data-integration/reference/api-reference.md new file mode 100644 index 0000000000..0708e20c7f --- /dev/null +++ b/content/integrate/redis-data-integration/reference/api-reference.md @@ -0,0 +1,24 @@ +--- +Title: Redis Data Integration API Reference +linkTitle: API Reference +description: REST API reference for Redis Data Integration +weight: 20 +alwaysopen: false +categories: ["redis-di"] +aliases: +--- + +# Redis Data Integration API + +This is the REST API reference for Redis Data Integration services. +The API provides endpoints to manage and monitor your Redis Data Integration deployments. + +[Open API Documentation]({{< relref "/integrate/redis-data-intrgration/reference/api/api-reference" >}}) + +## Authentication + +When authentication is enabled, you need to obtain a token using the login endpoint before accessing secured endpoints. + +## Base URL + +All API endpoints are relative to the base URL of your Redis Data Integration API server. diff --git a/content/integrate/redis-data-integration/reference/cli/_index.md b/content/integrate/redis-data-integration/reference/cli/_index.md index a27c40e535..e9554b86f1 100644 --- a/content/integrate/redis-data-integration/reference/cli/_index.md +++ b/content/integrate/redis-data-integration/reference/cli/_index.md @@ -3,15 +3,16 @@ Title: CLI reference aliases: /integrate/redis-data-integration/ingest/reference/cli/ alwaysopen: false categories: -- docs -- integrate -- rs -- rdi + - docs + - integrate + - rs + - rdi description: Reference for the RDI CLI commands group: di hideListLinks: false linkTitle: CLI commands -summary: Redis Data Integration keeps Redis in sync with the primary database in near +summary: + Redis Data Integration keeps Redis in sync with the primary database in near real time. type: integration weight: 60 diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-add-context.md b/content/integrate/redis-data-integration/reference/cli/redis-di-add-context.md index 72e5221569..2b4cf36c77 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-add-context.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-add-context.md @@ -1,13 +1,15 @@ --- Title: redis-di add-context linkTitle: redis-di add-context -description: Adds a new context +description: Adds a new context weight: 10 alwaysopen: false categories: ["redis-di"] aliases: --- +Adds a new context + ## Usage ``` @@ -15,104 +17,84 @@ Usage: redis-di add-context [OPTIONS] CONTEXT_NAME ``` ## Options -* `context_name` (REQUIRED): - * Type: STRING - * Default: `none` - * Usage: `context-name` - - - - -* `log_level`: - * Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL']) - * Default: `info` - * Usage: `--log-level --l` - - +- `context_name` (REQUIRED): -* `cluster_host` (REQUIRED): - * Type: STRING - * Default: `none` - * Usage: `--cluster-host` + - Type: STRING + - Default: `none` + - Usage: `context-name` - Host/IP of Redis Enterprise Cluster (service name in case of k8s) +- `log_level`: + - Type: Choice(['TRACE', 'DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']) + - Default: `info` + - Usage: `--log-level +-l` -* `cluster_api_port` (REQUIRED): - * Type: - * Default: `9443` - * Usage: `--cluster-api-port` - - API Port of Redis Enterprise Cluster - +- `rdi_namespace`: -* `cluster_user` (REQUIRED): - * Type: STRING - * Default: `none` - * Usage: `--cluster-user` + - Type: STRING + - Default: `rdi` + - Usage: `--rdi-namespace` - Redis Enterprise Cluster username with either DB Member, Cluster Member or Cluster Admin roles + RDI Kubernetes namespace +- `rdi_host` (REQUIRED): -* `rdi_host` (REQUIRED): - * Type: STRING - * Default: `none` - * Usage: `--rdi-host` + - Type: STRING + - Default: `none` + - Usage: `--rdi-host` Host/IP of RDI Database +- `rdi_port` (REQUIRED): -* `rdi_port` (REQUIRED): - * Type: - * Default: `none` - * Usage: `--rdi-port` + - Type: + - Default: `none` + - Usage: `--rdi-port` Port of RDI Database +- `rdi_user`: -* `rdi_user`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-user` + - Type: STRING + - Default: `none` + - Usage: `--rdi-user` RDI Database Username +- `rdi_key`: -* `rdi_key`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-key` + - Type: STRING + - Default: `none` + - Usage: `--rdi-key` Private key file to authenticate with +- `rdi_cert`: -* `rdi_cert`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-cert` + - Type: STRING + - Default: `none` + - Usage: `--rdi-cert` Client certificate file to authenticate with +- `rdi_cacert`: -* `rdi_cacert`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-cacert` + - Type: STRING + - Default: `none` + - Usage: `--rdi-cacert` CA certificate file to verify with +- `help`: -* `help`: - * Type: BOOL - * Default: `false` - * Usage: `--help` + - Type: BOOL + - Default: `false` + - Usage: `--help` Show this message and exit. - - ## CLI help ``` @@ -121,16 +103,9 @@ Usage: redis-di add-context [OPTIONS] CONTEXT_NAME Adds a new context Options: - -l, --log-level [DEBUG|INFO|WARN|ERROR|CRITICAL] + -l, --log-level [TRACE|DEBUG|INFO|WARNING|ERROR|CRITICAL] [default: INFO] - --cluster-host TEXT Host/IP of Redis Enterprise Cluster (service - name in case of k8s) [required] - --cluster-api-port INTEGER RANGE - API Port of Redis Enterprise Cluster - [default: 9443; 1<=x<=65535; required] - --cluster-user TEXT Redis Enterprise Cluster username with - either DB Member, Cluster Member or Cluster - Admin roles [required] + --rdi-namespace TEXT RDI Kubernetes namespace [default: rdi] --rdi-host TEXT Host/IP of RDI Database [required] --rdi-port INTEGER RANGE Port of RDI Database [1<=x<=65535; required] diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-config-rdi.md b/content/integrate/redis-data-integration/reference/cli/redis-di-config-rdi.md deleted file mode 100644 index b9d8ac55d8..0000000000 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-config-rdi.md +++ /dev/null @@ -1,121 +0,0 @@ ---- -Title: redis-di config-rdi -linkTitle: redis-di config-rdi -description: Configures RDI DB connection credentials -weight: 10 -alwaysopen: false -categories: ["redis-di"] -aliases: ---- - -## Usage - -``` -Usage: redis-di config-rdi [OPTIONS] -``` - -## Options -* `log_level`: - * Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL']) - * Default: `info` - * Usage: `--log-level --l` - - - - -* `rdi_host` (REQUIRED): - * Type: STRING - * Default: `none` - * Usage: `--rdi-host` - - Host/IP of RDI Database - - -* `rdi_port` (REQUIRED): - * Type: - * Default: `none` - * Usage: `--rdi-port` - - Port of RDI Database - - -* `rdi_user`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-user` - - RDI Database Username - - -* `rdi_password`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-password` - - RDI Database Password - - -* `rdi_key`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-key` - - Private key file to authenticate with - - -* `rdi_cert`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-cert` - - Client certificate file to authenticate with - - -* `rdi_cacert`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-cacert` - - CA certificate file to verify with - - -* `rdi_key_password`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-key-password` - - Password for unlocking an encrypted private key - - -* `help`: - * Type: BOOL - * Default: `false` - * Usage: `--help` - - Show this message and exit. - - - -## CLI help - -``` -Usage: redis-di config-rdi [OPTIONS] - - Configures RDI db connection credentials - -Options: - -l, --log-level [DEBUG|INFO|WARN|ERROR|CRITICAL] - [default: INFO] - --rdi-host TEXT Host/IP of RDI Database [required] - --rdi-port INTEGER RANGE Port of RDI Database [1<=x<=65535; - required] - --rdi-user TEXT RDI Database Username - --rdi-password TEXT RDI Database Password - --rdi-key TEXT Private key file to authenticate with - --rdi-cert TEXT Client certificate file to authenticate with - --rdi-cacert TEXT CA certificate file to verify with - --rdi-key-password TEXT Password for unlocking an encrypted private - key - --help Show this message and exit. -``` diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-configure-rdi.md b/content/integrate/redis-data-integration/reference/cli/redis-di-configure-rdi.md new file mode 100644 index 0000000000..d3dde4555e --- /dev/null +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-configure-rdi.md @@ -0,0 +1,140 @@ +--- +Title: redis-di configure-rdi +linkTitle: redis-di configure-rdi +description: Configures RDI db connection credentials +weight: 10 +alwaysopen: false +categories: ["redis-di"] +aliases: +--- + +Configures RDI db connection credentials + +## Usage + +``` +Usage: redis-di configure-rdi [OPTIONS] +``` + +## Options + +- `log_level`: + + - Type: Choice(['TRACE', 'DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']) + - Default: `info` + - Usage: `--log-level +-l` + +- `rdi_namespace`: + + - Type: STRING + - Default: `rdi` + - Usage: `--rdi-namespace` + + RDI Kubernetes namespace + +- `rdi_host` (REQUIRED): + + - Type: STRING + - Default: `none` + - Usage: `--rdi-host` + + Host/IP of RDI Database + +- `rdi_port` (REQUIRED): + + - Type: + - Default: `none` + - Usage: `--rdi-port` + + Port of RDI Database + +- `rdi_user`: + + - Type: STRING + - Default: `none` + - Usage: `--rdi-user` + + RDI Database Username + +- `rdi_password`: + + - Type: STRING + - Default: `none` + - Usage: `--rdi-password` + + RDI Database Password + +- `rdi_key`: + + - Type: STRING + - Default: `none` + - Usage: `--rdi-key` + + Private key file to authenticate with + +- `rdi_cert`: + + - Type: STRING + - Default: `none` + - Usage: `--rdi-cert` + + Client certificate file to authenticate with + +- `rdi_cacert`: + + - Type: STRING + - Default: `none` + - Usage: `--rdi-cacert` + + CA certificate file to verify with + +- `rdi_key_password`: + + - Type: STRING + - Default: `none` + - Usage: `--rdi-key-password` + + Password for unlocking an encrypted private key + +- `rdi_log_level`: + + - Type: Choice(['TRACE', 'DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']) + - Default: `none` + - Usage: `--rdi-log-level` + + Log level for RDI components + +- `help`: + + - Type: BOOL + - Default: `false` + - Usage: `--help` + + Show this message and exit. + +## CLI help + +``` +Usage: redis-di configure-rdi [OPTIONS] + + Configures RDI db connection credentials + +Options: + -l, --log-level [TRACE|DEBUG|INFO|WARNING|ERROR|CRITICAL] + [default: INFO] + --rdi-namespace TEXT RDI Kubernetes namespace [default: rdi] + --rdi-host TEXT Host/IP of RDI Database [required] + --rdi-port INTEGER RANGE Port of RDI Database [1<=x<=65535; + required] + --rdi-user TEXT RDI Database Username + --rdi-password TEXT RDI Database Password + --rdi-key TEXT Private key file to authenticate with + --rdi-cert TEXT Client certificate file to authenticate with + --rdi-cacert TEXT CA certificate file to verify with + --rdi-key-password TEXT Password for unlocking an encrypted private + key + --rdi-log-level [TRACE|DEBUG|INFO|WARNING|ERROR|CRITICAL] + Log level for RDI components + --help Show this message and exit. +``` diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-create.md b/content/integrate/redis-data-integration/reference/cli/redis-di-create.md new file mode 100644 index 0000000000..80b641d901 --- /dev/null +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-create.md @@ -0,0 +1,146 @@ +--- +Title: redis-di create +linkTitle: redis-di create +description: Creates the RDI Database instance +weight: 10 +alwaysopen: false +categories: ["redis-di"] +aliases: +--- + +Creates the RDI Database instance + +## Usage + +``` +Usage: redis-di create [OPTIONS] +``` + +## Options + +- `log_level`: + + - Type: Choice(['TRACE', 'DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']) + - Default: `info` + - Usage: `--log-level +-l` + +- `silent`: + + - Type: BOOL + - Default: `false` + - Usage: `--silent` + + Silent install. Do not prompt to enter missing parameters + +- `cluster_host` (REQUIRED): + + - Type: STRING + - Default: `none` + - Usage: `--cluster-host` + + Host/IP of Redis Enterprise Cluster (service name in case of k8s) + +- `cluster_api_port` (REQUIRED): + + - Type: + - Default: `9443` + - Usage: `--cluster-api-port` + + API Port of Redis Enterprise Cluster + +- `cluster_user` (REQUIRED): + + - Type: STRING + - Default: `none` + - Usage: `--cluster-user` + + Redis Enterprise Cluster username with either DB Member, Cluster Member or Cluster Admin roles + +- `cluster_password`: + + - Type: STRING + - Default: `none` + - Usage: `--cluster-password` + + Redis Enterprise Cluster Password + +- `rdi_port`: + + - Type: + - Default: `none` + - Usage: `--rdi-port` + + Port for the new RDI Database + +- `rdi_password`: + + - Type: STRING + - Default: `` + - Usage: `--rdi-password` + + Password for the new RDI Database (alphanumeric characters with zero or more of the following: ! & # $ ^ < > -) + +- `rdi_memory`: + + - Type: =30> + - Default: `100` + - Usage: `--rdi-memory` + + Memory for RDI Database (in MB) + +- `rdi_shards`: + + - Type: =1> + - Default: `1` + - Usage: `--rdi-shards` + + Number of database server-side shards + +- `replication`: + + - Type: BOOL + - Default: `false` + - Usage: `--replication` + + In-memory database replication + +- `help`: + + - Type: BOOL + - Default: `false` + - Usage: `--help` + + Show this message and exit. + +## CLI help + +``` +Usage: redis-di create [OPTIONS] + + Creates the RDI Database instance + +Options: + -l, --log-level [TRACE|DEBUG|INFO|WARNING|ERROR|CRITICAL] + [default: INFO] + --silent Silent install. Do not prompt to enter + missing parameters + --cluster-host TEXT Host/IP of Redis Enterprise Cluster (service + name in case of k8s) [required] + --cluster-api-port INTEGER RANGE + API Port of Redis Enterprise Cluster + [default: 9443; 1<=x<=65535; required] + --cluster-user TEXT Redis Enterprise Cluster username with + either DB Member, Cluster Member or Cluster + Admin roles [required] + --cluster-password TEXT Redis Enterprise Cluster Password + --rdi-port INTEGER RANGE Port for the new RDI Database [1<=x<=65535] + --rdi-password TEXT Password for the new RDI Database + (alphanumeric characters with zero or more + of the following: ! & # $ ^ < > -) + --rdi-memory INTEGER RANGE Memory for RDI Database (in MB) [x>=30] + --rdi-shards INTEGER RANGE Number of database server-side shards + [x>=1] + --replication In-memory database replication + --help Show this message and exit. +``` diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-delete-all-contexts.md b/content/integrate/redis-data-integration/reference/cli/redis-di-delete-all-contexts.md index dbca215a36..1f7792f2d7 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-delete-all-contexts.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-delete-all-contexts.md @@ -1,13 +1,15 @@ --- Title: redis-di delete-all-contexts linkTitle: redis-di delete-all-contexts -description: Deletes all contexts +description: Deletes all contexts weight: 10 alwaysopen: false categories: ["redis-di"] aliases: --- +Deletes all contexts + ## Usage ``` @@ -15,33 +17,31 @@ Usage: redis-di delete-all-contexts [OPTIONS] ``` ## Options -* `log_level`: - * Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL']) - * Default: `info` - * Usage: `--log-level --l` - +- `log_level`: + + - Type: Choice(['TRACE', 'DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']) + - Default: `info` + - Usage: `--log-level +-l` +- `force`: -* `force`: - * Type: BOOL - * Default: `false` - * Usage: `--force + - Type: BOOL + - Default: `false` + - Usage: `--force -f` Force operation. skips verification prompts +- `help`: -* `help`: - * Type: BOOL - * Default: `false` - * Usage: `--help` + - Type: BOOL + - Default: `false` + - Usage: `--help` Show this message and exit. - - ## CLI help ``` @@ -50,7 +50,7 @@ Usage: redis-di delete-all-contexts [OPTIONS] Deletes all contexts Options: - -l, --log-level [DEBUG|INFO|WARN|ERROR|CRITICAL] + -l, --log-level [TRACE|DEBUG|INFO|WARNING|ERROR|CRITICAL] [default: INFO] -f, --force Force operation. skips verification prompts --help Show this message and exit. diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-delete-context.md b/content/integrate/redis-data-integration/reference/cli/redis-di-delete-context.md index d1b6d5968c..5bbef80c32 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-delete-context.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-delete-context.md @@ -1,13 +1,15 @@ --- Title: redis-di delete-context linkTitle: redis-di delete-context -description: Deletes a context +description: Deletes a context weight: 10 alwaysopen: false categories: ["redis-di"] aliases: --- +Deletes a context + ## Usage ``` @@ -15,41 +17,37 @@ Usage: redis-di delete-context [OPTIONS] CONTEXT_NAME ``` ## Options -* `log_level`: - * Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL']) - * Default: `info` - * Usage: `--log-level --l` - +- `log_level`: + - Type: Choice(['TRACE', 'DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']) + - Default: `info` + - Usage: `--log-level +-l` -* `context_name` (REQUIRED): - * Type: STRING - * Default: `none` - * Usage: `context-name` +- `context_name` (REQUIRED): - + - Type: STRING + - Default: `none` + - Usage: `context-name` +- `force`: -* `force`: - * Type: BOOL - * Default: `false` - * Usage: `--force + - Type: BOOL + - Default: `false` + - Usage: `--force -f` Force operation. skips verification prompts +- `help`: -* `help`: - * Type: BOOL - * Default: `false` - * Usage: `--help` + - Type: BOOL + - Default: `false` + - Usage: `--help` Show this message and exit. - - ## CLI help ``` @@ -58,7 +56,7 @@ Usage: redis-di delete-context [OPTIONS] CONTEXT_NAME Deletes a context Options: - -l, --log-level [DEBUG|INFO|WARN|ERROR|CRITICAL] + -l, --log-level [TRACE|DEBUG|INFO|WARNING|ERROR|CRITICAL] [default: INFO] -f, --force Force operation. skips verification prompts --help Show this message and exit. diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-delete.md b/content/integrate/redis-data-integration/reference/cli/redis-di-delete.md deleted file mode 100644 index 4c93fa29e1..0000000000 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-delete.md +++ /dev/null @@ -1,172 +0,0 @@ ---- -Title: redis-di delete -linkTitle: redis-di delete -description: Deletes RDI database permanently -weight: 10 -alwaysopen: false -categories: ["redis-di"] -aliases: ---- - -## Usage - -``` -Usage: redis-di delete [OPTIONS] -``` - -## Options -* `log_level`: - * Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL']) - * Default: `info` - * Usage: `--log-level --l` - - - - -* `cluster_host` (REQUIRED): - * Type: STRING - * Default: `none` - * Usage: `--cluster-host` - - Host/IP of Redis Enterprise Cluster (service name in case of k8s) - - -* `cluster_api_port` (REQUIRED): - * Type: - * Default: `9443` - * Usage: `--cluster-api-port` - - API Port of Redis Enterprise Cluster - - -* `cluster_user` (REQUIRED): - * Type: STRING - * Default: `none` - * Usage: `--cluster-user` - - Redis Enterprise Cluster username with either DB Member, Cluster Member or Cluster Admin roles - - -* `cluster_password`: - * Type: STRING - * Default: `none` - * Usage: `--cluster-password` - - Redis Enterprise Cluster Password - - -* `rdi_host` (REQUIRED): - * Type: STRING - * Default: `none` - * Usage: `--rdi-host` - - Host/IP of RDI Database - - -* `rdi_port` (REQUIRED): - * Type: - * Default: `none` - * Usage: `--rdi-port` - - Port of RDI Database - - -* `rdi_user`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-user` - - RDI Database Username - - -* `rdi_password`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-password` - - RDI Database Password - - -* `rdi_key`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-key` - - Private key file to authenticate with - - -* `rdi_cert`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-cert` - - Client certificate file to authenticate with - - -* `rdi_cacert`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-cacert` - - CA certificate file to verify with - - -* `rdi_key_password`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-key-password` - - Password for unlocking an encrypted private key - - -* `force`: - * Type: BOOL - * Default: `false` - * Usage: `--force --f` - - Force operation. skips verification prompts - - -* `help`: - * Type: BOOL - * Default: `false` - * Usage: `--help` - - Show this message and exit. - - - -## CLI help - -``` -Usage: redis-di delete [OPTIONS] - - Deletes RDI database permanently - -Options: - -l, --log-level [DEBUG|INFO|WARN|ERROR|CRITICAL] - [default: INFO] - --cluster-host TEXT Host/IP of Redis Enterprise Cluster (service - name in case of k8s) [required] - --cluster-api-port INTEGER RANGE - API Port of Redis Enterprise Cluster - [default: 9443; 1<=x<=65535; required] - --cluster-user TEXT Redis Enterprise Cluster username with - either DB Member, Cluster Member or Cluster - Admin roles [required] - --cluster-password TEXT Redis Enterprise Cluster Password - --rdi-host TEXT Host/IP of RDI Database [required] - --rdi-port INTEGER RANGE Port of RDI Database [1<=x<=65535; - required] - --rdi-user TEXT RDI Database Username - --rdi-password TEXT RDI Database Password - --rdi-key TEXT Private key file to authenticate with - --rdi-cert TEXT Client certificate file to authenticate with - --rdi-cacert TEXT CA certificate file to verify with - --rdi-key-password TEXT Password for unlocking an encrypted private - key - -f, --force Force operation. skips verification prompts - --help Show this message and exit. -``` diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-deploy.md b/content/integrate/redis-data-integration/reference/cli/redis-di-deploy.md index 72e13a145f..f191a4d98d 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-deploy.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-deploy.md @@ -1,13 +1,15 @@ --- Title: redis-di deploy linkTitle: redis-di deploy -description: Deploys the RDI configurations including target +description: Deploys the RDI configurations including target weight: 10 alwaysopen: false categories: ["redis-di"] aliases: --- +Deploys the RDI configurations including target + ## Usage ``` @@ -15,96 +17,94 @@ Usage: redis-di deploy [OPTIONS] ``` ## Options -* `log_level`: - * Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL']) - * Default: `info` - * Usage: `--log-level --l` - +- `log_level`: + + - Type: Choice(['TRACE', 'DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']) + - Default: `info` + - Usage: `--log-level +-l` +- `rdi_host` (REQUIRED): -* `rdi_host` (REQUIRED): - * Type: STRING - * Default: `none` - * Usage: `--rdi-host` + - Type: STRING + - Default: `none` + - Usage: `--rdi-host` Host/IP of RDI Database +- `rdi_port` (REQUIRED): -* `rdi_port` (REQUIRED): - * Type: - * Default: `none` - * Usage: `--rdi-port` + - Type: + - Default: `none` + - Usage: `--rdi-port` Port of RDI Database +- `rdi_user`: -* `rdi_user`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-user` + - Type: STRING + - Default: `none` + - Usage: `--rdi-user` RDI Database Username +- `rdi_password`: -* `rdi_password`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-password` + - Type: STRING + - Default: `none` + - Usage: `--rdi-password` RDI Database Password +- `rdi_key`: -* `rdi_key`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-key` + - Type: STRING + - Default: `none` + - Usage: `--rdi-key` Private key file to authenticate with +- `rdi_cert`: -* `rdi_cert`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-cert` + - Type: STRING + - Default: `none` + - Usage: `--rdi-cert` Client certificate file to authenticate with +- `rdi_cacert`: -* `rdi_cacert`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-cacert` + - Type: STRING + - Default: `none` + - Usage: `--rdi-cacert` CA certificate file to verify with +- `rdi_key_password`: -* `rdi_key_password`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-key-password` + - Type: STRING + - Default: `none` + - Usage: `--rdi-key-password` Password for unlocking an encrypted private key +- `directory`: -* `directory`: - * Type: STRING - * Default: `.` - * Usage: `--dir` + - Type: STRING + - Default: `.` + - Usage: `--dir` Directory containing RDI configuration +- `help`: -* `help`: - * Type: BOOL - * Default: `false` - * Usage: `--help` + - Type: BOOL + - Default: `false` + - Usage: `--help` Show this message and exit. - - ## CLI help ``` @@ -113,7 +113,7 @@ Usage: redis-di deploy [OPTIONS] Deploys the RDI configurations including target Options: - -l, --log-level [DEBUG|INFO|WARN|ERROR|CRITICAL] + -l, --log-level [TRACE|DEBUG|INFO|WARNING|ERROR|CRITICAL] [default: INFO] --rdi-host TEXT Host/IP of RDI Database [required] --rdi-port INTEGER RANGE Port of RDI Database [1<=x<=65535; diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-describe-job.md b/content/integrate/redis-data-integration/reference/cli/redis-di-describe-job.md index 6651edf65c..2541269c23 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-describe-job.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-describe-job.md @@ -1,13 +1,15 @@ --- Title: redis-di describe-job linkTitle: redis-di describe-job -description: Describes a transformation engine's job +description: Describes a transformation engine's job weight: 10 alwaysopen: false categories: ["redis-di"] aliases: --- +Describes a transformation engine's job + ## Usage ``` @@ -15,96 +17,92 @@ Usage: redis-di describe-job [OPTIONS] JOB_NAME ``` ## Options -* `log_level`: - * Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL']) - * Default: `info` - * Usage: `--log-level --l` - +- `log_level`: + - Type: Choice(['TRACE', 'DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']) + - Default: `info` + - Usage: `--log-level +-l` -* `job_name` (REQUIRED): - * Type: STRING - * Default: `none` - * Usage: `job-name` +- `job_name` (REQUIRED): - + - Type: STRING + - Default: `none` + - Usage: `job-name` +- `rdi_host` (REQUIRED): -* `rdi_host` (REQUIRED): - * Type: STRING - * Default: `none` - * Usage: `--rdi-host` + - Type: STRING + - Default: `none` + - Usage: `--rdi-host` Host/IP of RDI Database +- `rdi_port` (REQUIRED): -* `rdi_port` (REQUIRED): - * Type: - * Default: `none` - * Usage: `--rdi-port` + - Type: + - Default: `none` + - Usage: `--rdi-port` Port of RDI Database +- `rdi_user`: -* `rdi_user`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-user` + - Type: STRING + - Default: `none` + - Usage: `--rdi-user` RDI Database Username +- `rdi_password`: -* `rdi_password`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-password` + - Type: STRING + - Default: `none` + - Usage: `--rdi-password` RDI Database Password +- `rdi_key`: -* `rdi_key`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-key` + - Type: STRING + - Default: `none` + - Usage: `--rdi-key` Private key file to authenticate with +- `rdi_cert`: -* `rdi_cert`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-cert` + - Type: STRING + - Default: `none` + - Usage: `--rdi-cert` Client certificate file to authenticate with +- `rdi_cacert`: -* `rdi_cacert`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-cacert` + - Type: STRING + - Default: `none` + - Usage: `--rdi-cacert` CA certificate file to verify with +- `rdi_key_password`: -* `rdi_key_password`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-key-password` + - Type: STRING + - Default: `none` + - Usage: `--rdi-key-password` Password for unlocking an encrypted private key +- `help`: -* `help`: - * Type: BOOL - * Default: `false` - * Usage: `--help` + - Type: BOOL + - Default: `false` + - Usage: `--help` Show this message and exit. - - ## CLI help ``` @@ -113,7 +111,7 @@ Usage: redis-di describe-job [OPTIONS] JOB_NAME Describes a transformation engine's job Options: - -l, --log-level [DEBUG|INFO|WARN|ERROR|CRITICAL] + -l, --log-level [TRACE|DEBUG|INFO|WARNING|ERROR|CRITICAL] [default: INFO] --rdi-host TEXT Host/IP of RDI Database [required] --rdi-port INTEGER RANGE Port of RDI Database [1<=x<=65535; diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-dump-support-package.md b/content/integrate/redis-data-integration/reference/cli/redis-di-dump-support-package.md index b6c7f69520..54316f8a4f 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-dump-support-package.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-dump-support-package.md @@ -1,13 +1,15 @@ --- Title: redis-di dump-support-package linkTitle: redis-di dump-support-package -description: Dumps RDI support package +description: Dumps RDI support package weight: 10 alwaysopen: false categories: ["redis-di"] aliases: --- +Dumps RDI support package + ## Usage ``` @@ -15,128 +17,134 @@ Usage: redis-di dump-support-package [OPTIONS] ``` ## Options -* `log_level`: - * Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL']) - * Default: `info` - * Usage: `--log-level + +- `log_level`: + + - Type: Choice(['TRACE', 'DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']) + - Default: `info` + - Usage: `--log-level -l` - +- `rdi_namespace`: + + - Type: STRING + - Default: `rdi` + - Usage: `--rdi-namespace` + RDI Kubernetes namespace -* `rdi_host` (REQUIRED): - * Type: STRING - * Default: `none` - * Usage: `--rdi-host` +- `rdi_host` (REQUIRED): + + - Type: STRING + - Default: `none` + - Usage: `--rdi-host` Host/IP of RDI Database +- `rdi_port` (REQUIRED): -* `rdi_port` (REQUIRED): - * Type: - * Default: `none` - * Usage: `--rdi-port` + - Type: + - Default: `none` + - Usage: `--rdi-port` Port of RDI Database +- `rdi_user`: -* `rdi_user`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-user` + - Type: STRING + - Default: `none` + - Usage: `--rdi-user` RDI Database Username +- `rdi_password`: -* `rdi_password`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-password` + - Type: STRING + - Default: `none` + - Usage: `--rdi-password` RDI Database Password +- `rdi_key`: -* `rdi_key`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-key` + - Type: STRING + - Default: `none` + - Usage: `--rdi-key` Private key file to authenticate with +- `rdi_cert`: -* `rdi_cert`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-cert` + - Type: STRING + - Default: `none` + - Usage: `--rdi-cert` Client certificate file to authenticate with +- `rdi_cacert`: -* `rdi_cacert`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-cacert` + - Type: STRING + - Default: `none` + - Usage: `--rdi-cacert` CA certificate file to verify with +- `rdi_key_password`: -* `rdi_key_password`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-key-password` + - Type: STRING + - Default: `none` + - Usage: `--rdi-key-password` Password for unlocking an encrypted private key +- `directory`: -* `directory`: - * Type: STRING - * Default: `.` - * Usage: `--dir` + - Type: STRING + - Default: `.` + - Usage: `--dir` Directory where the support file should be generated +- `dump_rejected`: -* `dump_rejected`: - * Type: INT - * Default: `none` - * Usage: `--dump-rejected` + - Type: INT + - Default: `none` + - Usage: `--dump-rejected` Dumps rejected records +- `trace_timeout`: -* `trace_timeout`: - * Type: - * Default: `none` - * Usage: `--trace-timeout` + - Type: + - Default: `none` + - Usage: `--trace-timeout` Stops the trace after exceeding this timeout (in seconds) +- `max_change_records`: -* `max_change_records`: - * Type: =1> - * Default: `10` - * Usage: `--max-change-records` + - Type: =1> + - Default: `10` + - Usage: `--max-change-records` Maximum traced change records +- `trace_only_rejected`: -* `trace_only_rejected`: - * Type: BOOL - * Default: `false` - * Usage: `--trace-only-rejected` + - Type: BOOL + - Default: `false` + - Usage: `--trace-only-rejected` Trace only rejected change records +- `help`: -* `help`: - * Type: BOOL - * Default: `false` - * Usage: `--help` + - Type: BOOL + - Default: `false` + - Usage: `--help` Show this message and exit. - - ## CLI help ``` @@ -145,8 +153,9 @@ Usage: redis-di dump-support-package [OPTIONS] Dumps RDI support package Options: - -l, --log-level [DEBUG|INFO|WARN|ERROR|CRITICAL] + -l, --log-level [TRACE|DEBUG|INFO|WARNING|ERROR|CRITICAL] [default: INFO] + --rdi-namespace TEXT RDI Kubernetes namespace [default: rdi] --rdi-host TEXT Host/IP of RDI Database [required] --rdi-port INTEGER RANGE Port of RDI Database [1<=x<=65535; required] diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-get-rejected.md b/content/integrate/redis-data-integration/reference/cli/redis-di-get-rejected.md index 640e5a5c9a..e7a90f8bec 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-get-rejected.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-get-rejected.md @@ -1,13 +1,15 @@ --- Title: redis-di get-rejected linkTitle: redis-di get-rejected -description: Returns all the stored rejected entries +description: Returns all the stored rejected entries weight: 10 alwaysopen: false categories: ["redis-di"] aliases: --- +Returns all the stored rejected entries + ## Usage ``` @@ -15,113 +17,111 @@ Usage: redis-di get-rejected [OPTIONS] ``` ## Options -* `log_level`: - * Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL']) - * Default: `info` - * Usage: `--log-level --l` - +- `log_level`: + + - Type: Choice(['TRACE', 'DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']) + - Default: `info` + - Usage: `--log-level +-l` +- `rdi_host` (REQUIRED): -* `rdi_host` (REQUIRED): - * Type: STRING - * Default: `none` - * Usage: `--rdi-host` + - Type: STRING + - Default: `none` + - Usage: `--rdi-host` Host/IP of RDI Database +- `rdi_port` (REQUIRED): -* `rdi_port` (REQUIRED): - * Type: - * Default: `none` - * Usage: `--rdi-port` + - Type: + - Default: `none` + - Usage: `--rdi-port` Port of RDI Database +- `rdi_user`: -* `rdi_user`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-user` + - Type: STRING + - Default: `none` + - Usage: `--rdi-user` RDI Database Username +- `rdi_password`: -* `rdi_password`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-password` + - Type: STRING + - Default: `none` + - Usage: `--rdi-password` RDI Database Password +- `rdi_key`: -* `rdi_key`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-key` + - Type: STRING + - Default: `none` + - Usage: `--rdi-key` Private key file to authenticate with +- `rdi_cert`: -* `rdi_cert`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-cert` + - Type: STRING + - Default: `none` + - Usage: `--rdi-cert` Client certificate file to authenticate with +- `rdi_cacert`: -* `rdi_cacert`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-cacert` + - Type: STRING + - Default: `none` + - Usage: `--rdi-cacert` CA certificate file to verify with +- `rdi_key_password`: -* `rdi_key_password`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-key-password` + - Type: STRING + - Default: `none` + - Usage: `--rdi-key-password` Password for unlocking an encrypted private key +- `max_records`: -* `max_records`: - * Type: =1> - * Default: `none` - * Usage: `--max-records` + - Type: =1> + - Default: `none` + - Usage: `--max-records` Maximum rejected records per DLQ +- `oldest`: -* `oldest`: - * Type: BOOL - * Default: `false` - * Usage: `--oldest + - Type: BOOL + - Default: `false` + - Usage: `--oldest -o` - Displays the oldest rejected records. If omitted, most recent records will be retrieved + Displays the oldest rejected records. If omitted, most resent records will be retrieved +- `dlq_name`: -* `dlq_name`: - * Type: STRING - * Default: `none` - * Usage: `--dlq-name` + - Type: STRING + - Default: `none` + - Usage: `--dlq-name` Only prints the rejected records for the specified DLQ (Dead Letter Queue) name +- `help`: -* `help`: - * Type: BOOL - * Default: `false` - * Usage: `--help` + - Type: BOOL + - Default: `false` + - Usage: `--help` Show this message and exit. - - ## CLI help ``` @@ -130,7 +130,7 @@ Usage: redis-di get-rejected [OPTIONS] Returns all the stored rejected entries Options: - -l, --log-level [DEBUG|INFO|WARN|ERROR|CRITICAL] + -l, --log-level [TRACE|DEBUG|INFO|WARNING|ERROR|CRITICAL] [default: INFO] --rdi-host TEXT Host/IP of RDI Database [required] --rdi-port INTEGER RANGE Port of RDI Database [1<=x<=65535; diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-install.md b/content/integrate/redis-data-integration/reference/cli/redis-di-install.md index 7e72fbd224..26ba92a316 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-install.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-install.md @@ -1,13 +1,15 @@ --- Title: redis-di install linkTitle: redis-di install -description: Installs RDI +description: Installs RDI weight: 10 alwaysopen: false categories: ["redis-di"] aliases: --- +Installs RDI + ## Usage ``` @@ -15,46 +17,62 @@ Usage: redis-di install [OPTIONS] ``` ## Options -* `log_level`: - * Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL']) - * Default: `warn` - * Usage: `--log-level --l` - +- `log_level`: + + - Type: Choice(['TRACE', 'DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']) + - Default: `warning` + - Usage: `--log-level +-l` +- `file`: -* `file`: - * Type: - * Default: `none` - * Usage: `-f + - Type: + - Default: `none` + - Usage: `-f --file` - Path to a TOML configuration file for silent installation + Path to a YAML configuration file for silent installation +- `online`: -* `online`: - * Type: BOOL - * Default: `false` - * Usage: `--online` + - Type: BOOL + - Default: `false` + - Usage: `--online` Run installer in online mode -* `installation-dir`: - * Type: STRING - * Default: `/var/lib` - * Usage: `--installation-dir` +- `k3s_only`: - Custom installation directory + - Type: BOOL + - Default: `false` + - Usage: `--k3s-only` -* `help`: - * Type: BOOL - * Default: `false` - * Usage: `--help` + Install only k3s components - Show this message and exit. +- `https_port`: + + - Type: INT + - Default: `443` + - Usage: `--https-port` + + HTTPS port for Traefik +- `installation_dir`: + - Type: + - Default: `none` + - Usage: `--installation-dir` + + Custom installation directory + +- `help`: + + - Type: BOOL + - Default: `false` + - Usage: `--help` + + Show this message and exit. ## CLI help @@ -64,11 +82,10 @@ Usage: redis-di install [OPTIONS] Installs RDI Options: - -l, --log-level [DEBUG|INFO|WARN|ERROR|CRITICAL] - [default: WARN] - -f, --file FILE Path to a TOML configuration file for silent + -l, --log-level [TRACE|DEBUG|INFO|WARNING|ERROR|CRITICAL] + [default: WARNING] + -f, --file FILE Path to a YAML configuration file for silent installation - --installation-dir Custom installation directory - + --installation-dir DIRECTORY Custom installation directory --help Show this message and exit. ``` diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-list-contexts.md b/content/integrate/redis-data-integration/reference/cli/redis-di-list-contexts.md index 0556e23b74..b23d2a88b1 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-list-contexts.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-list-contexts.md @@ -1,13 +1,15 @@ --- Title: redis-di list-contexts linkTitle: redis-di list-contexts -description: Lists all saved contexts +description: Lists all saved contexts weight: 10 alwaysopen: false categories: ["redis-di"] aliases: --- +Lists all saved contexts + ## Usage ``` @@ -15,14 +17,14 @@ Usage: redis-di list-contexts [OPTIONS] ``` ## Options -* `help`: - * Type: BOOL - * Default: `false` - * Usage: `--help` - Show this message and exit. +- `help`: + - Type: BOOL + - Default: `false` + - Usage: `--help` + Show this message and exit. ## CLI help diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-list-jobs.md b/content/integrate/redis-data-integration/reference/cli/redis-di-list-jobs.md index e9380b54f6..5d67e30e22 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-list-jobs.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-list-jobs.md @@ -1,13 +1,15 @@ --- Title: redis-di list-jobs linkTitle: redis-di list-jobs -description: Lists transformation engine's jobs +description: Lists transformation engine's jobs weight: 10 alwaysopen: false categories: ["redis-di"] aliases: --- +Lists transformation engine's jobs + ## Usage ``` @@ -15,88 +17,86 @@ Usage: redis-di list-jobs [OPTIONS] ``` ## Options -* `log_level`: - * Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL']) - * Default: `info` - * Usage: `--log-level --l` - +- `log_level`: + + - Type: Choice(['TRACE', 'DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']) + - Default: `info` + - Usage: `--log-level +-l` +- `rdi_host` (REQUIRED): -* `rdi_host` (REQUIRED): - * Type: STRING - * Default: `none` - * Usage: `--rdi-host` + - Type: STRING + - Default: `none` + - Usage: `--rdi-host` Host/IP of RDI Database +- `rdi_port` (REQUIRED): -* `rdi_port` (REQUIRED): - * Type: - * Default: `none` - * Usage: `--rdi-port` + - Type: + - Default: `none` + - Usage: `--rdi-port` Port of RDI Database +- `rdi_user`: -* `rdi_user`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-user` + - Type: STRING + - Default: `none` + - Usage: `--rdi-user` RDI Database Username +- `rdi_password`: -* `rdi_password`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-password` + - Type: STRING + - Default: `none` + - Usage: `--rdi-password` RDI Database Password +- `rdi_key`: -* `rdi_key`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-key` + - Type: STRING + - Default: `none` + - Usage: `--rdi-key` Private key file to authenticate with +- `rdi_cert`: -* `rdi_cert`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-cert` + - Type: STRING + - Default: `none` + - Usage: `--rdi-cert` Client certificate file to authenticate with +- `rdi_cacert`: -* `rdi_cacert`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-cacert` + - Type: STRING + - Default: `none` + - Usage: `--rdi-cacert` CA certificate file to verify with +- `rdi_key_password`: -* `rdi_key_password`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-key-password` + - Type: STRING + - Default: `none` + - Usage: `--rdi-key-password` Password for unlocking an encrypted private key +- `help`: -* `help`: - * Type: BOOL - * Default: `false` - * Usage: `--help` + - Type: BOOL + - Default: `false` + - Usage: `--help` Show this message and exit. - - ## CLI help ``` @@ -105,7 +105,7 @@ Usage: redis-di list-jobs [OPTIONS] Lists transformation engine's jobs Options: - -l, --log-level [DEBUG|INFO|WARN|ERROR|CRITICAL] + -l, --log-level [TRACE|DEBUG|INFO|WARNING|ERROR|CRITICAL] [default: INFO] --rdi-host TEXT Host/IP of RDI Database [required] --rdi-port INTEGER RANGE Port of RDI Database [1<=x<=65535; diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-monitor.md b/content/integrate/redis-data-integration/reference/cli/redis-di-monitor.md new file mode 100644 index 0000000000..c08002baa8 --- /dev/null +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-monitor.md @@ -0,0 +1,151 @@ +--- +Title: redis-di monitor +linkTitle: redis-di monitor +description: Monitors RDI by collecting metrics and exporting to Prometheus +weight: 10 +alwaysopen: false +categories: ["redis-di"] +aliases: +--- + +Monitors RDI by collecting metrics and exporting to Prometheus + +## Usage + +``` +Usage: redis-di monitor [OPTIONS] +``` + +## Options + +- `log_level`: + + - Type: Choice(['TRACE', 'DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']) + - Default: `info` + - Usage: `--log-level +-l` + +- `rdi_namespace`: + + - Type: STRING + - Default: `rdi` + - Usage: `--rdi-namespace` + + RDI Kubernetes namespace + +- `rdi_host` (REQUIRED): + + - Type: STRING + - Default: `none` + - Usage: `--rdi-host` + + Host/IP of RDI Database + +- `rdi_port` (REQUIRED): + + - Type: + - Default: `none` + - Usage: `--rdi-port` + + Port of RDI Database + +- `rdi_user`: + + - Type: STRING + - Default: `none` + - Usage: `--rdi-user` + + RDI Database Username + +- `rdi_password`: + + - Type: STRING + - Default: `none` + - Usage: `--rdi-password` + + RDI Database Password + +- `rdi_key`: + + - Type: STRING + - Default: `none` + - Usage: `--rdi-key` + + Private key file to authenticate with + +- `rdi_cert`: + + - Type: STRING + - Default: `none` + - Usage: `--rdi-cert` + + Client certificate file to authenticate with + +- `rdi_cacert`: + + - Type: STRING + - Default: `none` + - Usage: `--rdi-cacert` + + CA certificate file to verify with + +- `rdi_key_password`: + + - Type: STRING + - Default: `none` + - Usage: `--rdi-key-password` + + Password for unlocking an encrypted private key + +- `exporter_port`: + + - Type: + - Default: `9121` + - Usage: `--exporter-port` + + HTTP port to start the exporter on + +- `collect_interval`: + + - Type: + - Default: `5` + - Usage: `--collect-interval` + + Metrics collection interval (seconds) + +- `help`: + + - Type: BOOL + - Default: `false` + - Usage: `--help` + + Show this message and exit. + +## CLI help + +``` +Usage: redis-di monitor [OPTIONS] + + Monitors RDI by collecting metrics and exporting to Prometheus + +Options: + -l, --log-level [TRACE|DEBUG|INFO|WARNING|ERROR|CRITICAL] + [default: INFO] + --rdi-namespace TEXT RDI Kubernetes namespace [default: rdi] + --rdi-host TEXT Host/IP of RDI Database [required] + --rdi-port INTEGER RANGE Port of RDI Database [1<=x<=65535; + required] + --rdi-user TEXT RDI Database Username + --rdi-password TEXT RDI Database Password + --rdi-key TEXT Private key file to authenticate with + --rdi-cert TEXT Client certificate file to authenticate with + --rdi-cacert TEXT CA certificate file to verify with + --rdi-key-password TEXT Password for unlocking an encrypted private + key + --exporter-port INTEGER RANGE HTTP port to start the exporter on + [1<=x<=65535] + --collect-interval INTEGER RANGE + Metrics collection interval (seconds) + [1<=x<=60] + --help Show this message and exit. +``` diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-reset.md b/content/integrate/redis-data-integration/reference/cli/redis-di-reset.md index 3224624182..4f468cccb9 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-reset.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-reset.md @@ -1,13 +1,15 @@ --- Title: redis-di reset linkTitle: redis-di reset -description: Resets the pipeline into initial full sync mode +description: Resets the pipeline into initial full sync mode weight: 10 alwaysopen: false categories: ["redis-di"] aliases: --- +Resets the pipeline into initial full sync mode + ## Usage ``` @@ -15,105 +17,103 @@ Usage: redis-di reset [OPTIONS] ``` ## Options -* `log_level`: - * Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL']) - * Default: `info` - * Usage: `--log-level --l` - +- `log_level`: + + - Type: Choice(['TRACE', 'DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']) + - Default: `info` + - Usage: `--log-level +-l` +- `rdi_host` (REQUIRED): -* `rdi_host` (REQUIRED): - * Type: STRING - * Default: `none` - * Usage: `--rdi-host` + - Type: STRING + - Default: `none` + - Usage: `--rdi-host` Host/IP of RDI Database +- `rdi_port` (REQUIRED): -* `rdi_port` (REQUIRED): - * Type: - * Default: `none` - * Usage: `--rdi-port` + - Type: + - Default: `none` + - Usage: `--rdi-port` Port of RDI Database +- `rdi_user`: -* `rdi_user`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-user` + - Type: STRING + - Default: `none` + - Usage: `--rdi-user` RDI Database Username +- `rdi_password`: -* `rdi_password`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-password` + - Type: STRING + - Default: `none` + - Usage: `--rdi-password` RDI Database Password +- `rdi_key`: -* `rdi_key`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-key` + - Type: STRING + - Default: `none` + - Usage: `--rdi-key` Private key file to authenticate with +- `rdi_cert`: -* `rdi_cert`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-cert` + - Type: STRING + - Default: `none` + - Usage: `--rdi-cert` Client certificate file to authenticate with +- `rdi_cacert`: -* `rdi_cacert`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-cacert` + - Type: STRING + - Default: `none` + - Usage: `--rdi-cacert` CA certificate file to verify with +- `rdi_key_password`: -* `rdi_key_password`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-key-password` + - Type: STRING + - Default: `none` + - Usage: `--rdi-key-password` Password for unlocking an encrypted private key +- `force`: -* `force`: - * Type: BOOL - * Default: `false` - * Usage: `--force + - Type: BOOL + - Default: `false` + - Usage: `--force -f` Force operation. skips verification prompts +- `pause_for_confirmation`: -* `pause_for_confirmation`: - * Type: BOOL - * Default: `false` - * Usage: `--pause-for-confirmation` + - Type: BOOL + - Default: `false` + - Usage: `--pause-for-confirmation` Pause for user confirmation if manual shutdown of collector required +- `help`: -* `help`: - * Type: BOOL - * Default: `false` - * Usage: `--help` + - Type: BOOL + - Default: `false` + - Usage: `--help` Show this message and exit. - - ## CLI help ``` @@ -122,7 +122,7 @@ Usage: redis-di reset [OPTIONS] Resets the pipeline into initial full sync mode Options: - -l, --log-level [DEBUG|INFO|WARN|ERROR|CRITICAL] + -l, --log-level [TRACE|DEBUG|INFO|WARNING|ERROR|CRITICAL] [default: INFO] --rdi-host TEXT Host/IP of RDI Database [required] --rdi-port INTEGER RANGE Port of RDI Database [1<=x<=65535; diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-scaffold.md b/content/integrate/redis-data-integration/reference/cli/redis-di-scaffold.md index 85b40ebaf0..5525deda81 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-scaffold.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-scaffold.md @@ -1,13 +1,15 @@ --- Title: redis-di scaffold linkTitle: redis-di scaffold -description: Generates configuration files for RDI +description: Generates configuration files for RDI weight: 10 alwaysopen: false categories: ["redis-di"] aliases: --- +Generates configuration files for RDI + ## Usage ``` @@ -15,59 +17,48 @@ Usage: redis-di scaffold [OPTIONS] ``` ## Options -* `log_level`: - * Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL']) - * Default: `info` - * Usage: `--log-level --l` - - - - -* `db_type` (REQUIRED): - * Type: Choice([, , , , ]) - * Default: `none` - * Usage: `--db-type` - DB type +- `log_level`: + - Type: Choice(['TRACE', 'DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']) + - Default: `info` + - Usage: `--log-level +-l` -* `strategy`: - * Type: Choice([, ]) - * Default: `ingest` - * Usage: `--strategy` +- `db_type` (REQUIRED): - Strategy + - Type: Choice([, , , , , , ]) + - Default: `none` + - Usage: `--db-type` + DB type Output to directory or stdout +- `directory`: -* `directory`: - * Type: STRING - * Default: `none` - * Usage: `--dir` + - Type: STRING + - Default: `none` + - Usage: `--dir` Directory containing RDI configuration +- `preview`: -* `preview`: - * Type: STRING - * Default: `none` - * Usage: `--preview` + - Type: STRING + - Default: `none` + - Usage: `--preview` Print the content of the scaffolded config file to CLI output +- `help`: -* `help`: - * Type: BOOL - * Default: `false` - * Usage: `--help` + - Type: BOOL + - Default: `false` + - Usage: `--help` Show this message and exit. - - ## CLI help ``` @@ -76,12 +67,10 @@ Usage: redis-di scaffold [OPTIONS] Generates configuration files for RDI Options: - -l, --log-level [DEBUG|INFO|WARN|ERROR|CRITICAL] + -l, --log-level [TRACE|DEBUG|INFO|WARNING|ERROR|CRITICAL] [default: INFO] - --db-type [mysql|oracle|postgresql|redis|sqlserver] + --db-type [cassandra|mariadb|mongodb|mysql|oracle|postgresql|sqlserver] DB type [required] - --strategy [ingest|write_behind] - Strategy [default: ingest] Output formats: [mutually_exclusive, required] Output to directory or stdout --dir TEXT Directory containing RDI configuration diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-set-context.md b/content/integrate/redis-data-integration/reference/cli/redis-di-set-context.md index ebb1fbcdc6..219c865d4d 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-set-context.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-set-context.md @@ -1,13 +1,15 @@ --- Title: redis-di set-context linkTitle: redis-di set-context -description: Sets a context to be the active one +description: Sets a context to be the active one weight: 10 alwaysopen: false categories: ["redis-di"] aliases: --- +Sets a context to be the active one + ## Usage ``` @@ -15,32 +17,28 @@ Usage: redis-di set-context [OPTIONS] CONTEXT_NAME ``` ## Options -* `log_level`: - * Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL']) - * Default: `info` - * Usage: `--log-level --l` - +- `log_level`: + - Type: Choice(['TRACE', 'DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']) + - Default: `info` + - Usage: `--log-level +-l` -* `context_name` (REQUIRED): - * Type: STRING - * Default: `none` - * Usage: `context-name` +- `context_name` (REQUIRED): - + - Type: STRING + - Default: `none` + - Usage: `context-name` +- `help`: -* `help`: - * Type: BOOL - * Default: `false` - * Usage: `--help` + - Type: BOOL + - Default: `false` + - Usage: `--help` Show this message and exit. - - ## CLI help ``` @@ -49,7 +47,7 @@ Usage: redis-di set-context [OPTIONS] CONTEXT_NAME Sets a context to be the active one Options: - -l, --log-level [DEBUG|INFO|WARN|ERROR|CRITICAL] + -l, --log-level [TRACE|DEBUG|INFO|WARNING|ERROR|CRITICAL] [default: INFO] --help Show this message and exit. ``` diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-set-secret.md b/content/integrate/redis-data-integration/reference/cli/redis-di-set-secret.md index 6bba43e3ca..c88b525ee9 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-set-secret.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-set-secret.md @@ -8,6 +8,8 @@ categories: ["redis-di"] aliases: --- +Creates a secret of a specified key + ## Usage ``` @@ -17,21 +19,29 @@ Usage: redis-di set-secret [OPTIONS] {RDI_REDIS_USERNAME|RDI_REDIS_PASSWORD|RD WORD|SOURCE_DB_CACERT|SOURCE_DB_CERT|SOURCE_DB_KEY| SOURCE_DB_KEY_PASSWORD|TARGET_DB_USERNAME|TARGET_DB _PASSWORD|TARGET_DB_CACERT|TARGET_DB_CERT|TARGET_DB - _KEY|TARGET_DB_KEY_PASSWORD} [VALUE] + _KEY|TARGET_DB_KEY_PASSWORD|JWT_SECRET_KEY} [VALUE] ``` ## Options - `log_level`: - - Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL']) + - Type: Choice(['TRACE', 'DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']) - Default: `info` - Usage: `--log-level -l` +- `rdi_namespace`: + + - Type: STRING + - Default: `rdi` + - Usage: `--rdi-namespace` + + RDI Kubernetes namespace + - `key` (REQUIRED): - - Type: Choice(['RDI_REDIS_USERNAME', 'RDI_REDIS_PASSWORD', 'RDI_REDIS_CACERT', 'RDI_REDIS_CERT', 'RDI_REDIS_KEY', 'RDI_REDIS_KEY_PASSPHRASE', 'SOURCE_DB_USERNAME', 'SOURCE_DB_PASSWORD', 'SOURCE_DB_CACERT', 'SOURCE_DB_CERT', 'SOURCE_DB_KEY', 'SOURCE_DB_KEY_PASSWORD', 'TARGET_DB_USERNAME', 'TARGET_DB_PASSWORD', 'TARGET_DB_CACERT', 'TARGET_DB_CERT', 'TARGET_DB_KEY', 'TARGET_DB_KEY_PASSWORD']) + - Type: Choice(['RDI_REDIS_USERNAME', 'RDI_REDIS_PASSWORD', 'RDI_REDIS_CACERT', 'RDI_REDIS_CERT', 'RDI_REDIS_KEY', 'RDI_REDIS_KEY_PASSPHRASE', 'SOURCE_DB_USERNAME', 'SOURCE_DB_PASSWORD', 'SOURCE_DB_CACERT', 'SOURCE_DB_CERT', 'SOURCE_DB_KEY', 'SOURCE_DB_KEY_PASSWORD', 'TARGET_DB_USERNAME', 'TARGET_DB_PASSWORD', 'TARGET_DB_CACERT', 'TARGET_DB_CERT', 'TARGET_DB_KEY', 'TARGET_DB_KEY_PASSWORD', 'JWT_SECRET_KEY']) - Default: `none` - Usage: `key` @@ -58,12 +68,13 @@ Usage: redis-di set-secret [OPTIONS] {RDI_REDIS_USERNAME|RDI_REDIS_PASSWORD|RD WORD|SOURCE_DB_CACERT|SOURCE_DB_CERT|SOURCE_DB_KEY| SOURCE_DB_KEY_PASSWORD|TARGET_DB_USERNAME|TARGET_DB _PASSWORD|TARGET_DB_CACERT|TARGET_DB_CERT|TARGET_DB - _KEY|TARGET_DB_KEY_PASSWORD} [VALUE] + _KEY|TARGET_DB_KEY_PASSWORD|JWT_SECRET_KEY} [VALUE] Creates a secret of a specified key Options: - -l, --log-level [DEBUG|INFO|WARN|ERROR|CRITICAL] + -l, --log-level [TRACE|DEBUG|INFO|WARNING|ERROR|CRITICAL] [default: INFO] + --rdi-namespace TEXT RDI Kubernetes namespace [default: rdi] --help Show this message and exit. -``` \ No newline at end of file +``` diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-start.md b/content/integrate/redis-data-integration/reference/cli/redis-di-start.md index 968be480cf..6d3243d181 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-start.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-start.md @@ -1,13 +1,15 @@ --- Title: redis-di start linkTitle: redis-di start -description: Starts the pipeline +description: Starts the pipeline weight: 10 alwaysopen: false categories: ["redis-di"] aliases: --- +Starts the pipeline + ## Usage ``` @@ -15,88 +17,86 @@ Usage: redis-di start [OPTIONS] ``` ## Options -* `log_level`: - * Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL']) - * Default: `info` - * Usage: `--log-level --l` - +- `log_level`: + + - Type: Choice(['TRACE', 'DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']) + - Default: `info` + - Usage: `--log-level +-l` +- `rdi_host` (REQUIRED): -* `rdi_host` (REQUIRED): - * Type: STRING - * Default: `none` - * Usage: `--rdi-host` + - Type: STRING + - Default: `none` + - Usage: `--rdi-host` Host/IP of RDI Database +- `rdi_port` (REQUIRED): -* `rdi_port` (REQUIRED): - * Type: - * Default: `none` - * Usage: `--rdi-port` + - Type: + - Default: `none` + - Usage: `--rdi-port` Port of RDI Database +- `rdi_user`: -* `rdi_user`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-user` + - Type: STRING + - Default: `none` + - Usage: `--rdi-user` RDI Database Username +- `rdi_password`: -* `rdi_password`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-password` + - Type: STRING + - Default: `none` + - Usage: `--rdi-password` RDI Database Password +- `rdi_key`: -* `rdi_key`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-key` + - Type: STRING + - Default: `none` + - Usage: `--rdi-key` Private key file to authenticate with +- `rdi_cert`: -* `rdi_cert`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-cert` + - Type: STRING + - Default: `none` + - Usage: `--rdi-cert` Client certificate file to authenticate with +- `rdi_cacert`: -* `rdi_cacert`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-cacert` + - Type: STRING + - Default: `none` + - Usage: `--rdi-cacert` CA certificate file to verify with +- `rdi_key_password`: -* `rdi_key_password`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-key-password` + - Type: STRING + - Default: `none` + - Usage: `--rdi-key-password` Password for unlocking an encrypted private key +- `help`: -* `help`: - * Type: BOOL - * Default: `false` - * Usage: `--help` + - Type: BOOL + - Default: `false` + - Usage: `--help` Show this message and exit. - - ## CLI help ``` @@ -105,7 +105,7 @@ Usage: redis-di start [OPTIONS] Starts the pipeline Options: - -l, --log-level [DEBUG|INFO|WARN|ERROR|CRITICAL] + -l, --log-level [TRACE|DEBUG|INFO|WARNING|ERROR|CRITICAL] [default: INFO] --rdi-host TEXT Host/IP of RDI Database [required] --rdi-port INTEGER RANGE Port of RDI Database [1<=x<=65535; diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-status.md b/content/integrate/redis-data-integration/reference/cli/redis-di-status.md index 54e7c94815..b7372c9a3e 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-status.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-status.md @@ -1,13 +1,15 @@ --- Title: redis-di status linkTitle: redis-di status -description: Displays the status of the pipeline end to end +description: Displays the status of the pipeline end to end weight: 10 alwaysopen: false categories: ["redis-di"] aliases: --- +Displays the status of the pipeline end to end + ## Usage ``` @@ -15,124 +17,130 @@ Usage: redis-di status [OPTIONS] ``` ## Options -* `log_level`: - * Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL']) - * Default: `info` - * Usage: `--log-level + +- `log_level`: + + - Type: Choice(['TRACE', 'DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']) + - Default: `info` + - Usage: `--log-level -l` - +- `rdi_namespace`: + + - Type: STRING + - Default: `rdi` + - Usage: `--rdi-namespace` + RDI Kubernetes namespace -* `rdi_host` (REQUIRED): - * Type: STRING - * Default: `none` - * Usage: `--rdi-host` +- `rdi_host` (REQUIRED): + + - Type: STRING + - Default: `none` + - Usage: `--rdi-host` Host/IP of RDI Database +- `rdi_port` (REQUIRED): -* `rdi_port` (REQUIRED): - * Type: - * Default: `none` - * Usage: `--rdi-port` + - Type: + - Default: `none` + - Usage: `--rdi-port` Port of RDI Database +- `rdi_user`: -* `rdi_user`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-user` + - Type: STRING + - Default: `none` + - Usage: `--rdi-user` RDI Database Username +- `rdi_password`: -* `rdi_password`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-password` + - Type: STRING + - Default: `none` + - Usage: `--rdi-password` RDI Database Password +- `rdi_key`: -* `rdi_key`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-key` + - Type: STRING + - Default: `none` + - Usage: `--rdi-key` Private key file to authenticate with +- `rdi_cert`: -* `rdi_cert`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-cert` + - Type: STRING + - Default: `none` + - Usage: `--rdi-cert` Client certificate file to authenticate with +- `rdi_cacert`: -* `rdi_cacert`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-cacert` + - Type: STRING + - Default: `none` + - Usage: `--rdi-cacert` CA certificate file to verify with +- `rdi_key_password`: -* `rdi_key_password`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-key-password` + - Type: STRING + - Default: `none` + - Usage: `--rdi-key-password` Password for unlocking an encrypted private key +- `live`: -* `live`: - * Type: BOOL - * Default: `false` - * Usage: `--live + - Type: BOOL + - Default: `false` + - Usage: `--live -l` Live data flow monitoring +- `page_number`: -* `page_number`: - * Type: =1> - * Default: `none` - * Usage: `--page-number + - Type: =1> + - Default: `none` + - Usage: `--page-number -p` Set the page number (live mode only) +- `page_size`: -* `page_size`: - * Type: =1> - * Default: `none` - * Usage: `--page-size + - Type: =1> + - Default: `none` + - Usage: `--page-size -s` Set the page size (live mode only) +- `ingested_only`: -* `ingested_only`: - * Type: BOOL - * Default: `false` - * Usage: `--ingested-only + - Type: BOOL + - Default: `false` + - Usage: `--ingested-only -i` Display ingested data streams (live mode only) +- `help`: -* `help`: - * Type: BOOL - * Default: `false` - * Usage: `--help` + - Type: BOOL + - Default: `false` + - Usage: `--help` Show this message and exit. - - ## CLI help ``` @@ -141,8 +149,9 @@ Usage: redis-di status [OPTIONS] Displays the status of the pipeline end to end Options: - -l, --log-level [DEBUG|INFO|WARN|ERROR|CRITICAL] + -l, --log-level [TRACE|DEBUG|INFO|WARNING|ERROR|CRITICAL] [default: INFO] + --rdi-namespace TEXT RDI Kubernetes namespace [default: rdi] --rdi-host TEXT Host/IP of RDI Database [required] --rdi-port INTEGER RANGE Port of RDI Database [1<=x<=65535; required] diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-stop.md b/content/integrate/redis-data-integration/reference/cli/redis-di-stop.md index 98885ff17d..ab2bf0f9db 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-stop.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-stop.md @@ -1,13 +1,15 @@ --- Title: redis-di stop linkTitle: redis-di stop -description: Stops the pipeline +description: Stops the pipeline weight: 10 alwaysopen: false categories: ["redis-di"] aliases: --- +Stops the pipeline + ## Usage ``` @@ -15,88 +17,86 @@ Usage: redis-di stop [OPTIONS] ``` ## Options -* `log_level`: - * Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL']) - * Default: `info` - * Usage: `--log-level --l` - +- `log_level`: + + - Type: Choice(['TRACE', 'DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']) + - Default: `info` + - Usage: `--log-level +-l` +- `rdi_host` (REQUIRED): -* `rdi_host` (REQUIRED): - * Type: STRING - * Default: `none` - * Usage: `--rdi-host` + - Type: STRING + - Default: `none` + - Usage: `--rdi-host` Host/IP of RDI Database +- `rdi_port` (REQUIRED): -* `rdi_port` (REQUIRED): - * Type: - * Default: `none` - * Usage: `--rdi-port` + - Type: + - Default: `none` + - Usage: `--rdi-port` Port of RDI Database +- `rdi_user`: -* `rdi_user`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-user` + - Type: STRING + - Default: `none` + - Usage: `--rdi-user` RDI Database Username +- `rdi_password`: -* `rdi_password`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-password` + - Type: STRING + - Default: `none` + - Usage: `--rdi-password` RDI Database Password +- `rdi_key`: -* `rdi_key`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-key` + - Type: STRING + - Default: `none` + - Usage: `--rdi-key` Private key file to authenticate with +- `rdi_cert`: -* `rdi_cert`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-cert` + - Type: STRING + - Default: `none` + - Usage: `--rdi-cert` Client certificate file to authenticate with +- `rdi_cacert`: -* `rdi_cacert`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-cacert` + - Type: STRING + - Default: `none` + - Usage: `--rdi-cacert` CA certificate file to verify with +- `rdi_key_password`: -* `rdi_key_password`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-key-password` + - Type: STRING + - Default: `none` + - Usage: `--rdi-key-password` Password for unlocking an encrypted private key +- `help`: -* `help`: - * Type: BOOL - * Default: `false` - * Usage: `--help` + - Type: BOOL + - Default: `false` + - Usage: `--help` Show this message and exit. - - ## CLI help ``` @@ -105,7 +105,7 @@ Usage: redis-di stop [OPTIONS] Stops the pipeline Options: - -l, --log-level [DEBUG|INFO|WARN|ERROR|CRITICAL] + -l, --log-level [TRACE|DEBUG|INFO|WARNING|ERROR|CRITICAL] [default: INFO] --rdi-host TEXT Host/IP of RDI Database [required] --rdi-port INTEGER RANGE Port of RDI Database [1<=x<=65535; diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-trace.md b/content/integrate/redis-data-integration/reference/cli/redis-di-trace.md index 73608cf99e..03aeee733c 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di-trace.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-trace.md @@ -1,13 +1,15 @@ --- Title: redis-di trace linkTitle: redis-di trace -description: Starts a trace session for troubleshooting data transformation +description: Starts a trace session for troubleshooting data transformation weight: 10 alwaysopen: false categories: ["redis-di"] aliases: --- +Starts a trace session for troubleshooting data transformation + ## Usage ``` @@ -15,112 +17,110 @@ Usage: redis-di trace [OPTIONS] ``` ## Options -* `log_level`: - * Type: Choice(['DEBUG', 'INFO', 'WARN', 'ERROR', 'CRITICAL']) - * Default: `info` - * Usage: `--log-level --l` - +- `log_level`: + + - Type: Choice(['TRACE', 'DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']) + - Default: `info` + - Usage: `--log-level +-l` +- `rdi_host` (REQUIRED): -* `rdi_host` (REQUIRED): - * Type: STRING - * Default: `none` - * Usage: `--rdi-host` + - Type: STRING + - Default: `none` + - Usage: `--rdi-host` Host/IP of RDI Database +- `rdi_port` (REQUIRED): -* `rdi_port` (REQUIRED): - * Type: - * Default: `none` - * Usage: `--rdi-port` + - Type: + - Default: `none` + - Usage: `--rdi-port` Port of RDI Database +- `rdi_user`: -* `rdi_user`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-user` + - Type: STRING + - Default: `none` + - Usage: `--rdi-user` RDI Database Username +- `rdi_password`: -* `rdi_password`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-password` + - Type: STRING + - Default: `none` + - Usage: `--rdi-password` RDI Database Password +- `rdi_key`: -* `rdi_key`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-key` + - Type: STRING + - Default: `none` + - Usage: `--rdi-key` Private key file to authenticate with +- `rdi_cert`: -* `rdi_cert`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-cert` + - Type: STRING + - Default: `none` + - Usage: `--rdi-cert` Client certificate file to authenticate with +- `rdi_cacert`: -* `rdi_cacert`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-cacert` + - Type: STRING + - Default: `none` + - Usage: `--rdi-cacert` CA certificate file to verify with +- `rdi_key_password`: -* `rdi_key_password`: - * Type: STRING - * Default: `none` - * Usage: `--rdi-key-password` + - Type: STRING + - Default: `none` + - Usage: `--rdi-key-password` Password for unlocking an encrypted private key +- `max_change_records`: -* `max_change_records`: - * Type: =1> - * Default: `10` - * Usage: `--max-change-records` + - Type: =1> + - Default: `10` + - Usage: `--max-change-records` Maximum traced change records +- `timeout` (REQUIRED): -* `timeout` (REQUIRED): - * Type: - * Default: `20` - * Usage: `--timeout` + - Type: + - Default: `20` + - Usage: `--timeout` Stops the trace after exceeding this timeout (in seconds) +- `trace_only_rejected`: -* `trace_only_rejected`: - * Type: BOOL - * Default: `false` - * Usage: `--trace-only-rejected` + - Type: BOOL + - Default: `false` + - Usage: `--trace-only-rejected` Trace only rejected change records +- `help`: -* `help`: - * Type: BOOL - * Default: `false` - * Usage: `--help` + - Type: BOOL + - Default: `false` + - Usage: `--help` Show this message and exit. - - ## CLI help ``` @@ -129,7 +129,7 @@ Usage: redis-di trace [OPTIONS] Starts a trace session for troubleshooting data transformation Options: - -l, --log-level [DEBUG|INFO|WARN|ERROR|CRITICAL] + -l, --log-level [TRACE|DEBUG|INFO|WARNING|ERROR|CRITICAL] [default: INFO] --rdi-host TEXT Host/IP of RDI Database [required] --rdi-port INTEGER RANGE Port of RDI Database [1<=x<=65535; diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di-upgrade.md b/content/integrate/redis-data-integration/reference/cli/redis-di-upgrade.md new file mode 100644 index 0000000000..6412bf173f --- /dev/null +++ b/content/integrate/redis-data-integration/reference/cli/redis-di-upgrade.md @@ -0,0 +1,140 @@ +--- +Title: redis-di upgrade +linkTitle: redis-di upgrade +description: Upgrades RDI without losing data or downtime +weight: 10 +alwaysopen: false +categories: ["redis-di"] +aliases: +--- + +Upgrades RDI without losing data or downtime + +## Usage + +``` +Usage: redis-di upgrade [OPTIONS] +``` + +## Options + +- `log_level`: + + - Type: Choice(['TRACE', 'DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']) + - Default: `info` + - Usage: `--log-level +-l` + +- `rdi_namespace`: + + - Type: STRING + - Default: `rdi` + - Usage: `--rdi-namespace` + + RDI Kubernetes namespace + +- `rdi_host` (REQUIRED): + + - Type: STRING + - Default: `none` + - Usage: `--rdi-host` + + Host/IP of RDI Database + +- `rdi_port` (REQUIRED): + + - Type: + - Default: `none` + - Usage: `--rdi-port` + + Port of RDI Database + +- `rdi_user`: + + - Type: STRING + - Default: `none` + - Usage: `--rdi-user` + + RDI Database Username + +- `rdi_password`: + + - Type: STRING + - Default: `none` + - Usage: `--rdi-password` + + RDI Database Password + +- `rdi_key`: + + - Type: STRING + - Default: `none` + - Usage: `--rdi-key` + + Private key file to authenticate with + +- `rdi_cert`: + + - Type: STRING + - Default: `none` + - Usage: `--rdi-cert` + + Client certificate file to authenticate with + +- `rdi_cacert`: + + - Type: STRING + - Default: `none` + - Usage: `--rdi-cacert` + + CA certificate file to verify with + +- `rdi_key_password`: + + - Type: STRING + - Default: `none` + - Usage: `--rdi-key-password` + + Password for unlocking an encrypted private key + +- `force`: + + - Type: BOOL + - Default: `false` + - Usage: `--force +-f` + + Force operation. skips verification prompts + +- `help`: + + - Type: BOOL + - Default: `false` + - Usage: `--help` + + Show this message and exit. + +## CLI help + +``` +Usage: redis-di upgrade [OPTIONS] + + Upgrades RDI without losing data or downtime + +Options: + -l, --log-level [TRACE|DEBUG|INFO|WARNING|ERROR|CRITICAL] + [default: INFO] + --rdi-namespace TEXT RDI Kubernetes namespace [default: rdi] + --rdi-host TEXT Host/IP of RDI Database [required] + --rdi-port INTEGER RANGE Port of RDI Database [1<=x<=65535; + required] + --rdi-user TEXT RDI Database Username + --rdi-password TEXT RDI Database Password + --rdi-key TEXT Private key file to authenticate with + --rdi-cert TEXT Client certificate file to authenticate with + --rdi-cacert TEXT CA certificate file to verify with + --rdi-key-password TEXT Password for unlocking an encrypted private + key + -f, --force Force operation. skips verification prompts + --help Show this message and exit. +``` diff --git a/content/integrate/redis-data-integration/reference/cli/redis-di.md b/content/integrate/redis-data-integration/reference/cli/redis-di.md index d701f3df4b..3cc1de5440 100644 --- a/content/integrate/redis-data-integration/reference/cli/redis-di.md +++ b/content/integrate/redis-data-integration/reference/cli/redis-di.md @@ -1,13 +1,15 @@ --- Title: redis-di linkTitle: redis-di -description: A command line tool to manage & configure Redis Data Integration +description: A command line tool to manage & configure Redis Data Integration weight: 10 alwaysopen: false categories: ["redis-di"] aliases: --- +A command line tool to manage & configure Redis Data Integration + ## Usage ``` @@ -15,22 +17,22 @@ Usage: redis-di [OPTIONS] COMMAND [ARGS]... ``` ## Options -* `version`: - * Type: BOOL - * Default: `false` - * Usage: `--version` - Show the version and exit. +- `version`: + - Type: BOOL + - Default: `false` + - Usage: `--version` -* `help`: - * Type: BOOL - * Default: `false` - * Usage: `--help` + Show the version and exit. - Show this message and exit. +- `help`: + - Type: BOOL + - Default: `false` + - Usage: `--help` + Show this message and exit. ## CLI help @@ -45,8 +47,7 @@ Options: Commands: add-context Adds a new context - config-rdi Configures RDI db connection credentials - delete Deletes RDI database permanently + configure-rdi Configures RDI db connection credentials delete-all-contexts Deletes all contexts delete-context Deletes a context deploy Deploys the RDI configurations including target @@ -65,4 +66,5 @@ Commands: status Displays the status of the pipeline end to end stop Stops the pipeline trace Starts a trace session for troubleshooting data... + upgrade Upgrades RDI without losing data or downtime ``` diff --git a/content/integrate/redis-data-integration/reference/config-yaml-reference.md b/content/integrate/redis-data-integration/reference/config-yaml-reference.md index ae6989bd70..0ae91f2c11 100644 --- a/content/integrate/redis-data-integration/reference/config-yaml-reference.md +++ b/content/integrate/redis-data-integration/reference/config-yaml-reference.md @@ -1,326 +1,141 @@ --- Title: Redis Data Integration configuration file linkTitle: RDI configuration file -description: Reference for the RDI `config.yaml` file +description: Redis Data Integration configuration file reference weight: 10 alwaysopen: false categories: ["redis-di"] -aliases: /integrate/redis-data-integration/ingest/reference/config-yaml-reference/ +aliases: --- -This document describes the RDI `config.yaml` file in detail. See -[Configure data pipelines]({{< relref "/integrate/redis-data-integration/data-pipelines/data-pipelines" >}}) -for more information about the role `config.yaml` plays in defining a pipeline. +Configuration file for Redis Data Integration (RDI) source collectors and target connections -## Note about fully-qualified table names +**Properties** -Throughout this document we use the format `.` to refer to a fully-qualified table name. This format is actually the one used by MySQL and MariaDB, -but for Oracle, SQL Server, and PostgreSQL, you should use ``.`` instead. +| Name | Type | Description | Required | +| ----------------------------------------------------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | +| [**sources**](#sources)
(Source collectors) | `object` | Defines source collectors and their configurations. Each key represents a unique source identifier, and its value contains specific configuration for that collector
| | +| [**processors**](#processors)
(Data processing configuration) | `object`, `null` | Configuration settings that control how data is processed, including batch sizes, error handling, and performance tuning
| | +| [**targets**](#targets)
(Target connections) | `object` | Configuration for target Redis databases where processed data will be written
| | +| [**secret\-providers**](#secret-providers)
(Secret providers) | `object` | Configuration for secret management providers
| | -{{< note >}}You can specify the fully-qualified table name `.` as -a regular expression instead of providing the full name of the `databaseName` and `tableName`. -{{< /note >}} + -The example below shows the MySQL format specifying the desired columns for the -`chinook.customer` and `chinook.employee` tables: +## sources: Source collectors -```yaml -tables: - chinook.customer: - columns: - - CustID - - FirstName - - LastName - - Company - - Address - - Email - chinook.employee: - columns: - - EmpID - - FirstName - - LastName - - ReportsTo - - Address - - City - - State - ``` - -## Top level objects - -These objects define the sections at the root level of `config.yaml`. - -### Properties - -| Name | Type | Description | -| -- | -- | -- | -| [`sources`](#sources) | `object` | Source collectors | -| [`processors`](#processors)| `object`, `null` | RDI Processors | -| [`targets`](#targets) | `object` | Target connections | - -## `sources`: Source collectors {#sources} - -Each source database type has its own connector, but the basic configuration properties are -the same for all databases. - -See the Debezium documentation for more information about the specific connectors: - -- [MySQL/MariaDB](https://debezium.io/documentation/reference/stable/connectors/mysql.html) -- [Oracle](https://debezium.io/documentation/reference/stable/connectors/oracle.html) -- [PostgreSQL](https://debezium.io/documentation/reference/stable/connectors/postgresql.html) -- [SQL Server](https://debezium.io/documentation/reference/stable/connectors/sqlserver.html) - -### Essential properties - -#### `connection` - -| Name | Type | Source Databases | Description | -| -- | -- | -- | -- | -| `host` | `string` | MariaDB, MySQL, Oracle, PostgreSQL, SQL Server| The IP address of the database instance. | -| `port` | `integer` | MariaDB, MySQL, Oracle, PostgreSQL, SQL Server | The port of the database instance. | -| `database` | `string` | Oracle, PostgreSQL, SQL Server| The name of the database to capture changes from. For `SQL Server` you can define this as comma-separated list of database names. | -| `database.encrypt` | `string` | SQL Server| If SSL is enabled for your SQL Server database, you should also enable SSL in RDI by setting the value of this property to `true`.
Default: `false` | -| `database.server.id` | `integer` | MariaDB, MySQL | Numeric ID of this database client, which must be unique across all currently-running database processes in the MySQL cluster.
Default: 1| -| `database.url` | `string` | Oracle | Specifies the raw database JDBC URL. Use this property to define a custom database connection. Valid values include raw TNS names and RAC connection strings.| - -### Advanced properties - -#### `sink` - -| Name | Type | Description | -| -- | -- | -- | -| `redis.null.key` | `string` | Redis does not allow data objects without keys. This string will be used as the key for records that don't have a primary key.
Default: `"default"` | -| `redis.null.value` | `string` | Redis does not allow null object values (these occur with tombstone events, for example). This string will be used as the value for records without a payload.
Default: `"default"` | -| `redis.batch.size` | `integer` | Number of change records to insert in a single batch write (pipelined transaction).
Default: `500` | -| `redis.memory.limit.mb` | `integer` | The connector stops sending events when the Redis database size exceeds this size (in MB).
Default: `300` | -| `redis.wait.enabled` | `string` | If Redis is configured with a replica shard, this lets you verify that the data has been written to the replica.
Default: `false` | -| `redis.wait.timeout.ms` | `integer` | Defines the timeout in milliseconds when waiting for the replica.
Default: `1000` | -| `redis.wait.retry.enabled` | `string` | Enables retry on wait for replica failure.
Default: `false` | -| `redis.wait.retry.delay.ms` | `integer` | Defines the delay (in milliseconds) for retry on wait for replica failure.
Default: `1000` | -| `redis.retry.initial.delay.ms` | `integer` | Initial retry delay (in milliseconds) when encountering Redis connection or OOM issues. This value will be doubled upon every retry but won’t exceed `redis.retry.max.delay.ms`.
Default: `300` | -| `redis.retry.max.delay.ms` | `integer` | Maximum delay (in milliseconds) when encountering Redis connection or OOM issues.
Default: `10000` | - -#### `source` - -| Name | Type | Source Databases | Description | -|--|--|--|--| -| `snapshot.mode` | `string` | MariaDB, MySQL, Oracle, PostgreSQL, SQL Server | Specifies the mode that the connector uses to take snapshots of a captured table. See the [Debezium documentation](https://debezium.io/documentation/reference/stable/operations/debezium-server.html) for more details about the available options and configuration.
Default: `"initial"` | -| `topic.prefix` | `string` | MariaDB, MySQL, Oracle, PostgreSQL, SQL Server| A prefix for all topic names that receive events emitted by this connector.
Default: `"rdi"` | -| `database.exclude.list` | `string` | MariaDB, MySQL | An optional, comma-separated list of regular expressions that match the names of databases for which you do not want to capture changes. The connector captures changes in any database whose name is not included in `database.exclude.list`. Do not specify the `database` field in the `connection` configuration if you are using the `database.exclude.list` property to filter out databases. | -| `database.pdb.name` | `string` | Oracle |The name of the [Oracle Pluggable Database](https://docs.oracle.com/en/database/oracle/oracle-database/19/riwin/about-pluggable-databases-in-oracle-rac.html) that the connector captures changes from. Do not specify this property for a non-CDB installation.
Default: `"ORCLPDB1"` | -| `schema.exclude.list` | `string` | Oracle, PostgreSQL, SQL Server | An optional, comma-separated list of regular expressions that match names of schemas for which you do not want to capture changes. The connector captures changes in any schema whose name is not included in `schema.exclude.list`. Do not specify the `schemas` section if you are using the `schema.exclude.list` property to filter out schemas. | -| `table.exclude.list` | `string` | MariaDB, MySQL, Oracle, PostgreSQL, SQL Server | An optional comma-separated list of regular expressions that match fully-qualified table identifiers for the tables that you want to exclude from being captured; The connector captures all tables that are not included in `table.exclude.list`. Do not specify the `tables` block in the configuration if you are using the `table.exclude.list` property to filter out tables. | -| `column.exclude.list` | `string` | MariaDB, MySQL, Oracle, PostgreSQL, SQL Server | An optional comma-separated list of regular expressions that match the fully-qualified names of columns that should be excluded from change event message values. Fully-qualified names for columns are of the form `schemaName.tableName.columnName`. Do not specify the `columns` block in the configuration if you are using the `column.exclude.list` property to filter out columns. | -| `snapshot.select.statement.overrides` | `string` | MariaDB, MySQL, Oracle, PostgreSQL, SQL Server |Specifies the table rows to include in a snapshot. Use this property if you want a snapshot to include only a subset of the rows in a table. This property affects snapshots only. It does not apply to events that the connector reads from the log. See [Using custom queries in the initial snapshot](#custom-initial-query) below for more information. | -| `lob.enabled` | `string` | Oracle | Enables capturing and serialization of large object (CLOB, NCLOB, and BLOB) column values in change events.
Default: `false` | -| `unavailable.value.placeholder` | Special | Oracle | Specifies the constant that the connector provides to indicate that the original value is unchanged and not provided by the database (this has the type `__debezium_unavailable_value`). | - -### Oracle heartbeat mechanism - -{{< note >}} This section is only relevant for Oracle installations where many hours -or days may pass between changes to the source database. -{{< /note >}} - -Oracle uses *system change numbers* (SCNs) to record the order in which events happen -in a database. Any given event always has a higher SCN than any event that happened -previously, so the SCNs form a logical "timeline". (See -[Transactions](https://docs.oracle.com/cd/E11882_01/server.112/e40540/transact.htm#CNCPT016) -in the Oracle docs for more information.) - -For an Oracle source database, the RDI collector records the SCN of the most recent -transaction it has captured. When it checks the source for changes, it -uses this last recorded SCN to find all events that have happened in the meantime -and catch up with processing them. However, Oracle internally discards its SCN -information after a certain number of transactions have occurred. If these -transactions are against tables that RDI is *not* capturing, the last SCN -recorded by RDI might eventually be discarded by Oracle. When this happens, -the collector is unable to use its last SCN to detect new changes and data -may be lost as a result. - -If you are using Oracle as a source database and you expect updates to the -data to be very infrequent, you should enable the *heartbeat mechanism* in -the RDI collector. This writes data to a "dummy" table in the source on a -regular basis purely to generate a new SCN. To enable the heartbeat, -first create the dummy table with the following SQL: - -```sql -CREATE TABLE rdi_heartbeat ( - id NUMBER PRIMARY KEY, - last_heartbeat TIMESTAMP DEFAULT CURRENT_TIMESTAMP -); -``` +Defines source collectors and their configurations. Each key represents a unique source identifier, and its value contains specific configuration for that collector -Then, give WRITE/UPDATE permission to RDI on this table: +**Properties (Pattern)** -```sql -GRANT INSERT, UPDATE ON rdi_heartbeat TO rdi_user; -``` +| Name | Type | Description | Required | +| -------- | ---- | ----------- | -------- | +| **\.\*** | | | | -Finally, add the following lines to `config.yaml` under `sources.oracle.advanced`: + -```yaml -sources: - oracle: - advanced: - source: - heartbeat.interval.ms: 60000 - heartbeat.action.query: "MERGE INTO rdi_heartbeat t USING (SELECT 1 AS id FROM dual) s - ON (t.id = s.id) - WHEN MATCHED THEN UPDATE SET t.last_heartbeat = CURRENT_TIMESTAMP - WHEN NOT MATCHED THEN INSERT (id, last_heartbeat) VALUES (1, CURRENT_TIMESTAMP);" -``` +## processors: Data processing configuration -This sets the appropriate heartbeat update to occur once every minute. +Configuration settings that control how data is processed, including batch sizes, error handling, and performance tuning -### Using custom queries in the initial snapshot {#custom-initial-query} +**Properties** -{{< note >}}This section is relevant only for MySQL, MariaDB, Oracle, PostgreSQL, and SQL Server. -{{< /note >}} +| Name | Type | Description | Required | +| -------------------------------------------------------------------- | ------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------- | +| **on_failed_retry_interval**
(Retry interval on failure) | `integer`, `string` | Number of seconds to wait before retrying a failed operation
Default: `5`
Pattern: `^\${.*}$`
Minimum: `1`
| | +| **read_batch_size** | `integer`, `string` | Maximum number of records to process in a single batch
Default: `2000`
Pattern: `^\${.*}$`
Minimum: `1`
| | +| **dedup**
(Enable deduplication) | `boolean` | Enable the deduplication mechanism to handle duplicate records
Default: `false`
| | +| **dedup_max_size**
(Deduplication set size) | `integer` | Maximum number of entries to store in the deduplication set
Default: `1024`
Minimum: `1`
| | +| **dedup_strategy**
(Deduplication strategy) | `string` | (DEPRECATED)
Property 'dedup_strategy' is now deprecated. The only supported strategy is 'ignore'. Please remove from the configuration.
Default: `"ignore"`
Enum: `"reject"`, `"ignore"`
| | +| **duration**
(Batch duration limit) | `integer`, `string` | Maximum time in milliseconds to wait for a batch to fill before processing
Default: `100`
Pattern: `^\${.*}$`
Minimum: `1`
| | +| **write_batch_size** | `integer`, `string` | Maximum number of records to write to target Redis database in a single batch
Default: `200`
Pattern: `^\${.*}$`
Minimum: `1`
| | +| **error_handling**
(Error handling strategy) | `string` | Strategy for handling errors: ignore to skip errors, dlq to store rejected messages in dead letter queue
Default: `"dlq"`
Pattern: `^\${.*}$\|ignore\|dlq`
| | +| **dlq_max_messages**
(DLQ message limit) | `integer`, `string` | Maximum number of messages to store in dead letter queue per stream
Default: `1000`
Pattern: `^\${.*}$`
Minimum: `1`
| | +| **target_data_type**
(Target Redis data type) | `string` | Data type to use in Redis: hash for Redis Hash, json for RedisJSON (requires RedisJSON module)
Default: `"hash"`
Pattern: `^\${.*}$\|hash\|json`
| | +| **json_update_strategy** | `string` | Strategy for updating JSON data in Redis: replace to overwrite the entire JSON object, merge to merge new data with existing JSON object
Default: `"replace"`
Pattern: `^\${.*}$\|replace\|merge`
| | +| **initial_sync_processes** | `integer`, `string` | Number of parallel processes for performing initial data synchronization
Default: `4`
Pattern: `^\${.*}$`
Minimum: `1`
Maximum: `32`
| | +| **idle_sleep_time_ms**
(Idle sleep interval) | `integer`, `string` | Time in milliseconds to sleep between processing batches when idle
Default: `200`
Pattern: `^\${.*}$`
Minimum: `1`
Maximum: `999999`
| | +| **idle_streams_check_interval_ms**
(Idle streams check interval) | `integer`, `string` | Time in milliseconds between checking for new streams when processor is idle
Default: `1000`
Pattern: `^\${.*}$`
Minimum: `1`
Maximum: `999999`
| | +| **busy_streams_check_interval_ms**
(Busy streams check interval) | `integer`, `string` | Time in milliseconds between checking for new streams when processor is busy
Default: `5000`
Pattern: `^\${.*}$`
Minimum: `1`
Maximum: `999999`
| | +| **wait_enabled**
(Enable replica wait) | `boolean` | Enable verification that data has been written to replica shards
Default: `false`
| | +| **wait_timeout**
(Replica wait timeout) | `integer`, `string` | Maximum time in milliseconds to wait for replica write verification
Default: `1000`
Pattern: `^\${.*}$`
Minimum: `1`
| | +| **retry_on_replica_failure** | `boolean` | Continue retrying writes until successful replication to replica shards is confirmed
Default: `true`
| | -By default, the initial snapshot captures all rows from each table. -If you want the snapshot to include only a subset of the rows in a table, you can use a -custom `SELECT` statement to override the default and select only the rows you are interested in. -To do this, you must first specify the tables whose `SELECT` statement you want to override by adding a `snapshot.select.statement.overrides` in the `source` section with a comma-separated list of [fully-qualified table names](#fully-qualified-table-name). +**Additional Properties:** not allowed + -After the `snapshot.select.statement.overrides` list, you must then add another configuration property for each table in the list to specify the custom `SELECT` statement for that table. -The format of the property name depends on the database you are using: +## targets: Target connections -- For Oracle, SQL Server, and PostrgreSQL, use `snapshot.select.statement.overrides..` -- For MySQL and MariaDB, use: `snapshot.select.statement.overrides.` +Configuration for target Redis databases where processed data will be written -For example, with PostgreSQL, you would have a configuration like the following: +**Properties (Pattern)** -```yaml -source: - snapshot.select.statement.overrides: myschema.mytable - snapshot.select.statement.overrides.myschema.mytable: | - SELECT ... -``` +| Name | Type | Description | Required | +| -------- | ---- | ----------- | -------- | +| **\.\*** | | | | + + + +## secret\-providers: Secret providers + +Configuration for secret management providers + +**Properties (Pattern)** + +| Name | Type | Description | Required | +| --------------------------------------------------------- | -------- | ----------- | -------- | +| [**\.\***](#secret-providers)
(Secret provider entry) | `object` | | yes | + + -For MySQL, you would have: +#### secret\-providers\.\.\*: Secret provider entry + +**Properties** + +| Name | Type | Description | Required | +| ----------------------------------------------------------------------- | -------- | ----------------------------------------------------------------- | -------- | +| **type**
(Provider type) | `string` | Type of secret provider service
Enum: `"aws"`, `"vault"`
| yes | +| [**parameters**](#secret-providersparameters)
(Provider parameters) | `object` | Configuration parameters for the secret provider
| yes | + +**Additional Properties:** not allowed +**Example** ```yaml -source: - snapshot.select.statement.overrides: mydatabase.mytable - snapshot.select.statement.overrides.mydatabase.mytable: | - SELECT ... +parameters: + objects: + - {} ``` -You must also add the list of columns you want to include in the custom `SELECT` statement using fully-qualified names under "sources.tables". Specify each column in the configuration as shown below: + + +##### secret\-providers\.\.\*\.parameters: Provider parameters + +Configuration parameters for the secret provider + +**Properties** + +| Name | Type | Description | Required | +| ----------------------------------------------------------------------------- | ---------- | ------------------------------------------------------ | -------- | +| [**objects**](#secret-providersparametersobjects)
(Secrets objects array) | `object[]` | List of secret objects to fetch from the provider
| yes | + +**Example** ```yaml -tables: - # For MySQL and MariaDB: use database_name.table_name - schema_name.table_name: - columns: - - column_name1 # Each column on a new line - - column_name2 - - column_name3 +objects: + - {} ``` -If you want to capture all columns from a table, you can use empty curly braces `{}` instead of listing all the individual columns: + + +###### secret\-providers\.\.\*\.parameters\.objects\[\]: Secrets objects array - ```yaml - tables: - # Captures all columns. For MySQL and MariaDB: use database_name.table_name. - schema_name.table_name: {} - ``` +List of secret objects to fetch from the provider -The example configuration below selects the columns `CustomerId`, `FirstName` and `LastName` from the `customer` table and joins it with the `invoice` table to select customers with total invoices greater than 8000: +**Items: Secret object** + +**No properties.** + +**Example** ```yaml -tables: - chinook.customer: - columns: - - CustomerID - - FirstName - - LastName - -advanced: - source: - snapshot.select.statement.overrides: chinook.customer - snapshot.select.statement.overrides.chinook.customer: | - SELECT c.CustomerId, c.FirstName, c.LastName - FROM chinook.customer c - INNER JOIN chinook.invoice inv - ON c.CustomerId = inv.CustomerId - WHERE inv.total > 8000 +- {} ``` - -### Specifying custom message keys for change event records - -By default, Debezium uses the primary key column(s) of a table as the message key for the -records that it emits. However, you might want to configure custom message keys based on -one or more columns to override this default behavior, or to specify message keys for tables -that don't have a primary key. - -Use the `sources.tables` section of `config.yaml` to specify a custom message key for -one or more tables. List the columns you want to capture from the table under `columns` and -list the columns you want to use for the message key under `keys`, as shown below: - - ```yaml - # To include entries for multiple tables, simply add each table with its corresponding columns and keys under the 'tables' field. - tables: - .: - columns: - - # List of columns to include - keys: - - # Column(s) to be used together as the primary key - ``` - -Note that you must add the columns you use for the message key to both the `keys` list and the -`columns` list. You can use as many columns as you like to create the custom message keys but we -recommend you use only the minimum set of columns required to guarantee a unique key for each -message. - -## `processors`: RDI processors {#processors} - -### Properties - -| Name | Type | Description | -| -- | -- | -- | -| `on_failed_retry_interval` |`integer`, `string`| Interval (in seconds) between attempts to retry on failure.
Default: `5`
Pattern: `^\${.*}$`
Minimum: `1`| -| `read_batch_size` |`integer`, `string`| Batch size for reading data from the source database.
Default: `2000`
Pattern: `^\${.*}$`
Minimum: `1`| -| `debezium_lob_encoded_placeholder` |`string`| Enable Debezium LOB placeholders.
Default: `"__debezium_unavailable_value"`| -| `dedup` |`boolean`| Enable deduplication mechanism.
Default: `false`
|| -| `dedup_max_size` |`integer`| Maximum number of items in the deduplication set.
Default: `1024`
Minimum: `1`
| -| `dedup_strategy` |`string`| Deduplication strategy: `reject` - reject messages (dlq), `ignore` \- ignore messages.
(DEPRECATED)
The property `dedup_strategy` is now deprecated. The only supported strategy is `ignore`. Please remove from the configuration.
Default: `"ignore"`
Enum: `"reject"`, `"ignore"`
| -| `duration` |`integer`, `string`| Time (in ms) after which data will be read from stream even if `read_batch_size` was not reached.
Default: `100`
Pattern: `^\${.*}$`
Minimum: `1`
| -| `write_batch_size` |`integer`, `string`| The batch size for writing data to target Redis database\. Should be less or equal to `read_batch_size`.
Default: `200`
Pattern: `^\${.*}$`
Minimum: `1`
| -| `error_handling` |`string`| Error handling strategy: `ignore` - skip, `dlq` - store rejected messages in a dead letter queue.
Default: `"dlq"`
Pattern: `^\${.*}$\|ignore\|dlq`
| -| `dlq_max_messages` |`integer`, `string`| Maximum number of messages per stream in the dead letter queue .
Default: `1000`
Pattern: `^\${.*}$`
Minimum: `1`
| -| `target_data_type` |`string`| Target data type: `hash`/`json` (the RedisJSON module must be enabled in the target database to use JSON).
Default: `"hash"`
Pattern: `^\${.*}$\|hash\|json`
| -| `json_update_strategy` |`string`| Target update strategy: `replace`/`merge` (the RedisJSON module must be enabled in the target DB to use JSON).
(DEPRECATED)
The property `json_update_strategy` will be deprecated in future releases. Use the job-level property `on_update` to define the JSON update strategy.
Default: `"replace"`
Pattern: `^\${.*}$\|replace\|merge`
| -| `initial_sync_processes` |`integer`, `string`| Number of processes the RDI Engine creates to process the initial sync with the source.
Default: `4`
Pattern: `^\${.*}$`
Minimum: `1`
Maximum: `32`
| -| `idle_sleep_time_ms` |`integer`, `string`| Idle sleep time (in milliseconds) between batches.
Default: `200`
Pattern: `^\${.*}$`
Minimum: `1`
Maximum: `999999`
| -| `idle_streams_check_interval_ms` |`integer`, `string`| Interval (in milliseconds) for checking new streams when the stream processor is idling.
Default: `1000`
Pattern: `^\${.*}$`
Minimum: `1`
Maximum: `999999`
| -| `busy_streams_check_interval_ms` |`integer`, `string`| Interval (in milliseconds) for checking new streams when the stream processor is busy.
Default: `5000`
Pattern: `^\${.*}$`
Minimum: `1`
Maximum: `999999`
| -| `wait_enabled` |`boolean`| Check that the data has been written to the replica shard.
Default: `false`
| -| `wait_timeout` |`integer`, `string`| Timeout in milliseconds when checking writes to the replica shard.
Default: `1000`
Pattern: `^\${.*}$`
Minimum: `1`
| -| `retry_on_replica_failure` |`boolean`| Checks that the data has been written to the replica shard and keeps retrying if not.
Default: `true`
| - -### Additional properties - -Not allowed - -## `targets`: Target connections {#targets} - -## Properties - -| Name | Type | Description | -| -- | -- | -- | -| [`connection`](#targetsconnection) | `object` | Connection details | - -### `targets.connection`: Connection details {#targetsconnection} - -### Properties - -| Name | Type | Description | -| -- | -- | -- | -| `host` | `string` | IP address of the Redis database where RDI will write the processed data. | -| `port` | `integer` | Port of the Redis database where RDI will write the processed data. | -| `user` | `string` | User of the Redis database where RDI will write the processed data. Uncomment this if you are not using the default user. | -| `password` | `string` | Password for Redis target database. | -| `key` | `string` | Uncomment this line if you are using SSL/TLS. | -| `key_password` | `string` | Uncomment this line if you are using SSL/TLS. | -| `cert` | `string` | Uncomment this line if you are using SSL/TLS. | -| `cacert` | `string` | Uncomment this line if you are using SSL/TLS. | diff --git a/content/integrate/redis-data-integration/reference/data-transformation/_index.md b/content/integrate/redis-data-integration/reference/data-transformation/_index.md index f1fa56b043..9d1df9067c 100644 --- a/content/integrate/redis-data-integration/reference/data-transformation/_index.md +++ b/content/integrate/redis-data-integration/reference/data-transformation/_index.md @@ -3,15 +3,16 @@ Title: Data transformation reference aliases: /integrate/redis-data-integration/ingest/reference/data-transformation/ alwaysopen: false categories: -- docs -- integrate -- rs -- rdi + - docs + - integrate + - rs + - rdi description: View reference material for RDI data transformations group: di hideListLinks: false linkTitle: Data transformation -summary: Redis Data Integration keeps Redis in sync with the primary database in near +summary: + Redis Data Integration keeps Redis in sync with the primary database in near real time. type: integration weight: 60 diff --git a/content/integrate/redis-data-integration/reference/data-transformation/lookup.md b/content/integrate/redis-data-integration/reference/data-transformation/lookup.md index 919c0e16a8..478b148161 100644 --- a/content/integrate/redis-data-integration/reference/data-transformation/lookup.md +++ b/content/integrate/redis-data-integration/reference/data-transformation/lookup.md @@ -18,13 +18,13 @@ weight: 10 **Properties** -|Name|Type|Description|Required| -|----|----|-----------|--------| -| **connection** |`string`| Connection name |yes| -| **cmd** | `string` |The command to execute |yes| -| [**args**](#args) | `string[]` | Redis command arguments | yes | -| **language** | `string` |Language
Enum: `"jmespath"`, `"sql"`
| yes | -| **field** | `string` |The target field to write the result to
| yes | +| Name | Type | Description | Required | +| ----------------- | ---------- | --------------------------------------------- | -------- | +| **connection** | `string` | Connection name | yes | +| **cmd** | `string` | The command to execute | yes | +| [**args**](#args) | `string[]` | Redis command arguments | yes | +| **language** | `string` | Language
Enum: `"jmespath"`, `"sql"`
| yes | +| **field** | `string` | The target field to write the result to
| yes | **Additional Properties:** not allowed @@ -32,7 +32,6 @@ weight: 10 The list of expressions that produce arguments. - **Items** -**Item Type:** `string` \ No newline at end of file +**Item Type:** `string` diff --git a/content/integrate/redis-data-integration/reference/ports.md b/content/integrate/redis-data-integration/reference/ports.md index 2f3912075e..cecca95ab9 100644 --- a/content/integrate/redis-data-integration/reference/ports.md +++ b/content/integrate/redis-data-integration/reference/ports.md @@ -20,14 +20,14 @@ weight: 40 The table below shows the ports and protocols that RDI uses for its services: -| Name | Type | Port | Protocol | -| :-- | :-- | :-- | :-- | -| prometheus-service | Service | 9092 | TCP | -| prometheus-service | Ingress | 9092 | TCP/HTTP | -| rdi-api | Service | 8080 | TCP | -| rdi-api | Deployment | 8080 | TCP | -| rdi-api | Ingress | 8080 | TCP/HTTP | -| rdi-metric-exporter | Service | 9121 | TCP | -| rdi-metric-exporter | Ingress | 9121 | TCP/HTTP | -| rdi-operator | Deployment | 8080 | TCP | -| vm-dis-reloader | Deployment | 9090 | TCP | +| Name | Type | Port | Protocol | +| :------------------ | :--------- | :--- | :------- | +| prometheus-service | Service | 9092 | TCP | +| prometheus-service | Ingress | 9092 | TCP/HTTP | +| rdi-api | Service | 8080 | TCP | +| rdi-api | Deployment | 8080 | TCP | +| rdi-api | Ingress | 8080 | TCP/HTTP | +| rdi-metric-exporter | Service | 9121 | TCP | +| rdi-metric-exporter | Ingress | 9121 | TCP/HTTP | +| rdi-operator | Deployment | 8080 | TCP | +| vm-dis-reloader | Deployment | 9090 | TCP |