RC: RDI in the cloud

content/operate/rc/databases/rdi/_index.md

@@ -0,0 +1,71 @@
+---
+Title: Data Integration
+alwaysopen: false
+categories:
+- docs
+- operate
+- rc
+description: Use Redis Data Integration with Redis Cloud.
+hideListLinks: true
+weight: 99
+---
+
+Redis Cloud now supports [Redis Data Integration (RDI)]({{<relref "integrate/redis-data-integration">}}), a fast and simple way to bring your data into Redis from other types of primary databases.
+
+A relational database usually handles queries much more slowly than a Redis database. If your application uses a relational database and makes many more reads than writes (which is the typical case) then you can improve performance by using Redis as a cache to handle the read queries quickly. Redis Cloud uses [ingest]({{<relref "/integrate/redis-data-integration/">}}) to help you offload all read queries from the application database to Redis automatically.
+
+Using a data pipeline lets you have a cache that is always ready for queries. RDI Data pipelines ensure that any changes made to your primary database are captured in your Redis cache within a few seconds, preventing cache misses and stale data within the cache. 
+
+RDI helps Redis customers sync Redis Cloud with live data from their primary databases to:
+- Meet the required speed and scale of read queries and provide an excellent and predictable user experience.
+- Save resources and time when building pipelines and coding data transformations.
+- Reduce the total cost of ownership by saving money on expensive database read replicas.
+
+Using RDI with Redis Cloud simplifies managing your data integration pipeline. No need to worry about hardware or underlying infrastructure, as Redis Cloud manages that for you. Creating the data flow from source to target is much easier, and there are validations in place to reduce errors.
+
+## Data pipeline architecture
+
+A RDI data pipeline sits between your source database and your target Redis database. Initially, the pipeline reads all of the data and imports it into the target database during the *initial cache loading* phase. After this initial sync is complete, the data pipeline enters the *change streaming* phase, where changes are captured as they happen. Changes in the source database are added to the target within a few seconds of capture. The data pipeline translates relational database rows to Redis hashes or JSON documents. 
+
+For more info on how RDI works, see [RDI Architecture]({{<relref "/integrate/redis-data-integration/architecture">}}).
+
+### Pipeline security
+
+Data pipelines are set up to ensure a high level of data security. Source database credentials and TLS secrets are stored in AWS secret manager and shared using the AWS Secret Manager CSI driver for secrets. See [Share source database credentials]({{<relref "/operate/rc/databases/rdi/setup#share-source-database-credentials">}}) to learn how to share your source database credentials and TLS certificates with Redis Cloud.
+
+Connections to the source database use Java Database Connectivity (JDBC) through [AWS PrivateLink](https://aws.amazon.com/privatelink/), ensuring that the data pipeline is only exposed to the specific database endpoint. See [Set up connectivity]({{<relref "/operate/rc/databases/rdi/setup#set-up-connectivity">}}) to learn how to connect your PrivateLink to the Redis Cloud VPC.
+
+RDI encrypts all network connections with TLS. The pipeline will process data from the source database in-memory and write it to the target database using a TLS connection. There are no external connections to your data pipeline except from Redis Cloud management services.
+
+## Prerequisites
+
+Before you can create a data pipeline, you must have:
+
+- A [Redis Cloud Pro database]({{< relref "/operate/rc/databases/create-database/create-pro-database-new" >}}) hosted on Amazon Web Services (AWS). This will be the target database.
+- One supported source database, also hosted on AWS and connected to [AWS PrivateLink](https://aws.amazon.com/privatelink/):
+    - MySQL
+    - Oracle
+    - SQL Server
+    - PostgreSQL
+    - mariaDB
+    - Amazon Aurora
+
+{{< note >}}
+Please be aware of the following limitations:
+
+- The target database must be a Redis Cloud Pro database hosted on Amazon Web Services (AWS). Redis Cloud Essentials databases and databases hosted on Google Cloud do not support Data Integration.
+- The target database must use multi-zone [high availability]({{< relref "/operate/rc/databases/configuration/high-availability" >}}).
+- The target database can use TLS, but can not use mutual TLS.
+- The target database cannot be in the same subscription as another database that has a data pipeline.
+- Source databases must also be hosted on AWS.
+- One source database can only be synced to one target database.
+{{< /note >}} 
+
+## Get started
+
+To create a new data pipeline, you need to:
+
+1. [Prepare your source database]({{<relref "/operate/rc/databases/rdi/setup">}}) and any associated credentials.
+2. [Define the source connection and data pipeline]({{<relref "/operate/rc/databases/rdi/define">}}) by selecting which tables to sync.
+
+Once your data pipeline is defined, you can [view and edit]({{<relref "/operate/rc/databases/rdi/view-edit">}}) it.

content/operate/rc/databases/rdi/define.md

@@ -0,0 +1,95 @@
+---
+Title: Define data pipeline
+alwaysopen: false
+categories:
+- docs
+- operate
+- rc
+description: Define the source connction and data pipeline.
+hideListLinks: true
+weight: 2
+---
+
+After you have [prepared your source database]({{<relref "/operate/rc/databases/rdi/setup">}}) and connection information, you can set up your new pipeline. To do this:
+
+1. [Define the source connection](#define-source-connection) by entering all required source database information.
+2. [Define the data pipeline](#define-data-pipeline) by selecting the data that you want to sync from your source database to the target database.
+
+## Define source connection
+
+1. In the [Redis Cloud console](https://cloud.redis.io/), go to your target database and select the **Data Pipeline** tab.
+1. Select **Define source database**.
+    {{<image filename="images/rc/rdi/rdi-define-source-database.png" alt="The define source database button." width=200px >}}
+1. Enter a **Pipeline name**. This pipeline name will be the prefix to all keys generated by this pipeline in the target database.
+    {{<image filename="images/rc/rdi/rdi-define-pipeline-cidr.png" alt="The pipeline name and deployment CIDR fields." >}}
+1. Enter the **Deployment CIDR** for your pipeline, or use the one generated for you. This CIDR should not conflict with your apps or other databases.
+1. In the **Source database connectivity** section, enter the **PrivateLink service name** of the [PrivateLink connected to your source database]({{< relref "/operate/rc/databases/rdi/setup#set-up-connectivity" >}}).
+    {{<image filename="images/rc/rdi/rdi-define-connectivity.png" alt="The Source database connectivity section, with database connection details and connectivity options." >}}
+1. Enter your database details. This depends on your database type, and includes:
+    - **Port**: The database's port
+    - **Database**: Your database's name, or the root database *(PostgreSQL, Oracle only)*, or a comma-separated list of one or more databases you want to connect to *(SQL Server only)*
+    - **Database Server ID**: Unique ID for the replication client. Leave as default if you don't use replication *(mySQL and mariaDB only)*
+    - **PDB**: Name of the Oracle pluggable database *(Oracle only)*
+1. Enter the ARN of your [database credentials secret]({{< relref "/operate/rc/databases/rdi/setup#share-source-database-credentials" >}}) in the **Source database secrets ARN** field.
+1. Select **Start pipeline setup**.
+    {{<image filename="images/rc/rdi/rdi-start-pipeline-setup.png" alt="The start pipeline setup button." width=200px >}}
+1. Redis Cloud will attempt to connect to PrivateLink. If your PrivateLink does not allow automatic acceptance of incoming connections, accept the incoming connection on AWS PrivateLink to proceed. See [Accept or Reject PrivateLink connection requests](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html#accept-reject-connection-requests).
+
+    If Redis Cloud can't find your PrivateLink connection, make sure that the PrivateLink service name is correct and that Redis Cloud is listed as an Allowed Principal for your VPC. See [Set up connectivity]({{<relref "/operate/rc/databases/rdi/setup#set-up-connectivity">}}) for more info.
+
+At this point, Redis Cloud will provision the pipeline infrastructure that will allow you to define your data pipeline. 
+
+{{<image filename="images/rc/rdi/rdi-pipeline-setup-in-progress.png" alt="The Pipeline setup in progress screen." width=75% >}}
+
+Pipelines are provisioned in the background. You aren't allowed to make changes to your data pipeline or to your database during provisioning. This process will take a long time, so you can close the window and come back later.
+
+When your pipeline is provisioned, select **Complete setup**. You will then [define your data pipeline](#define-data-pipeline).
+
+{{<image filename="images/rc/rdi/rdi-complete-setup.png" alt="The complete setup button." width=200px >}}
+
+## Define data pipeline
+
+After your pipeline is provisioned, you will be able to define your pipeline. You will select the database schemas, tables, and columns that you want to import and synchronize with your primary database.
+
+### Configure a new pipeline
+
+1. In the [Redis Cloud console](https://cloud.redis.io/), go to your target database and select the **Data Pipeline** tab. If your pipeline is already provisioned, select **Complete setup** to go to the **Pipeline definition** section.
+    {{<image filename="images/rc/rdi/rdi-complete-setup.png" alt="The complete setup button." width=200px >}}
+1. For the **Configure a new pipeline** option, select the Redis data type to write keys to the target. You can choose **Hash** or **JSON** if the target database supports JSON. 
+    {{<image filename="images/rc/rdi/rdi-configure-new-pipeline.png" alt="The Pipeline definition screen. Configure a new pipeline is selected." width=75% >}}
+    Select **Continue**.
+    {{<image filename="images/rc/rdi/rdi-continue-button.png" alt="The continue button." width=150px >}}
+1. Select the Schema and Tables you want to migrate to the target database from the **Source data selection** list. 
+    {{<image filename="images/rc/rdi/rdi-select-source-data.png" alt="The select source data section. " width=75% >}}
+
+    You can select any number of columns from a table.
+
+    {{<image filename="images/rc/rdi/rdi-select-columns.png" alt="The select source data section. A table is expanded with a few columns selected." width=75% >}}
+
+    If any tables are missing a unique constraint, the **Missing unique constraint** list will appear. Select the columns that define a unique constraint for those tables from the list.
+
+    {{<image filename="images/rc/rdi/rdi-missing-unique-constraint.png" alt="The missing unique constraint list." width=75% >}}
+
+    {{<image filename="images/rc/rdi/rdi-select-constraints.png" alt="The missing unique constraint list with columns selected." width=75% >}}
+
+    Select **Add schema** to add more database schemas.
+
+    {{<image filename="images/rc/rdi/rdi-add-schema.png" alt="The add schema button." width=150px >}}
+
+    Select **Delete** to delete a schema. You must have at least one schema to continue.
+
+    {{<image filename="images/rc/rdi/rdi-delete-schema.png" alt="The delete schema button." width=50px >}}
+
+    After you've selected the schemas and tables you want to sync, select **Continue**.
+
+     {{<image filename="images/rc/rdi/rdi-continue-button.png" alt="The continue button." width=150px >}}
+
+1. Review the tables you selected in the **Summary**. If everything looks correct, select **Start ingest** to start ingesting data from your source database.
+
+    {{<image filename="images/rc/rdi/rdi-start-ingest.png" alt="The start ingest button." width=175px >}}
+
+At this point, the data pipeline will ingest data from the source database to your target Redis database. This process will take time, especially if you have a lot of records in your source database. 
+
+After this initial sync is complete, the data pipeline enters the *change streaming* phase, where changes are captured as they happen. Changes in the source database are added to the target within a few seconds of capture. 
+
+You can view the status of your data pipeline in the **Data pipeline** tab of your database. See [View and edit data pipeline]({{<relref "/operate/rc/databases/rdi/view-edit">}}) to learn more.

content/operate/rc/databases/rdi/setup.md

@@ -0,0 +1,125 @@
+---
+Title: Prepare source database
+alwaysopen: false
+categories:
+- docs
+- operate
+- rc
+description: Prepare your source database and database credentials for Data integration.
+hideListLinks: true
+weight: 1
+---
+
+## Create new data pipeline
+
+1. In the [Redis Cloud console](https://cloud.redis.io/), go to your target database and select the **Data Pipeline** tab.
+1. Select **Create data pipeline**.
+    {{<image filename="images/rc/rdi/rdi-create-data-pipeline.png" alt="The create data pipeline button." width=200px >}}
+1. Select your source database type. The following database types are supported:
+    - MySQL
+    - mariaDB
+    - Oracle
+    - SQL Server
+    - PostgreSQL
+    {{<image filename="images/rc/rdi/rdi-select-source-db.png" alt="The select source database type list." width=500px >}}
+1. If you know the size of your source database, enter it into the **Source dataset size** field.
+    {{<image filename="images/rc/rdi/rdi-source-dataset-size.png" alt="Enter the amount of source data you plan to ingest." width=400px >}}
+1. Under **Setup connectivity**, save the provided ARN and extract the AWS account ID for the account associated with your Redis Cloud cluster from it. 
+
+    {{<image filename="images/rc/rdi/rdi-setup-connectivity-arn.png" alt="The select source database type list." width=80% >}}
+
+    The AWS account ID is the string of numbers after `arn:aws:iam::` in the ARN. For example, if the ARN is `arn:aws:iam::123456789012:role/redis-data-pipeline`, the AWS account ID is `123456789012`.
+
+## Prepare source database
+
+Before using the pipeline, you must first prepare your source database to use the Debezium connector for change data capture (CDC).
+
+See [Prepare source databases]({{<relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/">}}) to find steps for your database type:
+- [MySQL and mariaDB]({{<relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/my-sql-mariadb">}})
+- [Oracle]({{<relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/oracle">}})
+- [SQL Server]({{<relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/sql-server">}})
+- [PostgreSQL]({{<relref "/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql">}})
+
+See the [RDI architecture overview]({{< relref "/integrate/redis-data-integration/architecture#overview" >}}) for more information about CDC.
+
+## Share source database credentials
+
+You need to share your source database credentials and certificates in an Amazon secret with Redis Cloud so that the pipeline can connect to your database.
+
+To do this, you need to:
+1. [Create an encryption key](#create-encryption-key) using AWS Key Management Service with the right permissions.
+1. [Create a secret](#create-database-credentials-secret) containing the source database credentials encrypted using that key.
+
+### Create encryption key
+
+In the [AWS Management Console](https://console.aws.amazon.com/), use the **Services** menu to locate and select **Security, Identity, and Compliance** > **Key Management Service**. [Create an encryption key](https://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html) with the following settings:
+
+1. In **Step 1 - Configure key**:
+    - **Key type**: Select **Symmetric**.
+    - **Key usage**: Select **Encrypt and decrypt**.
+    - Under **Advanced options**, set the following:
+        - **Key material origin**: Select **KMS - recommended**.
+        - **Regionality**: Select **Single-Region key**.
+1. In **Step 2 - Add labels**, add an alias and description for the key.
+1. In **Step 3 - Define key administrative permissions**, under **Key deletion**, select **Allow key administrators to delete this key**.
+1. In **Step 4 - Define key usage permissions**, under **Other AWS accounts**, select **Add another AWS account**. Enter the AWS account ID for the Redis Cloud cluster that you saved earlier.
+
+### Create database credentials secret
+
+In the [AWS Management Console](https://console.aws.amazon.com/), use the **Services** menu to locate and select **Security, Identity, and Compliance** > **Secrets Manager**. [Create a secret](https://docs.aws.amazon.com/secretsmanager/latest/userguide/create_secret.html) of type **Other type of secret** with the following settings:
+
+- **Key/value pairs**: Enter the following key/value pairs.
+
+    - `username`: Database username
+    - `password`: Database password
+    - `trust_certificate`: Server certificate in PEM format *(TLS only)*
+    - `client_public_key`: [X.509 client certificate](https://en.wikipedia.org/wiki/X.509) or chain in PEM format *(mTLS only)*
+    - `client_private_key`: Key for the client certificate or chain in PEM format *(mTLS only)*
+    - `client_private_key_passphrase`: Passphrase or password for the client certificate or chain in PEM format *(mTLS only)*
+
+    {{<note>}}
+If your source database has TLS or mTLS enabled, we recommend that you enter the `trust_certificate`, `client_public_key`, and `client_private_key` into the secret editor using the **Key/Value** input method instead of the **JSON** input method. Pasting directly into the JSON editor may cause an error. 
+    {{</note>}}
+
+- **Encryption key**: Select the [encryption key](#create-encryption-key) you created earlier.
+
+- **Resource permissions**: Add the following permissions to your secret to allow the Redis data pipeline to access your secret. Replace `<AWS ACCOUNT ID>` with the AWS account ID for the Redis Cloud cluster that you saved earlier.
+
+    ```json
+    {
+        "Version" : "2012-10-17",
+        "Statement" : [ {
+            "Sid" : "RedisDataIntegrationRoleAccess",
+            "Effect" : "Allow",
+            "Principal" : "*",
+            "Action" : [ "secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret" ],
+            "Resource" : "*",
+            "Condition" : {
+                "StringLike" : {
+                    "aws:PrincipalArn" : "arn:aws:iam::<AWS ACCOUNT ID>:role/redis-data-pipeline-secrets-role"
+                }
+            }
+        } ]
+    }
+    ```
+
+After you store this secret, you can view and copy the [Amazon Resource Name (ARN)](https://docs.aws.amazon.com/secretsmanager/latest/userguide/reference_iam-permissions.html#iam-resources) of your secret on the secret details page. 
+
+## Set up connectivity
+
+To expose your source database to Redis, you need to add Redis Cloud as an Allowed Principal on the [AWS PrivateLink VPC permissions](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html#add-remove-permissions) for the PrivateLink connected to your source database.
+
+1. Copy the Amazon Resource Name (ARN) provided in the **Setup connectivity** section.
+1. Open the [Amazon VPC console](https://console.aws.amazon.com/vpc/) and select **Endpoint services**.
+1. Navigate to **Allow principals** tab.
+1. Add the Redis Cloud ARN and choose **Allow principals**.
+1. Copy your PrivateLink service name for later.
+
+For more details on AWS PrivateLink, see [Share your services through AWS PrivateLink](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-share-your-services.html).
+
+
+## Next steps
+
+After you have set up your source database and prepared connectivity and credentials, select **Define source database** to [define your source connection and data pipeline]({{<relref "/operate/rc/databases/rdi/define">}}).
+
+{{<image filename="images/rc/rdi/rdi-define-source-database.png" alt="ADD ALT TEXT" width=200px >}}