Merge pull request #1066 from redis/preview-rc-rdi

RC: RDI In the Cloud

Author: cmilesb

content/embeds/rc-rdi-secrets-encryption-permissions.md

@@ -0,0 +1,5 @@
+* **Encryption key**: Select the [encryption key](#create-encryption-key) you created earlier.
+
+* **Resource permissions**: Add the following permissions to your secret to allow the Redis data pipeline to access your secret. Replace `<AWS ACCOUNT ID>` with the AWS account ID for the Redis Cloud cluster that you saved earlier.
+
+{{< embed-md "rc-rdi-secrets-permissions.md" >}}

content/embeds/rc-rdi-secrets-permissions.md

@@ -0,0 +1,19 @@
+```json
+{
+    "Version" : "2012-10-17",
+    "Statement" : [ {
+        "Sid" : "RedisDataIntegrationRoleAccess",
+        "Effect" : "Allow",
+        "Principal" : "*",
+        "Action" : [ "secretsmanager:GetSecretValue", "secretsmanager:DescribeSecret" ],
+        "Resource" : "*",
+        "Condition" : {
+            "StringLike" : {
+                "aws:PrincipalArn" : "arn:aws:iam::<AWS ACCOUNT ID>:role/redis-data-pipeline-secrets-role"
+            }
+        }
+    } ]
+}
+```
+
+After you store this secret, you can view and copy the [Amazon Resource Name (ARN)](https://docs.aws.amazon.com/secretsmanager/latest/userguide/reference_iam-permissions.html#iam-resources) of your secret on the secret details page. Save the secret ARN to use when you [define your source database]({{<relref "/operate/rc/databases/rdi/define">}}).

content/operate/rc/changelog/october-2025.md

@@ -14,6 +14,16 @@ tags:
 - changelog
 ---
 
+## New features
+
+### Data Integration
+
+Redis Cloud now supports [Redis Data Integration (RDI)]({{< relref "/operate/rc/databases/rdi" >}}) to create data pipelines that ingest data from a supported primary database to Redis. 
+
+Using a data pipeline lets you have a cache that is always ready for queries. RDI Data pipelines ensure that any changes made to your primary database are captured in your Redis cache within a few seconds, preventing cache misses and stale data within the cache. 
+
+See [Data Integration]({{< relref "/operate/rc/databases/rdi" >}}) to learn how to set up data pipelines with Redis Cloud.
+
 ## Security fixes
 
 Redis Cloud has already been updated with a patches for the following vulnerabilities: 

content/operate/rc/databases/rdi/_index.md

@@ -0,0 +1,78 @@
+---
+Title: Data Integration
+alwaysopen: false
+categories:
+- docs
+- operate
+- rc
+description: Use Redis Data Integration with Redis Cloud.
+hideListLinks: true
+weight: 99
+---
+
+Redis Cloud now supports [Redis Data Integration (RDI)]({{<relref "integrate/redis-data-integration">}}), a fast and simple way to bring your data into Redis from other types of primary databases.
+
+A relational database usually handles queries much more slowly than a Redis database. If your application uses a relational database and makes many more reads than writes (which is the typical case) then you can improve performance by using Redis as a cache to handle the read queries quickly. Redis Cloud uses [ingest]({{<relref "/integrate/redis-data-integration/">}}) to help you offload all read queries from the application database to Redis automatically.
+
+Using a data pipeline lets you have a cache that is always ready for queries. RDI Data pipelines ensure that any changes made to your primary database are captured in your Redis cache within a few seconds, preventing cache misses and stale data within the cache. 
+
+RDI helps Redis customers sync Redis Cloud with live data from their primary databases to:
+- Meet the required speed and scale of read queries and provide an excellent and predictable user experience.
+- Save resources and time when building pipelines and coding data transformations.
+- Reduce the total cost of ownership by saving money on expensive database read replicas.
+
+Using RDI with Redis Cloud simplifies managing your data integration pipeline. No need to worry about hardware or underlying infrastructure, as Redis Cloud manages that for you. Creating the data flow from source to target is much easier, and there are validations in place to reduce errors.
+
+## Data pipeline architecture
+
+An RDI data pipeline sits between your source database and your target Redis database. Initially, the pipeline reads all of the data and imports it into the target database during the *initial sync* phase. After this initial sync is complete, the data pipeline enters the *streaming* phase, where changes are captured as they happen. Changes in the source database are added to the target within a few seconds of capture. The data pipeline translates relational database rows to Redis hashes or JSON documents. 
+
+For more info on how RDI works, see [RDI Architecture]({{<relref "/integrate/redis-data-integration/architecture">}}).
+
+### Pipeline security
+
+Data pipelines are set up to ensure a high level of data security. Source database credentials and TLS secrets are stored in AWS secret manager and shared using the Kubernetes CSI driver for secrets. See [Share source database credentials]({{<relref "/operate/rc/databases/rdi/setup#share-source-database-credentials">}}) to learn how to share your source database credentials and TLS certificates with Redis Cloud.
+
+Connections to the source database use Java Database Connectivity (JDBC) through [AWS PrivateLink](https://aws.amazon.com/privatelink/), ensuring that the data pipeline is only exposed to the specific database endpoint. See [Set up connectivity]({{<relref "/operate/rc/databases/rdi/setup#set-up-connectivity">}}) to learn how to connect your PrivateLink to the Redis Cloud VPC.
+
+RDI encrypts all network connections with TLS. The pipeline will process data from the source database in-memory and write it to the target database using a TLS connection. There are no external connections to your data pipeline except from Redis Cloud management services.
+
+## Prerequisites
+
+Before you can create a data pipeline, you must have:
+
+- A [Redis Cloud Pro database]({{< relref "/operate/rc/databases/create-database/create-pro-database-new" >}}) hosted on Amazon Web Services (AWS). This will be the target database.
+- One supported source database, hosted on an AWS EC2 instance, AWS RDS, or AWS Aurora:
+
+| Database | Versions | AWS RDS  Versions |
+|:---|:---|:---|
+| Oracle | 19c, 21c | 19c, 21c |
+| MariaDB | 10.5, 11.4.3 | 10.4 to 10.11, 11.4.3 |
+| MySQL | 5.7, 8.0.x, 8.2 | 8.0.x |
+| PostgreSQL | 10, 11, 12, 13, 14, 15, 16 | 11, 12, 13, 14, 15, 16 |
+| AWS Aurora PostgreSQL | 15 | 15 |
+| SQL Server | 2017, 2019, 2022 | 2016, 2017, 2019, 2022 |
+
+
+{{< note >}}
+Please be aware of the following limitations:
+
+- The target database must be a Redis Cloud Pro database hosted on Amazon Web Services (AWS). Redis Cloud Essentials databases and databases hosted on Google Cloud do not support Data Integration.
+- The target database must use multi-zone [high availability]({{< relref "/operate/rc/databases/configuration/high-availability" >}}).
+- The target database can use TLS, but can not use mutual TLS.
+- The target database cannot be in the same subscription as another database that has a data pipeline.
+- Source databases must also be hosted on AWS.
+- You must use a [custom encryption key on AWS](https://docs.aws.amazon.com/kms/latest/developerguide/create-keys.html) to create the instance hosting the database.
+- One source database can only be synced to one target database.
+- You must be able to set up AWS PrivateLink to connect your source database to your target database. RDI only works with AWS PrivateLink and not VPC Peering or other private connectivity options.
+- Mutual TLS is not supported for AWS RDS and AWS Aurora source databases.
+{{< /note >}} 
+
+## Get started
+
+To create a new data pipeline, you need to:
+
+1. [Prepare your source database]({{<relref "/operate/rc/databases/rdi/setup">}}) and any associated credentials.
+2. [Define the source connection and data pipeline]({{<relref "/operate/rc/databases/rdi/define">}}) by selecting which tables to sync.
+
+Once your data pipeline is defined, you can [view and edit]({{<relref "/operate/rc/databases/rdi/view-edit">}}) it.

content/operate/rc/databases/rdi/define.md

@@ -0,0 +1,116 @@
+---
+Title: Define data pipeline
+alwaysopen: false
+categories:
+- docs
+- operate
+- rc
+description: Define the source connection and data pipeline.
+hideListLinks: true
+weight: 2
+---
+
+After you have [prepared your source database]({{<relref "/operate/rc/databases/rdi/setup">}}) and connection information, you can set up your new pipeline. To do this:
+
+1. [Define the source connection](#define-source-connection) by entering all required source database information.
+2. [Define the data pipeline](#define-data-pipeline) by selecting the data that you want to sync from your source database to the target database.
+
+## Define source connection
+
+1. In the [Redis Cloud console](https://cloud.redis.io/), go to your target database and select the **Data Pipeline** tab.
+1. Select **Define source database**.
+    {{<image filename="images/rc/rdi/rdi-define-source-database.png" alt="The define source database button." width=200px >}}
+1. Enter a **Pipeline name**. 
+    {{<image filename="images/rc/rdi/rdi-define-pipeline-cidr.png" alt="The pipeline name and deployment CIDR fields." >}}
+1. A **Deployment CIDR** is automatically generated for you. If, for any reason, a CIDR is not generated, enter a valid CIDR that does not conflict with your applications or other databases.
+1. In the **Source database connectivity** section, enter the **PrivateLink service name** of the [PrivateLink connected to your source database]({{< relref "/operate/rc/databases/rdi/setup#set-up-connectivity" >}}).
+    {{<image filename="images/rc/rdi/rdi-define-connectivity.png" alt="The Source database connectivity section, with database connection details and connectivity options." >}}
+1. Enter your database details. This depends on your database type, and includes:
+    - **Port**: The database's port
+    - **Database**: Your database's name, or the root database *(PostgreSQL, Oracle only)*, or a comma-separated list of one or more databases you want to connect to *(SQL Server only)*
+    - **Database Server ID**: Unique ID for the replication client. Enter a number that is not used by any existing replication clients *(mySQL and mariaDB only)*
+    - **PDB**: Name of the Oracle pluggable database *(Oracle only)*
+1. Enter the ARN of your [database credentials secret]({{< relref "/operate/rc/databases/rdi/setup#create-database-credentials-secrets" >}}) in the **Source database secrets ARN** field.
+1. If your database requires TLS, select **Use TLS**. Enter the ARN of your [CA certificate secret]({{< relref "/operate/rc/databases/rdi/setup#create-database-credentials-secrets" >}}) in the **CA Cert Secret ARN** field.
+    {{<image filename="images/rc/rdi/rdi-define-tls.png" alt="The Source database connectivity section, with Use TLS selected and the CA Cert Secret ARN field." >}}
+1. If your database requires mTLS, select **Use mTLS**. Enter the ARN of your [Client certificate secret]({{< relref "/operate/rc/databases/rdi/setup#create-database-credentials-secrets" >}}) in the **Client Certificate Secret ARN** field and the ARN of your [Client key secret]({{< relref "/operate/rc/databases/rdi/setup#create-database-credentials-secrets" >}}) in the **Client Key Secret ARN** field.
+    {{<image filename="images/rc/rdi/rdi-define-mtls.png" alt="The Source database connectivity section, with Use TLS selected and the Client Certificate Secret ARN and Client Key Secret ARN fields." >}}
+1. If your database requires mTLS with a client key passphrase, enter the ARN of your [Client key passphrase secret]({{< relref "/operate/rc/databases/rdi/setup#create-database-credentials-secrets" >}}) in the **Please add a secret ARN for the password to use with the secret store** field.
+1. Select **Start pipeline setup**.
+    {{<image filename="images/rc/rdi/rdi-start-pipeline-setup.png" alt="The start pipeline setup button." width=200px >}}
+1. Redis Cloud will attempt to connect to PrivateLink. If your PrivateLink does not allow automatic acceptance of incoming connections, accept the incoming connection on AWS PrivateLink to proceed. See [Accept or Reject PrivateLink connection requests](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html#accept-reject-connection-requests).
+
+    If Redis Cloud can't find your PrivateLink connection, make sure that the PrivateLink service name is correct and that Redis Cloud is listed as an Allowed Principal for your VPC. See [Set up connectivity]({{<relref "/operate/rc/databases/rdi/setup#set-up-connectivity">}}) for more info.
+
+At this point, Redis Cloud will provision the pipeline infrastructure that will allow you to define your data pipeline. 
+
+{{<image filename="images/rc/rdi/rdi-pipeline-setup-in-progress.png" alt="The Pipeline setup in progress screen." width=75% >}}
+
+Pipelines are provisioned in the background. You aren't allowed to make changes to your data pipeline or to your database during provisioning. This process will take about an hour, so you can close the window and come back later.
+
+When your pipeline is provisioned, select **Complete setup**. You will then [define your data pipeline](#define-data-pipeline).
+
+{{<image filename="images/rc/rdi/rdi-complete-setup.png" alt="The complete setup button." width=200px >}}
+
+## Define data pipeline
+
+After your pipeline is provisioned, you will be able to define your pipeline. You will select the database schemas, tables, and columns that you want to import and synchronize with your primary database.
+
+### Configure a new pipeline
+
+1. In the [Redis Cloud console](https://cloud.redis.io/), go to your target database and select the **Data Pipeline** tab. If your pipeline is already provisioned, select **Complete setup** to go to the **Data modeling** section.
+    {{<image filename="images/rc/rdi/rdi-complete-setup.png" alt="The complete setup button." width=200px >}}
+1. Select the Schema and Tables you want to migrate to the target database from the list. 
+    {{<image filename="images/rc/rdi/rdi-select-source-data.png" alt="The data modeling section. " width=75% >}}
+
+    Select **Manage Columns** to choose which columns you want to import.
+
+    {{<image filename="images/rc/rdi/rdi-manage-columns.png" alt="The manage columns button." width=150px >}}
+
+    You can select any number of columns from a table.
+
+    {{<image filename="images/rc/rdi/rdi-select-columns.png" alt="The manage columns screen, with a few columns selected from one table" width=75% >}}
+
+    If any tables are missing a unique constraint, a warning will appear in the **Data modeling** section. Select **Manage columns** to select the columns that define a unique constraint for those tables.
+
+    {{<image filename="images/rc/rdi/rdi-missing-unique-constraint.png" alt="The missing unique constraint list." width=75% >}}
+
+    {{<image filename="images/rc/rdi/rdi-select-constraints.png" alt="The missing unique constraint list with columns selected." width=75% >}}
+
+    Select **Save** to save your column changes and go back to schema selection.
+
+    {{<image filename="images/rc/button-save.png" alt="The save button." width=100px >}}
+
+    Select **Add schema** to add more database schemas.
+
+    {{<image filename="images/rc/rdi/rdi-add-schema.png" alt="The add schema button." width=150px >}}
+    
+    Select **Delete** to delete a schema. You must have at least one schema to continue.
+
+    {{<image filename="images/rc/rdi/rdi-delete-schema.png" alt="The delete schema button." width=50px >}}
+
+    After you've selected the schemas and tables you want to sync, select **Continue**.
+
+     {{<image filename="images/rc/rdi/rdi-continue-button.png" alt="The continue button." width=150px >}}
+    
+1. Select the Redis data type to write keys to the target. You can choose **Hash** or **JSON** if the target database supports JSON. 
+    {{<image filename="images/rc/rdi/rdi-configure-new-pipeline.png" alt="The pipeline definition screen." width=75% >}}
+
+    You can also supply one or more [transformation job files]({{< relref "/integrate/redis-data-integration/data-pipelines/transform-examples" >}}) that specify how you want to transform the captured data before writing it to the target. Select **Upload jobs** to upload your job files.
+
+    {{<image filename="images/rc/rdi/rdi-transformation-jobs.png" alt="The transformation jobs section. Select Upload jobs to upload transformation jobs." >}}
+
+    When you upload job files, Redis Cloud will validate the job files to check for errors. 
+
+    Select **Continue**.
+    {{<image filename="images/rc/rdi/rdi-continue-button.png" alt="The continue button." width=150px >}}
+
+1. Review the tables you selected in the **Review and deploy** section. If everything looks correct, select **Confirm & Deploy** to start ingesting data from your source database.
+
+    {{<image filename="images/rc/rdi/rdi-confirm-deploy.png" alt="The Confirm & Deploy button." width=175px >}}
+
+At this point, the data pipeline will ingest data from the source database to your target Redis database. This process will take time, especially if you have a lot of records in your source database. 
+
+After this initial sync is complete, the data pipeline enters the *change streaming* phase, where changes are captured as they happen. Changes in the source database are added to the target within a few seconds of capture. 
+
+You can view the status of your data pipeline in the **Data pipeline** tab of your database. See [View and edit data pipeline]({{<relref "/operate/rc/databases/rdi/view-edit">}}) to learn more.