Skip to content

Commit a32204e

Browse files
committed
Add define steps
1 parent 1551e1d commit a32204e

File tree

5 files changed

+74
-39
lines changed

5 files changed

+74
-39
lines changed

content/operate/rc/databases/rdi/_index.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ For more info on how RDI works, see [RDI Architecture]({{<relref "/integrate/red
3535

3636
Before you can create a data pipeline, you must have:
3737

38-
- A [Redis Cloud Pro database]() hosted on Amazon Web Services (AWS)
38+
- A [Redis Cloud Pro database]({{< relref "/operate/rc/databases/create-database/create-pro-database-new" >}}) hosted on Amazon Web Services (AWS)
3939
- One supported source database, also hosted on AWS and connected to [AWS PrivateLink](https://aws.amazon.com/privatelink/):
4040
- MySQL
4141
- Oracle
@@ -56,7 +56,7 @@ Please be aware of the following limitations:
5656
To create a new data pipeline, you need to:
5757

5858
1. [Prepare your source database]({{<relref "content/operate/rc/databases/rdi/setup">}}) and any associated credentials.
59-
2. [Provision data pipeline]
60-
3. [Define data pipeline]
59+
2. [Create data pipeline]({{<relref "content/operate/rc/databases/rdi/create">}}) and troubleshoot errors.
60+
3. [Define the data pipeline]({{<relref "content/operate/rc/databases/rdi/define">}}) by selecting which tables to sync.
6161

6262
Once your data pipeline is defined, you can [manage]() and [observe]() it.
Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
---
2+
Title: Create data pipeline
3+
alwaysopen: false
4+
categories:
5+
- docs
6+
- operate
7+
- rc
8+
description: Create and troubleshoot your data pipeline.
9+
hideListLinks: true
10+
weight: 2
11+
---
12+
13+
After you have [prepared your source database]({{<relref "content/operate/rc/databases/rdi/setup">}}) and connection information, you can set up your new pipeline.
14+
15+
1. In the [Redis Cloud console](https://cloud.redis.io/), go to your target database and select the **Data Pipeline** tab.
16+
1. Select **Start pipeline setup**.
17+
1. Enter a **Pipeline name**. This pipeline name will be the prefix to all keys generated by this pipeline in the target database.
18+
1. Enter the **Deployment CIDR** for your pipeline, or use the one generated for you. This CIDR should not conflict with your apps or other databases.
19+
1. In the **Connectivity** section, enter the **PrivateLink service name** of the [PrivateLink connected to your source database]({{< relref "/operate/rc/databases/rdi/setup#set-up-connectivity" >}}).
20+
1. Enter your database details. This depends on your database type, and includes:
21+
- **Port**: The database's port
22+
- **Database**: Your database's name, or the root database
23+
- **Database Server ID**: Unique ID for the replication client. Leave as default if you don't use replication *(mySQL and mariaDB only)*
24+
- **PDB**: Name of the Oracle pluggable database *(Oracle only)*
25+
1. Enter the ARN of your [database credentials secret]({{< relref "/operate/rc/databases/rdi/setup#share-source-database-credentials" >}}) in the **Source database secrets ARN** field.
26+
1. Select **Start pipeline setup**.
27+
28+
At this point, Redis Cloud will attempt to create your new pipeline.
29+
30+
Pipelines are created in the background. While they're being created, you aren't allowed to make changes. This process will take some time.
31+
32+
See [Pipeline creation errors](#pipeline-creation-errors) to view a list of errors that can occur at this point.
33+
34+
When your pipeline is created, select **Complete setup**. You will then [define your data pipeline]({{<relref "content/operate/rc/databases/rdi/define">}}).
35+
36+
## Pipeline creation errors
37+
38+
Add errors here.
Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
---
2+
Title: Define data pipeline
3+
alwaysopen: false
4+
categories:
5+
- docs
6+
- operate
7+
- rc
8+
description: Define your data pipeline by selecting which tables to sync.
9+
hideListLinks: true
10+
weight: 3
11+
---
12+
13+
After you have [created your data pipeline]({{<relref "content/operate/rc/databases/rdi/create">}}), you need to define it. You will select the database schemas and columns that you want to import and synchronize with your primary database.
14+
15+
## Configure a new pipeline
16+
17+
1. In the [Redis Cloud console](https://cloud.redis.io/), go to your target database and select the **Data Pipeline** tab. If your pipeline is already created, select **Complete setup** to go to the **Pipeline definition** section.
18+
1. For the **Configure a new pipeline** option, select the Redis data type to write keys to the target. You can choose **Hash** or **JSON**.
19+
20+
Select **Continue**.
21+
1. Select the Schema and Tables you want to migrate to the target database from the **Source data selection** list.
22+
23+
If any tables are missing a unique constraint, the **Missing unique constraint** list will appear. Select the columns that define a unique constraint for those tables from the list.
24+
25+
After you've selected the tables you want to sync, select **Continue**.
26+
27+
1. Review the tables you selected in the **Summary**. If everything looks correct, select **Start ingest** to start ingesting data from your source database.
28+
29+
At this point, the data pipeline will ingest data from the source database to your target Redis database. This process will take time, especially if you have a lot of records in your source database.
30+
31+
After this initial sync is complete, the data pipeline enters the *change streaming* phase, where changes are captured as they happen. Changes in the source database are added to the target within a few seconds of capture.

content/operate/rc/databases/rdi/provision.md

Lines changed: 0 additions & 31 deletions
This file was deleted.

content/operate/rc/databases/rdi/setup.md

Lines changed: 2 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ categories:
55
- docs
66
- operate
77
- rc
8-
description: null
8+
description: Prepare your source database and database credentials for Data integration.
99
hideListLinks: true
1010
weight: 1
1111
---
@@ -35,9 +35,6 @@ You need to share your source database credentials in an Amazon secret with Redi
3535

3636
In the [AWS Management Console](https://console.aws.amazon.com/), use the **Services** menu to locate and select **Security, Identity, and Compliance** > **Secrets Manager**. [Create a secret](https://docs.aws.amazon.com/secretsmanager/latest/userguide/create_secret.html) of type **Other type of secret** with the following key/value fields:
3737

38-
- `port`: Database port
39-
- `schema_name`: Database name or schema name
40-
- `pdb_name`: Database name *(Oracle only)*
4138
- `username`: Database username
4239
- `password`: Database password
4340
- `server_certificate`: Server certificate in PEM format *(TLS only)*
@@ -66,4 +63,4 @@ For more details on AWS PrivateLink, see [Share your services through AWS Privat
6663

6764
## Next steps
6865

69-
After you have prepared your source database and provide
66+
After you have set up your source database and prepared connectivity and credentials, select **Start pipeline setup** to [create your data pipeline.]({{<relref "/operate/rc/databases/rdi/create.md/">}})

0 commit comments

Comments
 (0)