-
Notifications
You must be signed in to change notification settings - Fork 270
RC: RDI in the cloud #570
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
RC: RDI in the cloud #570
Changes from 10 commits
Commits
Show all changes
57 commits
Select commit
Hold shift + click to select a range
c4d39e6
begin index and setup drafts
kaitlynmichael d586971
overview draft
kaitlynmichael b5beb6a
edit desc
kaitlynmichael 10d5de8
fix relref
kaitlynmichael c9e5161
get started, doc link
kaitlynmichael 42fcff9
prepare source database with link
kaitlynmichael 17270d8
setup connectivity
kaitlynmichael 9e0de8e
connectivity steps
kaitlynmichael efc8b0b
pane > box
kaitlynmichael 20c3050
Limitations draft
kaitlynmichael c3b7bca
Apply suggestions from code review
cmilesb 85c6d08
Merge branch 'main' into DOC-4137
cmilesb e938762
Edits and DB credentials changes
cmilesb a5977a2
Add setup step and more edits
cmilesb 289f0e4
Move last step to provision
cmilesb 1d6fce4
Merge branch 'main' into DOC-4137
cmilesb 272e518
Add some of Yaron's suggestions to intro
cmilesb 9d12553
Apply suggestions from code review
cmilesb 1551e1d
Merge branch 'main' into DOC-4137
cmilesb a32204e
Add define steps
cmilesb 07bc249
Merge branch 'main' into DOC-4137
cmilesb bc41625
Merge branch 'main' into DOC-4137
cmilesb 71f431c
Fix relrefs
cmilesb e8ea310
stash commit
cmilesb 860944a
Add security information
cmilesb 639fc50
Merge branch 'main' into DOC-4137
cmilesb 697d665
Merge branch 'main' into DOC-4137
cmilesb fa87f86
Add View/Edit
cmilesb 49d1f54
Incorporate Yaron's feedback, part 1
cmilesb 4abac23
Merge branch 'main' into DOC-4137
cmilesb 4d69274
stash commit
cmilesb 129d6af
More suggestions from Yaron
cmilesb 99229b1
Merge branch 'main' into DOC-4137
cmilesb c3e501e
Apply suggestions from code review
cmilesb 01907d8
Merge branch 'main' into DOC-4137
cmilesb c9469e0
Combine define and provision and remove errors section
cmilesb d56188a
Merge branch 'main' into DOC-4137
cmilesb f817848
Add secret permissions and keys
cmilesb 8f274f5
Update content/operate/rc/databases/rdi/define.md
cmilesb 2d2820d
Fix note
cmilesb 46603a6
replace account ID in resource permissions
cmilesb 89e0c4f
DOC-4548 Setup and define screenshots
cmilesb 962c1e9
Merge branch 'main' into DOC-4137
cmilesb 2b110b4
stash commit
cmilesb a51743b
Merge branch 'main' into DOC-4137
cmilesb 35c8dc5
Fix screenshot widths
cmilesb 52412cc
add edit pipeline images
cmilesb 7ef44a2
Apply suggestions from code review
cmilesb d0bf623
Merge branch 'main' into DOC-4137
cmilesb 07b45a0
Merge branch 'main' into DOC-4137
cmilesb ff2c061
Added metrics
cmilesb 79fee3a
Merge branch 'main' into DOC-4137
cmilesb 40891de
DOC-4674 Addressing feedback from Georgiana, Yaron, and Cornel
cmilesb 375cf15
Update content/operate/rc/databases/rdi/view-edit.md
cmilesb c8483de
Apply suggestions from code review
cmilesb ed0a757
Merge branch 'main' into DOC-4137
cmilesb 5ff8b8b
Merge branch 'main' into DOC-4137
cmilesb File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| --- | ||
| Title: Data Integration | ||
| alwaysopen: false | ||
| categories: | ||
| - docs | ||
| - operate | ||
| - rc | ||
| description: Use Redis Data Integration with Redis Cloud. | ||
| hideListLinks: true | ||
| weight: 99 | ||
| --- | ||
|
|
||
| Redis Cloud now supports [Redis Data Integration (RDI)]({{<relref "integrate/redis-data-integration">}}), a fast and simple way to bring your data into Redis. | ||
|
|
||
| The applications that write data to your primary database are not always the same application that need faster access to the data. Redis Cloud uses [ingest]({{<relref "/integrate/redis-data-integration/ingest/">}}) to help you offload all read queries from the application database to Redis. | ||
|
|
||
| RDI's helps Redis customers sync Redis Cloud with live data from their primary databases to: | ||
| - Meet the required speed and scale of read queries and provide an excellent and predictable user experience. | ||
| - Save resources and time when building pipelines and coding data transformations. | ||
| - Reduce the total cost of ownership by saving money on expensive database read replicas. | ||
cmilesb marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| Using RDI with Redis Cloud simplifies managing your data integration pipeline. No need to worry about hardware or underlying infrastructure, as Redis Cloud manages that for you. Creating the data flow from source to target is much easier, and there are validations in place to reduce errors. | ||
|
|
||
| ## Get started | ||
|
|
||
| To start creating your new data pipeline, see [Prepare source database]({{<relref "content/operate/rc/databases/rdi/setup.md">}}). | ||
cmilesb marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| ## Limitations | ||
|
|
||
| Only the following are supported at this time: | ||
|
|
||
| - One source database connected to one target database | ||
cmilesb marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| - AWS source databases | ||
| - Ingest use case | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| --- | ||
| Title: Prepare source database | ||
| alwaysopen: false | ||
| categories: | ||
| - docs | ||
| - operate | ||
| - rc | ||
| description: null | ||
| hideListLinks: true | ||
| weight: 1 | ||
| --- | ||
|
|
||
| ## Create new data pipeline | ||
|
|
||
| 1. Log in to Redis Cloud. | ||
| 1. (Create a new database){{<relref "/operate/rc/databases/create-database/create-pro-database-existing/">}}. | ||
| 1. Once your database is created and running, navigate to the **Data pipeline** tab. | ||
| 1. Select **Create data pipeline**. | ||
| 1. Select your database type (currently, only **TBD** is supported). | ||
| 1. If you know the size of your source database, enter it into the **Source dataset size** field. | ||
|
|
||
| ## Prepare source database | ||
|
|
||
| Before using the pipeline you must first prepare your source database to use | ||
| the Debezium connector for change data capture (CDC). | ||
|
|
||
| See [Prepare source databases]({{<relref "/integrate/redis-data-integration/ingest/data-pipelines/prepare-dbs/">}}) to find steps for your database type. | ||
cmilesb marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| See the [RDI architecture overview]({{< relref "/integrate/redis-data-integration/ingest/architecture#overview" >}}) for more information about CDC. | ||
cmilesb marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| ## Source database credentials | ||
|
|
||
| ## Setup connectivity | ||
|
|
||
| To expose your source database to Redis, you need to [edit your AWS PrivateLink VPC permissions](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html#add-remove-permissions). | ||
|
|
||
| 1. Open your [Amazon VPC console]() and navigate to **Endpoint services**. | ||
| 1. Copy the Amazon Resource Name (ARN) provided in **Setup connectivity** pane. | ||
| 1. Navigate to **Allow principals** tab. | ||
| 1. Add the Redis Cloud ARN and choose **Allow principals**. | ||
|
|
||
| For more details on AWS PrivateLink, see [Share your services through AWS PrivateLink](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-share-your-services.html). | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.