- 
                Notifications
    
You must be signed in to change notification settings  - Fork 270
 
RC: RDI in the cloud #570
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
          
     Merged
      
      
    
  
     Merged
                    RC: RDI in the cloud #570
Changes from 8 commits
      Commits
    
    
            Show all changes
          
          
            57 commits
          
        
        Select commit
          Hold shift + click to select a range
      
      c4d39e6
              
                begin index and setup drafts
              
              
                kaitlynmichael d586971
              
                overview draft
              
              
                kaitlynmichael b5beb6a
              
                edit desc
              
              
                kaitlynmichael 10d5de8
              
                fix relref
              
              
                kaitlynmichael c9e5161
              
                get started, doc link
              
              
                kaitlynmichael 42fcff9
              
                prepare source database with link
              
              
                kaitlynmichael 17270d8
              
                setup connectivity
              
              
                kaitlynmichael 9e0de8e
              
                connectivity steps
              
              
                kaitlynmichael efc8b0b
              
                pane > box
              
              
                kaitlynmichael 20c3050
              
                Limitations draft
              
              
                kaitlynmichael c3b7bca
              
                Apply suggestions from code review
              
              
                cmilesb 85c6d08
              
                Merge branch 'main' into DOC-4137
              
              
                cmilesb e938762
              
                Edits and DB credentials changes
              
              
                cmilesb a5977a2
              
                Add setup step and more edits
              
              
                cmilesb 289f0e4
              
                Move last step to provision
              
              
                cmilesb 1d6fce4
              
                Merge branch 'main' into DOC-4137
              
              
                cmilesb 272e518
              
                Add some of Yaron's suggestions to intro
              
              
                cmilesb 9d12553
              
                Apply suggestions from code review
              
              
                cmilesb 1551e1d
              
                Merge branch 'main' into DOC-4137
              
              
                cmilesb a32204e
              
                Add define steps
              
              
                cmilesb 07bc249
              
                Merge branch 'main' into DOC-4137
              
              
                cmilesb bc41625
              
                Merge branch 'main' into DOC-4137
              
              
                cmilesb 71f431c
              
                Fix relrefs
              
              
                cmilesb e8ea310
              
                stash commit
              
              
                cmilesb 860944a
              
                Add security information
              
              
                cmilesb 639fc50
              
                Merge branch 'main' into DOC-4137
              
              
                cmilesb 697d665
              
                Merge branch 'main' into DOC-4137
              
              
                cmilesb fa87f86
              
                Add View/Edit
              
              
                cmilesb 49d1f54
              
                Incorporate Yaron's feedback, part 1
              
              
                cmilesb 4abac23
              
                Merge branch 'main' into DOC-4137
              
              
                cmilesb 4d69274
              
                stash commit
              
              
                cmilesb 129d6af
              
                More suggestions from Yaron
              
              
                cmilesb 99229b1
              
                Merge branch 'main' into DOC-4137
              
              
                cmilesb c3e501e
              
                Apply suggestions from code review
              
              
                cmilesb 01907d8
              
                Merge branch 'main' into DOC-4137
              
              
                cmilesb c9469e0
              
                Combine define and provision and remove errors section
              
              
                cmilesb d56188a
              
                Merge branch 'main' into DOC-4137
              
              
                cmilesb f817848
              
                Add secret permissions and keys
              
              
                cmilesb 8f274f5
              
                Update content/operate/rc/databases/rdi/define.md
              
              
                cmilesb 2d2820d
              
                Fix note
              
              
                cmilesb 46603a6
              
                replace account ID in resource permissions
              
              
                cmilesb 89e0c4f
              
                DOC-4548 Setup and define screenshots
              
              
                cmilesb 962c1e9
              
                Merge branch 'main' into DOC-4137
              
              
                cmilesb 2b110b4
              
                stash commit
              
              
                cmilesb a51743b
              
                Merge branch 'main' into DOC-4137
              
              
                cmilesb 35c8dc5
              
                Fix screenshot widths
              
              
                cmilesb 52412cc
              
                add edit pipeline images
              
              
                cmilesb 7ef44a2
              
                Apply suggestions from code review
              
              
                cmilesb d0bf623
              
                Merge branch 'main' into DOC-4137
              
              
                cmilesb 07b45a0
              
                Merge branch 'main' into DOC-4137
              
              
                cmilesb ff2c061
              
                Added metrics
              
              
                cmilesb 79fee3a
              
                Merge branch 'main' into DOC-4137
              
              
                cmilesb 40891de
              
                DOC-4674 Addressing feedback from Georgiana, Yaron, and Cornel
              
              
                cmilesb 375cf15
              
                Update content/operate/rc/databases/rdi/view-edit.md
              
              
                cmilesb c8483de
              
                Apply suggestions from code review
              
              
                cmilesb ed0a757
              
                Merge branch 'main' into DOC-4137
              
              
                cmilesb 5ff8b8b
              
                Merge branch 'main' into DOC-4137
              
              
                cmilesb File filter
Filter by extension
Conversations
          Failed to load comments.   
        
        
          
      Loading
        
  Jump to
        
          Jump to file
        
      
      
          Failed to load files.   
        
        
          
      Loading
        
  Diff view
Diff view
There are no files selected for viewing
  
    
      This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
      Learn more about bidirectional Unicode characters
    
  
  
    
              | Original file line number | Diff line number | Diff line change | 
|---|---|---|
| @@ -0,0 +1,26 @@ | ||
| --- | ||
| Title: Data Integration | ||
| alwaysopen: false | ||
| categories: | ||
| - docs | ||
| - operate | ||
| - rc | ||
| description: Use Redis Data Integration with Redis Cloud. | ||
| hideListLinks: true | ||
| weight: 99 | ||
| --- | ||
| 
     | 
||
| Redis Cloud now supports [Redis Data Integration (RDI)]({{<relref "integrate/redis-data-integration">}}), a fast and simple way to bring your data into Redis. | ||
| 
     | 
||
| The applications that write data to your primary database are not always the same application that need faster access to the data. Redis Cloud uses [ingest]({{<relref "/integrate/redis-data-integration/ingest/">}}) to help you offload all read queries from the application database to Redis. | ||
| 
     | 
||
| RDI's helps Redis customers sync Redis Cloud with live data from their primary databases to: | ||
| - Meet the required speed and scale of read queries and provide an excellent and predictable user experience. | ||
| - Save resources and time when building pipelines and coding data transformations. | ||
| - Reduce the total cost of ownership by saving money on expensive database read replicas. | ||
                
      
                  cmilesb marked this conversation as resolved.
               
              
                Outdated
          
            Show resolved
            Hide resolved
         | 
||
| 
     | 
||
| Using RDI with Redis Cloud simplifies managing your data integration pipeline. No need to worry about hardware or underlying infrastructure, as Redis Cloud manages that for you. Creating the data flow from source to target is much easier, and there are validations in place to reduce errors. | ||
| 
     | 
||
| ## Get started | ||
| 
     | 
||
| To start creating your new data pipeline, see [Prepare source database]({{<relref "content/operate/rc/databases/rdi/setup.md">}}). | ||
                
      
                  cmilesb marked this conversation as resolved.
               
              
                Outdated
          
            Show resolved
            Hide resolved
         | 
||
  
    
      This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
      Learn more about bidirectional Unicode characters
    
  
  
    
              | Original file line number | Diff line number | Diff line change | 
|---|---|---|
| @@ -0,0 +1,42 @@ | ||
| --- | ||
| Title: Prepare source database | ||
| alwaysopen: false | ||
| categories: | ||
| - docs | ||
| - operate | ||
| - rc | ||
| description: null | ||
| hideListLinks: true | ||
| weight: 1 | ||
| --- | ||
| 
     | 
||
| ## Create new data pipeline | ||
| 
     | 
||
| 1. Log in to Redis Cloud. | ||
| 1. (Create a new database){{<relref "/operate/rc/databases/create-database/create-pro-database-existing/">}}. | ||
| 1. Once your database is created and running, navigate to the **Data pipeline** tab. | ||
| 1. Select **Create data pipeline**. | ||
| 1. Select your database type (currently, only **TBD** is supported). | ||
| 1. If you know the size of your source database, enter it into the **Source dataset size** field. | ||
| 
     | 
||
| ## Prepare source database | ||
| 
     | 
||
| Before using the pipeline you must first prepare your source database to use | ||
| the Debezium connector for change data capture (CDC). | ||
| 
     | 
||
| See [Prepare source databases]({{<relref "/integrate/redis-data-integration/ingest/data-pipelines/prepare-dbs/">}}) to find steps for your database type. | ||
                
      
                  cmilesb marked this conversation as resolved.
               
              
                Outdated
          
            Show resolved
            Hide resolved
         | 
||
| 
     | 
||
| See the [RDI architecture overview]({{< relref "/integrate/redis-data-integration/ingest/architecture#overview" >}}) for more information about CDC. | ||
                
      
                  cmilesb marked this conversation as resolved.
               
              
                Outdated
          
            Show resolved
            Hide resolved
         | 
||
| 
     | 
||
| ## Source database credentials | ||
| 
     | 
||
| ## Setup connectivity | ||
| 
     | 
||
| To expose your source database to Redis, you need to [edit your AWS PrivateLink VPC permissions](https://docs.aws.amazon.com/vpc/latest/privatelink/configure-endpoint-service.html#add-remove-permissions). | ||
| 
     | 
||
| 1. Open your [Amazon VPC console]() and navigate to **Endpoint services**. | ||
| 1. Copy the Amazon Resource Name (ARN) provided in **Setup connectivity** box. | ||
| 1. Navigate to **Allow principals** tab. | ||
| 1. Add the Redis Cloud ARN and choose **Allow principals**. | ||
| 
     | 
||
| For more details on AWS PrivateLink, see [Share your services through AWS PrivateLink](https://docs.aws.amazon.com/vpc/latest/privatelink/privatelink-share-your-services.html). | ||
  Add this suggestion to a batch that can be applied as a single commit.
  This suggestion is invalid because no changes were made to the code.
  Suggestions cannot be applied while the pull request is closed.
  Suggestions cannot be applied while viewing a subset of changes.
  Only one suggestion per line can be applied in a batch.
  Add this suggestion to a batch that can be applied as a single commit.
  Applying suggestions on deleted lines is not supported.
  You must change the existing code in this line in order to create a valid suggestion.
  Outdated suggestions cannot be applied.
  This suggestion has been applied or marked resolved.
  Suggestions cannot be applied from pending reviews.
  Suggestions cannot be applied on multi-line comments.
  Suggestions cannot be applied while the pull request is queued to merge.
  Suggestion cannot be applied right now. Please check back later.
  
    
  
    
Uh oh!
There was an error while loading. Please reload this page.