11(integrations-mongo-cdc)=
22# MongoDB CDC (Preview)
33
4- CrateDB Cloud enables continuous data ingestion from MongoDB using Change Data
4+ CrateDB Cloud enables continuous data ingestion from MongoDB using Change Data
55Capture (CDC), providing seamless, real-time synchronization of your data.
66
77:::{caution}
@@ -11,31 +11,31 @@ For more information, please [contact us](https://cratedb.com/contact).
1111
1212## Key Concepts
1313
14- The MongoDB CDC integration in CrateDB Cloud allows you to keep your data
15- synchronized between your MongoDB Atlas cluster and your CrateDB Cloud cluster
16- in real-time.
14+ The MongoDB CDC integration in CrateDB Cloud allows you to keep your data
15+ synchronized between your MongoDB Atlas cluster and your CrateDB Cloud cluster
16+ in real-time.
1717
1818### How It Works
1919
2020The integration functions in two main stages:
2121
22- 1 . ** Initial Sync:**
23- The integration performs a complete scan of your MongoDB collections,
22+ 1 . ** Initial Sync:**
23+ The integration performs a complete scan of your MongoDB collections,
2424 importing all existing data into your CrateDB Cloud cluster.
2525
26- 2 . ** Continuous Sync:**
27- The integration uses MongoDB Change Streams to monitor changes in your
28- MongoDB collections and syncs these updates to your CrateDB Cloud cluster
26+ 2 . ** Continuous Sync:**
27+ The integration uses MongoDB Change Streams to monitor changes in your
28+ MongoDB collections and syncs these updates to your CrateDB Cloud cluster
2929 in real-time, ensuring that your data remains current.
3030
3131### Data Consistency and Mode
3232
33- For continuous sync, CrateDB Cloud uses MongoDB's ** full document mode** to
34- ensure data consistency. This mode guarantees that MongoDB returns the latest
35- majority-committed version of the updated document.
33+ For continuous sync, CrateDB Cloud uses MongoDB's ** full document mode** to
34+ ensure data consistency. This mode guarantees that MongoDB returns the latest
35+ majority-committed version of the updated document.
3636
37- While receiving partial deltas is more efficient, full document mode provides
38- robust functionality. Support for partial deltas may be added in the future to
37+ While receiving partial deltas is more efficient, full document mode provides
38+ robust functionality. Support for partial deltas may be added in the future to
3939enhance performance and flexibility.
4040
4141---
@@ -55,81 +55,81 @@ Supported authentication methods:
5555The following steps should be performed in the MongoDB Atlas UI.
5656
5757#### Step 1: Create a Custom Role
58- 1 . ** Navigate to Database Access**
59- Go to ** Database Access** in the MongoDB Atlas UI for the cluster you want to
58+ 1 . ** Navigate to Database Access**
59+ Go to ** Database Access** in the MongoDB Atlas UI for the cluster you want to
6060 connect to CrateDB Cloud.
6161
62- 2 . ** Add a Custom Role**
62+ 2 . ** Add a Custom Role**
6363 Under ** Custom Roles** , click ** Add New Custom Role** .
6464
65- 3 . ** Set Up Read-Only Access**
65+ 3 . ** Set Up Read-Only Access**
6666 Assign the following actions or roles to the custom role:
6767 - ` find `
6868 - ` changeStream `
6969 - ` collStats `
7070
71- Specify the databases and collections for these actions. You can update
71+ Specify the databases and collections for these actions. You can update
7272 access permissions in the MongoDB Atlas UI later if needed.
7373
7474
7575#### Step 2: Create a User
7676
77- Depending on whether you plan to use SCRAM or X.509 authentication, create a
77+ Depending on whether you plan to use SCRAM or X.509 authentication, create a
7878database user with one of the following methods:
7979
8080:::{tab} SCRAM Auhentication
8181
82- 1 . ** Navigate to Database Access**
83- In the MongoDB Atlas UI, go to ** Database Access** and click ** Add New
82+ 1 . ** Navigate to Database Access**
83+ In the MongoDB Atlas UI, go to ** Database Access** and click ** Add New
8484 Database User** .
8585
86- 2 . ** Set Authentication Method**
87- Choose ** Password** as the authentication method and enter a username and
86+ 2 . ** Set Authentication Method**
87+ Choose ** Password** as the authentication method and enter a username and
8888 password for the database user.
8989
90- 3 . ** Assign the Role**
90+ 3 . ** Assign the Role**
9191 Under ** Database User Privileges** , select the custom role created in Step 1.
9292
93- 4 . ** Copy User Credentials**
94- Click ** Add User** , and make sure to record the username and password. These
93+ 4 . ** Copy User Credentials**
94+ Click ** Add User** , and make sure to record the username and password. These
9595 credentials will be used later in the CrateDB Cloud Console.
9696:::
9797
9898:::{tab} x.509 Authentication
9999
100- 1 . ** Navigate to Database Access**
101- In the MongoDB Atlas UI, go to ** Database Access** and click ** Add New
100+ 1 . ** Navigate to Database Access**
101+ In the MongoDB Atlas UI, go to ** Database Access** and click ** Add New
102102 Database User** .
103103
104- 2 . ** Set Authentication Method**
104+ 2 . ** Set Authentication Method**
105105 Choose ** Certificate** as the authentication method.
106106
107- 3 . ** Assign the Role**
107+ 3 . ** Assign the Role**
108108 Under ** Database User Privileges** , select the custom role created in Step 1.
109109
110- 4 . ** Save the Certificate**
111- Click ** Add User** , and store the certificate securely. This will be required
110+ 4 . ** Save the Certificate**
111+ Click ** Add User** , and store the certificate securely. This will be required
112112 later in the CrateDB Cloud Console.
113113:::
114114
115115
116116#### Step 3: Configure IP Access
117117
118- To allow CrateDB Cloud to access your MongoDB Atlas cluster, you must add the
118+ To allow CrateDB Cloud to access your MongoDB Atlas cluster, you must add the
119119CrateDB Cloud IP addresses to the IP Access List in MongoDB Atlas.
120120
121- 1 . ** Navigate to Network Access**
121+ 1 . ** Navigate to Network Access**
122122 In the MongoDB Atlas UI, go to ** Network Access** from the left navigation.
123123
124- 2 . ** Add IP Address**
125- Click ** Add IP Address** and choose an IP address or range to allow access.
126- For testing purposes, you can select ** Allow Access from Anywhere** , but for
124+ 2 . ** Add IP Address**
125+ Click ** Add IP Address** and choose an IP address or range to allow access.
126+ For testing purposes, you can select ** Allow Access from Anywhere** , but for
127127 production, it is recommended to specify only the required IPs.
128128
129129:::{note}
130- The specific IP addresses depend on the region of your CrateDB Cloud cluster.
131- These IP addresses can also be found in the ** Connection Details** section of the
132- CrateDB Cloud Console, just before you click ** Test Connection** during the
130+ The specific IP addresses depend on the region of your CrateDB Cloud cluster.
131+ These IP addresses can also be found in the ** Connection Details** section of the
132+ CrateDB Cloud Console, just before you click ** Test Connection** during the
133133setup process.
134134
135135 ** Outbound IP Addresses** :
@@ -156,17 +156,17 @@ out to our support team.
156156You'll need to provide the connection string for your MongoDB Atlas cluster so
157157that CrateDB Cloud can connect to it.
158158
159- 1 . ** Navigate to Your Cluster**
159+ 1 . ** Navigate to Your Cluster**
160160 In the MongoDB Atlas UI, navigate to the cluster you want to connect to CrateDB Cloud.
161161
162- 2 . ** Click "Connect"**
162+ 2 . ** Click "Connect"**
163163 From the cluster view, click on ** Connect** .
164164
165- 3 . ** Select "Connect Your Application"**
165+ 3 . ** Select "Connect Your Application"**
166166 Choose ** Connect your application** as the connection method.
167167
168- 4 . ** Copy the Connection String**
169- Copy the connection string provided in the MongoDB Atlas UI. It will look like this:
168+ 4 . ** Copy the Connection String**
169+ Copy the connection string provided in the MongoDB Atlas UI. It will look like this:
170170
171171```
172172mongodb+srv://:@/?retryWrites=true&w=majority
@@ -193,55 +193,55 @@ in CrateDB Cloud.
193193
194194Follow these steps in the CrateDB Cloud Console to set up the MongoDB CDC integration:
195195
196- #### Step 1: Create an Integration
197- 1 . Navigate to the ** Import** section in the CrateDB Cloud Console.
196+ #### Step 1: Create an Integration
197+ 1 . Navigate to the ** Import** section in the CrateDB Cloud Console.
1981982 . Click ** Create Integration** and select ** MongoDB** as the source type.
199199
200- #### Step 2: Configure Connection
200+ #### Step 2: Configure Connection
2012011 . Choose ** Create New Connection** or select an existing one.
2022022 . Fill in the following details:
203203 :::{tab} SCRAM Auhentication
204- - ** Connection Name** : Provide a unique name for the connection.
205- - ** Connection String** : Paste the connection string from MongoDB Atlas.
206- - ** Username** : Enter the database username (required for SCRAM).
204+ - ** Connection Name** : Provide a unique name for the connection.
205+ - ** Connection String** : Paste the connection string from MongoDB Atlas.
206+ - ** Username** : Enter the database username (required for SCRAM).
207207 - ** Password** : Enter the database password (required for SCRAM).
208208 - ** Default Database** : Specify the default database to use for this connection.
209209 :::
210210 :::{tab} X.509 Auhentication
211- - ** Connection Name** : Provide a unique name for the connection.
212- - ** Connection String** : Paste the connection string from MongoDB Atlas.
211+ - ** Connection Name** : Provide a unique name for the connection.
212+ - ** Connection String** : Paste the connection string from MongoDB Atlas.
213213 - ** Certificate** : Upload the X.509 certificate file.
214214 - ** Default Database** : Specify the default database to use for this connection.
215215 :::
216216
217- #### Step 3: Test the Connection
217+ #### Step 3: Test the Connection
218218Click ** Test Connection** to verify CrateDB Cloud can connect to your MongoDB
219219Atlas cluster. Resolve any issues if the test fails.
220220
221- #### Step 4: Select Collection
221+ #### Step 4: Select Collection
222222Enter the database and collection name from your MongoDB Atlas cluster, that you
223223want to sync with CrateDB Cloud.
224224
225- #### Step 5: Select Target Table
226- 1 . Specify the target table in your CrateDB Cloud cluster where the data will be synced.
227- 2 . MongoDB records will be inserted into an object column called ` document ` .
228- 3 . Select the object type for the column:
229- - ** ` dynamic ` ** : Allows indexing and columnar storage for faster querying.
230- - ** ` ignored ` ** : Prevents type conflicts in CrateDB if your source data lacks a strict schema.
225+ #### Step 5: Select Target Table
226+ 1 . Specify the target table in your CrateDB Cloud cluster where the data will be synced.
227+ 2 . MongoDB records will be inserted into an object column called ` document ` .
228+ 3 . Select the object type for the column:
229+ - ** ` dynamic ` ** : Allows indexing and columnar storage for faster querying.
230+ - ** ` ignored ` ** : Prevents type conflicts in CrateDB if your source data lacks a strict schema.
231231
232232 :::{note}
233- If your source data doesn't follow a strict schema, select ` ignored ` to avoid type conflicts.
234- However, selecting ` dynamic ` provides faster query performance by utilizing indexes and columnar storage.
233+ If your source data doesn't follow a strict schema, select ` ignored ` to avoid type conflicts.
234+ However, selecting ` dynamic ` provides faster query performance by utilizing indexes and columnar storage.
235235 :::
236236
237- #### Step 6: Configure Integration Settings
238- 1 . Enter a name for the integration.
239- 2 . Select the integration mode:
240- - ** Full Load Only** : Imports the data once but doesn’t sync changes.
241- - ** Full Load and CDC** : Imports the data and syncs changes in real-time.
237+ #### Step 6: Configure Integration Settings
238+ 1 . Enter a name for the integration.
239+ 2 . Select the integration mode:
240+ - ** Full Load Only** : Imports the data once but doesn’t sync changes.
241+ - ** Full Load and CDC** : Imports the data and syncs changes in real-time.
242242 - ** CDC Only** : Syncs only new changes in real-time without importing existing data.
243243
244- #### Step 7: Create the Integration
244+ #### Step 7: Create the Integration
245245Click ** Create Integration** to finalize the setup. CrateDB Cloud will now sync
246246your MongoDB data based on the selected settings.
247247
@@ -275,5 +275,5 @@ CDC integration:
275275 value.
276276- ** Binary data types** other than UUIDs, which are converted to ` TEXT ` and
277277 ** vectors** , which are converted to ` ARRAY ` s of numbers.
278- - The ` Decimal128 ` data type is not supported and is converted to a string, as
278+ - The ` Decimal128 ` data type is not supported and is converted to a string, as
279279 CrateDB does not support a decimal data type.
0 commit comments