|
1 | 1 | (mongodb-cdc)= |
2 | 2 | # Relay MongoDB Change Stream into CrateDB |
3 | 3 |
|
4 | | -## About |
5 | | -[mongodb_cdc_cratedb.py] demonstrates a basic example program to relay event |
6 | | -records from [MongoDB Change Streams] into [CrateDB]. |
| 4 | +:::{rubric} What's inside |
| 5 | +::: |
| 6 | + |
| 7 | +Documentation and [example program][mongodb_cdc_cratedb.py] how to relay data |
| 8 | +from MongoDB into [CrateDB], using [MongoDB Change Streams]. |
7 | 9 |
|
8 | 10 | > Change streams allow applications to access real-time data changes without the prior |
9 | 11 | > complexity and risk of manually tailing the oplog. Applications can use change streams |
10 | 12 | > to subscribe to all data changes on a single collection, a database, or an entire |
11 | 13 | > deployment, and immediately react to them. |
12 | 14 | > |
13 | | -> - https://www.mongodb.com/docs/manual/changeStreams/ |
14 | | -> - https://www.mongodb.com/developer/languages/python/python-change-streams/ |
| 15 | +> - [MongoDB Change Streams] |
| 16 | +> - [Monitor Data with Change Streams] |
| 17 | +
|
| 18 | +:::{note} |
| 19 | +`commons-codec` includes `MongoDBFullLoadTranslator` and `MongoDBCDCTranslator`. |
| 20 | +This document and example program is exclusively about the latter. |
| 21 | +::: |
15 | 22 |
|
| 23 | +## Prerequisites |
16 | 24 |
|
17 | | -## Services |
| 25 | +Start services CrateDB and MongoDB. |
18 | 26 |
|
19 | | -### CrateDB |
20 | | -Start CrateDB. |
| 27 | +:::{rubric} Start CrateDB |
| 28 | +::: |
21 | 29 | ```shell |
22 | | -docker run --rm -it --name=cratedb --publish=4200:4200 --env=CRATE_HEAP_SIZE=2g \ |
23 | | - crate:5.7 -Cdiscovery.type=single-node |
| 30 | +docker run --rm --name=cratedb --publish=4200:4200 --env=CRATE_HEAP_SIZE=2g \ |
| 31 | + docker.io/crate:latest '-Cdiscovery.type=single-node' |
24 | 32 | ``` |
25 | 33 |
|
26 | | -### MongoDB |
27 | | -Start MongoDB. |
| 34 | +:::{rubric} Start MongoDB |
| 35 | +::: |
28 | 36 | Please note that change streams are only available for replica sets and |
29 | 37 | sharded clusters, so let's define a replica set by using the |
30 | 38 | `--replSet rs-testdrive` option when starting the MongoDB server. |
31 | 39 | ```shell |
32 | | -docker run -it --rm --name=mongodb --publish=27017:27017 \ |
33 | | - mongo:7 mongod --replSet rs-testdrive |
| 40 | +docker run --rm --name=mongodb --publish=27017:27017 \ |
| 41 | + docker.io/mongo:8 mongod --replSet rs-testdrive |
34 | 42 | ``` |
35 | 43 |
|
36 | 44 | Now, initialize the replica set, by using the `mongosh` command to invoke |
37 | 45 | the `rs.initiate()` operation. |
38 | 46 | ```shell |
39 | 47 | export MONGODB_URL="mongodb://localhost/" |
40 | | -docker run -i --rm --network=host mongo:7 mongosh ${MONGODB_URL} <<EOF |
| 48 | +docker run -i --rm --network=host docker.io/mongo:8 mongosh ${MONGODB_URL} <<EOF |
41 | 49 |
|
| 50 | +disableTelemetry() |
42 | 51 | config = { |
43 | 52 | _id: "rs-testdrive", |
44 | 53 | members: [{ _id : 0, host : "localhost:27017"}] |
@@ -100,3 +109,4 @@ time, but, again, it has not been correctly initialized. |
100 | 109 | [CrateDB]: https://github.com/crate/crate |
101 | 110 | [mongodb_cdc_cratedb.py]: https://github.com/daq-tools/commons-codec/raw/main/examples/mongodb_cdc_cratedb.py |
102 | 111 | [MongoDB Change Streams]: https://www.mongodb.com/docs/manual/changeStreams/ |
| 112 | +[Monitor Data with Change Streams]: https://www.mongodb.com/developer/languages/python/python-change-streams/ |
0 commit comments