Skip to content

Commit 8621b9b

Browse files
DOC-4852 added RDI heartbeat mechanism info
1 parent 345a5a3 commit 8621b9b

File tree

1 file changed

+56
-0
lines changed

1 file changed

+56
-0
lines changed

content/integrate/redis-data-integration/reference/config-yaml-reference.md

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -114,6 +114,62 @@ See the Debezium documentation for more information about the specific connector
114114
| `lob.enabled` | `string` | Oracle | Enables capturing and serialization of large object (CLOB, NCLOB, and BLOB) column values in change events.<br/>Default: `false` |
115115
| `unavailable.value.placeholder` | Special | Oracle | Specifies the constant that the connector provides to indicate that the original value is unchanged and not provided by the database (this has the type `__debezium_unavailable_value`). |
116116

117+
### Oracle heartbeat mechanism
118+
119+
{{< note >}} This section is only relevant for Oracle installations where many hours
120+
or days may pass between changes to the source database.
121+
{{< /note >}}
122+
123+
Oracle uses *system change numbers* (SCNs) to record the order in which events happen
124+
in a database. Any given event always has a higher SCN than any event that happened
125+
previously, so the SCNs form a logical "timeline". (See
126+
[Transactions](https://docs.oracle.com/cd/E11882_01/server.112/e40540/transact.htm#CNCPT016)
127+
in the Oracle docs for more information.)
128+
129+
For an Oracle source database, the RDI collector records the SCN of the most recent
130+
transaction it has captured. If the collector gets stopped and later restarted, it
131+
uses this last recorded SCN to find all events that have happened in the meantime,
132+
and catch up with processing them. However, Oracle internally discards its SCN
133+
information after a certain period of time. If RDI's last recorded SCN has been
134+
discarded by the Oracle database, then there is no way to detect which
135+
events have happened since the collector was stopped, and change data may be
136+
lost.
137+
138+
If you are using Oracle as a source database and you expect updates to the
139+
data to be very infrequent, you should enable the *heartbeat mechanism* in
140+
the RDI collector. This writes data to a "dummy" table in the source on a
141+
regular basis purely to generate a new SCN. To enable the heartbeat,
142+
first create the dummy table with the following SQL:
143+
144+
```sql
145+
CREATE TABLE rdi_heartbeat (
146+
id NUMBER PRIMARY KEY,
147+
last_heartbeat TIMESTAMP DEFAULT CURRENT_TIMESTAMP
148+
);
149+
```
150+
151+
Then, give WRITE/UPDATE permission to RDI on this table:
152+
153+
```sql
154+
GRANT INSERT, UPDATE ON rdi_heartbeat TO rdi_user;
155+
```
156+
157+
Finally, add the following lines to `config.yaml` under `sources.oracle.advanced`:
158+
159+
```yaml
160+
sources:
161+
oracle:
162+
advanced:
163+
source:
164+
heartbeat.interval.ms: 60000
165+
heartbeat.action.query: "MERGE INTO rdi_heartbeat t USING (SELECT 1 AS id FROM dual) s
166+
ON (t.id = s.id)
167+
WHEN MATCHED THEN UPDATE SET t.last_heartbeat = CURRENT_TIMESTAMP
168+
WHEN NOT MATCHED THEN INSERT (id, last_heartbeat) VALUES (1, CURRENT_TIMESTAMP);"
169+
```
170+
171+
This sets the appropriate heartbeat update to occur once every minute.
172+
117173
### Using custom queries in the initial snapshot {#custom-initial-query}
118174

119175
{{< note >}}This section is relevant only for MySQL, MariaDB, Oracle, PostgreSQL, and SQL Server.

0 commit comments

Comments
 (0)