Skip to content

Commit 40c358d

Browse files
anders-swansonanders-swanson
authored
TxEventQ Advanced Features, kafka concepts (#1011)
Signed-off-by: Anders Swanson <[email protected]>
1 parent 29b97f2 commit 40c358d

File tree

6 files changed

+478
-7
lines changed

6 files changed

+478
-7
lines changed

‎docs-source/transactional-event-queues/content/getting-started/advanced-features.md‎

Lines changed: 373 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,4 +2,376 @@
22
archetype = "page"
33
title = "Advanced Features"
44
weight = 4
5-
+++
5+
+++
6+
7+
This section explains advanced features of Transactional Event Queues, including transactional messaging, message propagation between queues and the database, and error handling.
8+
9+
10+
* [Transactional Messaging: Combine Messaging with Database Queries](#transactional-messaging-combine-messaging-with-database-queries)
11+
* [SQL Example](#sql-example)
12+
* [Kafka Example](#kafka-example)
13+
* [Transactional Produce](#transactional-produce)
14+
* [Producer Methods](#producer-methods)
15+
* [Transactional Produce Example](#transactional-produce-example)
16+
* [Transactional Consume](#transactional-consume)
17+
* [Consumer Methods](#consumer-methods)
18+
* [Transactional Consume Example](#transactional-consume-example)
19+
* [Message Propagation](#message-propagation)
20+
* [Queue to Queue Message Propagation](#queue-to-queue-message-propagation)
21+
* [Removing Subscribers and Stopping Propagation](#removing-subscribers-and-stopping-propagation)
22+
* [Using Database Links](#using-database-links)
23+
* [Error Handling](#error-handling)
24+
25+
26+
## Transactional Messaging: Combine Messaging with Database Queries
27+
28+
Enqueue and dequeue operations occur within database transactions, allowing developers to combine database queries (DML) with messaging operations. This is particularly useful when the message contains data relevant to other tables or services within your schema.
29+
30+
### SQL Example
31+
32+
In the following example, a DML operation (an `INSERT` query) is combined with an enqueue operation in the same transaction. If the enqueue operation fails, the `INSERT` is rolled back. The orders table serves as the example.
33+
34+
```sql
35+
create table orders (
36+
id number generated always as identity primary key,
37+
product_id number not null,
38+
quantity number not null,
39+
order_date date default sysdate
40+
);
41+
42+
declare
43+
enqueue_options dbms_aq.enqueue_options_t;
44+
message_properties dbms_aq.message_properties_t;
45+
msg_id raw(16);
46+
message json;
47+
body varchar2(200) := '{"product_id": 1, "quantity": 5}';
48+
product_id number;
49+
quantity number;
50+
begin
51+
-- Convert the JSON string to a JSON object
52+
message := json(body);
53+
54+
-- Extract product_id and quantity from the JSON object
55+
product_id := json_value(message, '$.product_id' returning number);
56+
quantity := json_value(message, '$.quantity' returning number);
57+
58+
-- Insert data into the orders table
59+
insert into orders (product_id, quantity)
60+
values (product_id, quantity);
61+
62+
-- Enqueue the message
63+
dbms_aq.enqueue(
64+
queue_name => 'json_queue',
65+
enqueue_options => enqueue_options,
66+
message_properties => message_properties,
67+
payload => message,
68+
msgid => msg_id
69+
);
70+
commit;
71+
end;
72+
/
73+
```
74+
75+
> Note: The same pattern applies to the `dbms_aq.dequeue` procedure, allowing developers to perform DML operations within dequeue transactions.
76+
77+
### Kafka Example
78+
79+
The KafkaProducer and KafkaConsumer classes implemented by the [Kafka Java Client for Oracle Transactional Event Queues](https://github.com/oracle/okafka) provide functionality for transactional messaging, allowing developers to run database queries within a produce or consume transaction.
80+
81+
#### Transactional Produce
82+
83+
To configure a transactional producer, configure the org.oracle.okafka.clients.producer.KafkaProducer class with the `oracle.transactional.producer=true` property.
84+
85+
Once the producer instance is created, initialize database transactions with the `producer.initTransactions()` method.
86+
87+
```java
88+
Properties props = new Properties();
89+
// Use your database service name
90+
props.put("oracle.service.name", "freepdb1");
91+
// Choose PLAINTEXT or SSL as appropriate for your database connection
92+
props.put("security.protocol", "SSL");
93+
// Your database server
94+
props.put("bootstrap.servers", "my-db-server");
95+
// Path to directory containing ojdbc.properties
96+
// If using Oracle Wallet, this directory must contain the unzipped wallet (such as in sqlnet.ora)
97+
props.put("oracle.net.tns_admin", "/my/path/");
98+
props.put("enable.idempotence", "true");
99+
props.put("key.serializer", "org.apache.kafka.common.serialization.StringSerializer");
100+
props.put("value.serializer", "org.apache.kafka.common.serialization.StringSerializer");
101+
102+
// Enable Transactional messaging with the producer
103+
props.put("oracle.transactional.producer", "true");
104+
KafkaProducer<String, String> producer = new KafkaProducer<>(
105+
producerProps
106+
);
107+
108+
// Initialize the producer for database transactions
109+
producer.initTransactions();
110+
```
111+
112+
##### Producer Methods
113+
114+
- To start a database transaction, use the `producer.beginTransaction()` method.
115+
- To commit the transaction, use the `producer.commitTransaction()` method.
116+
- To retrieve the current database connection within the transaction, use the `producer.getDBConnection()` method.
117+
- To abort the transaction, use the `producer.abortTransaction()` method.
118+
119+
##### Transactional Produce Example
120+
121+
The following Java method takes in input record and processes it using a transactional producer. On error, the transaction is aborted and neither the DML nor topic produce are committed to the database. Assume the `processRecord` method does some DML operation with the record, like inserting or updating a table.
122+
123+
```java
124+
public void produce(String record) {
125+
// 1. Begin the current transaction
126+
producer.beginTransaction();
127+
128+
try {
129+
// 2. Create the producer record and prepare to send it to a topic
130+
ProducerRecord<String, String> pr = new ProducerRecord<>(
131+
topic,
132+
Integer.toString(idx),
133+
record
134+
);
135+
producer.send(pr);
136+
137+
// 3. Use the record in a database query
138+
processRecord(record, conn);
139+
} catch (Exception e) {
140+
// 4. On error, abort the transaction
141+
System.out.println("Error processing record", e);
142+
producer.abortTransaction();
143+
}
144+
145+
// 5. Once complete, commit the transaction
146+
producer.commitTransaction();
147+
System.out.println("Processed record");
148+
}
149+
```
150+
151+
#### Transactional Consume
152+
153+
To configure a transactional consumer, configure a org.oracle.okafka.clients.consumer.KafkaConsumer class with `auto.commit=false`. Disabling auto-commit will allow great control of database transactions through the `commitSync()` and `commitAsync()` methods.
154+
155+
```java
156+
Properties props = new Properties();
157+
// Use your database service name
158+
props.put("oracle.service.name", "freepdb1");
159+
// Choose PLAINTEXT or SSL as appropriate for your database connection
160+
props.put("security.protocol", "SSL");
161+
// Your database server
162+
props.put("bootstrap.servers", "my-db-server");
163+
// Path to directory containing ojdbc.properties
164+
// If using Oracle Wallet, this directory must contain the unzipped wallet (such as in sqlnet.ora)
165+
props.put("oracle.net.tns_admin", "/my/path/");
166+
167+
props.put("group.id" , "MY_CONSUMER_GROUP");
168+
// Set auto-commit to false for direct transaction management.
169+
props.put("enable.auto.commit","false");
170+
props.put("max.poll.records", 2000);
171+
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
172+
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
173+
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
174+
```
175+
176+
##### Consumer Methods
177+
178+
- To retrieve the current database connection within the transaction, use the `consumer.getDBConnection()` method.
179+
- To commit the current transaction synchronously, use the `consumer.commitSync()` method.
180+
- To commit the current transaction asynchronously, use the `consumer.commitAsync()` method.
181+
182+
##### Transactional Consume Example
183+
184+
The following Java method demonstrates how to use a KafkaConsumer for transactional messaging. Assume the `processRecord` method does some DML operation with the record, like inserting or updating a table.
185+
186+
```java
187+
public void run() {
188+
this.consumer.subscribe(List.of("topic1"));
189+
while (true) {
190+
try {
191+
// 1. Poll a batch of records from the subscribed topics
192+
ConsumerRecords<String, String> records = consumer.poll(
193+
Duration.ofMillis(100)
194+
);
195+
System.out.println("Consumed records: " + records.count());
196+
// 2. Get the current transaction's database connection
197+
Connection conn = consumer.getDBConnection();
198+
for (ConsumerRecord<String, String> record : records) {
199+
// 3. Do some DML with the record and connection
200+
processRecord(record, conn);
201+
}
202+
203+
// 4. Do a blocking commit on the current batch of records. For non-blocking, use commitAsync()
204+
consumer.commitSync();
205+
} catch (Exception e) {
206+
// 5. Since auto-commit is disabled, transactions are not
207+
// committed when commitSync() is not called.
208+
System.out.println("Unexpected error processing records. Aborting transaction!");
209+
}
210+
}
211+
}
212+
```
213+
214+
## Message Propagation
215+
216+
Messages can be propagated within the same database or across a [database link](https://docs.oracle.com/en/database/oracle/oracle-database/23/sqlrf/CREATE-DATABASE-LINK.html) to different queues or topics. Message propagation is useful for workflows that require message processing d by different consumers or for event-driven actions that need to trigger subsequent processes.
217+
218+
#### Queue to Queue Message Propagation
219+
220+
Create and start two queues. q1 will be the source queue, and q2 will be the propagated queue.
221+
222+
```sql
223+
begin
224+
dbms_aqadm.create_transactional_event_queue(
225+
queue_name => 'q1',
226+
queue_payload_type => 'JSON',
227+
multiple_consumers => true
228+
);
229+
dbms_aqadm.start_queue(
230+
queue_name => 'q1'
231+
);
232+
dbms_aqadm.create_transactional_event_queue(
233+
queue_name => 'q2',
234+
queue_payload_type => 'JSON',
235+
multiple_consumers => true
236+
);
237+
dbms_aqadm.start_queue(
238+
queue_name => 'q2'
239+
);
240+
end;
241+
/
242+
```
243+
244+
Add a subscriber to q2 using the [`DBMS_AQADM.ADD_SUBSCRIBER` procedure](https://docs.oracle.com/en/database/oracle/oracle-database/23/arpls/DBMS_AQADM.html#GUID-2B4498B0-7851-4520-89DD-E07FC4C5B2C7):
245+
246+
```sql
247+
begin
248+
dbms_aqadm.add_subscriber(
249+
queue_name => 'q2',
250+
subscriber => sys.aq$_agent(
251+
'q2_test_subscriber',
252+
null,
253+
null
254+
)
255+
);
256+
end;
257+
/
258+
```
259+
260+
Schedule message propagation so messages from q1 are propagated to q2, using the [`DBMS_AQADM.SCHEDULE_PROPAGATION` procedure](https://docs.oracle.com/en/database/oracle/oracle-database/23/arpls/DBMS_AQADM.html#GUID-E97FCD3F-D96B-4B01-A57F-23AC9A110A0D):
261+
262+
```sql
263+
begin
264+
dbms_aqadm.schedule_propagation(
265+
queue_name => 'q1',
266+
destination_queue => 'q2',
267+
latency => 0, -- latency, in seconds, before propagating
268+
start_time => sysdate, -- begin propagation immediately
269+
duration => null -- propagate until stopped
270+
);
271+
end;
272+
/
273+
```
274+
275+
Let's enqueue a message into q1. We expect this message to be propagated to q2:
276+
277+
```sql
278+
declare
279+
enqueue_options dbms_aq.enqueue_options_t;
280+
message_properties dbms_aq.message_properties_t;
281+
msg_id raw(16);
282+
message json;
283+
body varchar2(200) := '{"content": "this message is propagated!"}';
284+
begin
285+
select json(body) into message;
286+
dbms_aq.enqueue(
287+
queue_name => 'q1',
288+
enqueue_options => enqueue_options,
289+
message_properties => message_properties,
290+
payload => message,
291+
msgid => msg_id
292+
);
293+
commit;
294+
end;
295+
/
296+
```
297+
298+
#### Removing Subscribers and Stopping Propagation
299+
300+
You can remove subscribers and stop propagation using the DBMS_AQADM.STOP_PROPAGATION procedures:
301+
302+
```sql
303+
begin
304+
dbms_aqadm.unschedule_propagation(
305+
queue_name => 'q1',
306+
destination_queue => 'q2'
307+
);
308+
end;
309+
/
310+
```
311+
312+
Remove the subscriber:
313+
314+
```sql
315+
begin
316+
dbms_aqadm.remove_subscriber(
317+
queue_name => 'q2',
318+
subscriber => sys.aq$_agent(
319+
'q2_test_subscriber',
320+
null,
321+
null
322+
)
323+
);
324+
end;
325+
/
326+
```
327+
328+
Your can view queue subscribers and propagation schedules from the respective `DBA_QUEUE_SCHEDULES` and `DBA_QUEUE_SUBSCRIBERS` system views.
329+
330+
#### Using Database Links
331+
332+
To propagate messages between databases, a [database link](https://docs.oracle.com/en/database/oracle/oracle-database/23/sqlrf/CREATE-DATABASE-LINK.html) from the local database to the remote database must be created. The subscribe and propagation commands must be altered to use the database link.
333+
334+
```sql
335+
begin
336+
dbms_aqadm.schedule_propagation(
337+
queue_name => 'json_queue_1',
338+
destination => '<database link>.<schema name>' -- replace with your database link and schema name,
339+
destination_queue => 'json_queue_2'
340+
);
341+
end;
342+
/
343+
```
344+
345+
## Error Handling
346+
347+
Error handling is a critical component of message processing, ensuring malformed or otherwise unprocessable messages are handled correctly. Depending on the message payload and exception, an appropriate action should be taken to either replay or store the message for inspection.
348+
349+
If a message cannot be dequeued due to errors, it may be moved to the [exception queue](./message-operations.md#message-expiry-and-exception-queues), if one exists. You can handle such errors by using PL/SQL exception handling mechanisms.
350+
351+
```sql
352+
declare
353+
dequeue_options dbms_aq.dequeue_options_t;
354+
message_properties dbms_aq.message_properties_t;
355+
msg_id raw(16);
356+
message json;
357+
message_buffer varchar2(500);
358+
begin
359+
dequeue_options.navigation := dbms_aq.first_message;
360+
dequeue_options.wait := dbms_aq.no_wait;
361+
362+
dbms_aq.dequeue(
363+
queue_name => 'json_queue',
364+
dequeue_options => dequeue_options,
365+
message_properties => message_properties,
366+
payload => message,
367+
msgid => msg_id
368+
);
369+
select json_value(message, '$.content') into message_buffer;
370+
dbms_output.put_line('message: ' || message_buffer);
371+
exception
372+
when others then
373+
dbms_output.put_line('error dequeuing message: ' || sqlerrm);
374+
end;
375+
/
376+
```
377+

0 commit comments

Comments
 (0)