Skip to content

Commit c06c99e

Browse files
authored
Merge pull request #111636 from M-Lipin/users/v-milipin/CosmosDB-kafka-connector-bulk-mode-doc
Adding Docs for Gateway and Bulk mode supports
2 parents f43a852 + 8b82d40 commit c06c99e

File tree

1 file changed

+32
-0
lines changed

1 file changed

+32
-0
lines changed

articles/cosmos-db/nosql/kafka-connector-sink.md

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -274,6 +274,38 @@ This error is likely caused by data in the source topic being serialized in eith
274274
"value.converter.schema.registry.url": "http://schema-registry:8081",
275275
```
276276

277+
### Gateway mode support
278+
```connect.cosmos.connection.gateway.enabled``` is a configuration option for the Cosmos DB Kafka Sink Connector that enhances data ingestion by utilizing the Cosmos DB gateway service. This service acts as a front-end for Cosmos DB, offering benefits such as load balancing, request routing, and protocol translation. By leveraging the gateway service, the connector achieves improved throughput and scalability when writing data to Cosmos DB. For more information, see [connectivity modes](/azure/cosmos-db/nosql/sdk-connection-modes).
279+
280+
```json
281+
"connect.cosmos.connection.gateway.enabled": true
282+
```
283+
284+
### Bulk mode support
285+
```connect.cosmos.sink.bulk.enabled``` property determines whether the bulk write feature is enabled for writing data from Kafka topics to Azure Cosmos DB.
286+
287+
When this property is set to `true` (by default), it enables the bulk write mode, allowing Kafka Connect to use the bulk import API of Azure Cosmos DB for performing efficient batch writes utilizing ```CosmosContainer.executeBulkOperations()``` method. Bulk write mode significantly improves the write performance and reduces the overall latency when ingesting data into Cosmos DB in comparison with non-bulk mode when ```CosmosContainer.upsertItem()``` method is used.
288+
289+
Bulk mode is enabled by default. To disable the `connect.cosmos.sink.bulk.enabled` property, you need to set it to `false` in the configuration for the Cosmos DB sink connector. Here's an example configuration property file:
290+
291+
```json
292+
"name": "my-cosmosdb-connector",
293+
"connector.class": "io.confluent.connect.azure.cosmosdb.CosmosDBSinkConnector",
294+
"tasks.max": 1,
295+
"topics": "my-topic"
296+
"connect.cosmos.endpoint": "https://<cosmosdb-account>.documents.azure.com:443/"
297+
"connect.cosmos.master.key": "<cosmosdb-master-key>"
298+
"connect.cosmos.database": "my-database"
299+
"connect.cosmos.collection": "my-collection"
300+
"connect.cosmos.sink.bulk.enabled": false
301+
```
302+
303+
By enabling the `connect.cosmos.sink.bulk.enabled` property, you can leverage the bulk write functionality in Kafka Connect for Azure Cosmos DB to achieve improved write performance when replicating data from Kafka topics to Azure Cosmos DB.
304+
305+
```json
306+
"connect.cosmos.sink.bulk.enabled": true
307+
```
308+
277309
### Read non-Avro data with AvroConverter
278310

279311
This scenario is applicable when you try to use the Avro converter to read data from a topic that isn't in Avro format. Which, includes data written by an Avro serializer other than the Confluent Schema Registry’s Avro serializer, which has its own wire format.

0 commit comments

Comments
 (0)