[DE-627] Kafka Connector: batch writes (#354)

* Kafka Connector: batch writes

* Review

---------

Co-authored-by: Simran Spiller <[email protected]>

Author: rashtao

site/content/3.10/develop/integrations/kafka-connect-arangodb-sink-connector/_index.md

@@ -18,7 +18,7 @@ This connector is compatible with:
 
 - Kafka `2.x` (from version `2.6` onward) and Kafka `3.x` (all versions)
 - JDK 8 and higher versions
-- all the non-EOLed [ArangoDB versions](https://www.arangodb.com/eol-notice)
+- ArangoDB 3.11.1 and higher versions
 
 ## Installation
 
@@ -254,13 +254,11 @@ See [SSL configuration](configuration.md#ssl) for further options.
 ## Limitations
 
 - The `VST` communication protocol (`connection.protocol=VST`) is currently not working (DE-619)
-- Documents are inserted one by one, bulk inserts will be implemented in a future release (DE-627)
-- In case of transient error, the entire Kafka Connect batch is retried (DE-651)
 - Record values are required to be object-like structures (DE-644)
 - Auto-creation of ArangoDB collection is not supported (DE-653)
 - `ssl.cert.value` does not support multiple certificates (DE-655)
-- Batch inserts are not guaranteed to be executed serially (FRB-300)
-- Batch inserts may succeed for some documents while failing for others (FRB-300)
+- Batch writes are not guaranteed to be executed serially (FRB-300)
+- Batch writes may succeed for some documents while failing for others (FRB-300)
   This has two important consequences:
   - Transient errors might be retried and succeed at a later point
   - Data errors might be asynchronously reported to the DLQ

site/content/3.10/develop/integrations/kafka-connect-arangodb-sink-connector/configuration.md

@@ -173,6 +173,14 @@ to `update`:
 - `true`: objects are merged
 - `false`: existing document fields are overwritten
 
+### batch.size
+
+- type: _int_
+- default: `3_000`
+
+Specifies how many records to attempt to batch together for insertion or deletion
+into the destination collection.
+
 ### insert.timeout.ms
 
 - type: _int_

site/content/3.11/develop/integrations/kafka-connect-arangodb-sink-connector/_index.md

@@ -18,7 +18,7 @@ This connector is compatible with:
 
 - Kafka `2.x` (from version `2.6` onward) and Kafka `3.x` (all versions)
 - JDK 8 and higher versions
-- all the non-EOLed [ArangoDB versions](https://www.arangodb.com/eol-notice)
+- ArangoDB 3.11.1 and higher versions
 
 ## Installation
 
@@ -254,13 +254,11 @@ See [SSL configuration](configuration.md#ssl) for further options.
 ## Limitations
 
 - The `VST` communication protocol (`connection.protocol=VST`) is currently not working (DE-619)
-- Documents are inserted one by one, bulk inserts will be implemented in a future release (DE-627)
-- In case of transient error, the entire Kafka Connect batch is retried (DE-651)
 - Record values are required to be object-like structures (DE-644)
 - Auto-creation of ArangoDB collection is not supported (DE-653)
 - `ssl.cert.value` does not support multiple certificates (DE-655)
-- Batch inserts are not guaranteed to be executed serially (FRB-300)
-- Batch inserts may succeed for some documents while failing for others (FRB-300)
+- Batch writes are not guaranteed to be executed serially (FRB-300)
+- Batch writes may succeed for some documents while failing for others (FRB-300)
   This has two important consequences:
   - Transient errors might be retried and succeed at a later point
   - Data errors might be asynchronously reported to the DLQ

site/content/3.11/develop/integrations/kafka-connect-arangodb-sink-connector/configuration.md

@@ -173,6 +173,14 @@ to `update`:
 - `true`: objects are merged
 - `false`: existing document fields are overwritten
 
+### batch.size
+
+- type: _int_
+- default: `3_000`
+
+Specifies how many records to attempt to batch together for insertion or deletion
+into the destination collection.
+
 ### insert.timeout.ms
 
 - type: _int_

site/content/3.12/develop/integrations/kafka-connect-arangodb-sink-connector/_index.md

@@ -18,7 +18,7 @@ This connector is compatible with:
 
 - Kafka `2.x` (from version `2.6` onward) and Kafka `3.x` (all versions)
 - JDK 8 and higher versions
-- all the non-EOLed [ArangoDB versions](https://www.arangodb.com/eol-notice)
+- ArangoDB 3.11.1 and higher versions
 
 ## Installation
 
@@ -254,13 +254,11 @@ See [SSL configuration](configuration.md#ssl) for further options.
 ## Limitations
 
 - The `VST` communication protocol (`connection.protocol=VST`) is currently not working (DE-619)
-- Documents are inserted one by one, bulk inserts will be implemented in a future release (DE-627)
-- In case of transient error, the entire Kafka Connect batch is retried (DE-651)
 - Record values are required to be object-like structures (DE-644)
 - Auto-creation of ArangoDB collection is not supported (DE-653)
 - `ssl.cert.value` does not support multiple certificates (DE-655)
-- Batch inserts are not guaranteed to be executed serially (FRB-300)
-- Batch inserts may succeed for some documents while failing for others (FRB-300)
+- Batch writes are not guaranteed to be executed serially (FRB-300)
+- Batch writes may succeed for some documents while failing for others (FRB-300)
   This has two important consequences:
   - Transient errors might be retried and succeed at a later point
   - Data errors might be asynchronously reported to the DLQ

site/content/3.12/develop/integrations/kafka-connect-arangodb-sink-connector/configuration.md

@@ -173,6 +173,14 @@ to `update`:
 - `true`: objects are merged
 - `false`: existing document fields are overwritten
 
+### batch.size
+
+- type: _int_
+- default: `3_000`
+
+Specifies how many records to attempt to batch together for insertion or deletion
+into the destination collection.
+
 ### insert.timeout.ms
 
 - type: _int_