Update to librdkafka 2.3.0 (#1047)

Author: riley-pikus

CONTRIBUTING.md

@@ -215,12 +215,10 @@ Steps to update:
     ```
     Note: This is ran automatically during CI flows but it's good to run it during the version upgrade pull request.
 
-1. Run `npm install` to build with the new version and fix any build errors that occur.
+1. Run `npm install --lockfile-version 2` to build with the new version and fix any build errors that occur.
 
 1. Run unit tests: `npm run test`
 
-1. Run end to end tests: `npm run test:e2e`. This requires running kafka & zookeeper locally.
-
 1. Update the version numbers referenced in the [`README.md`](https://github.com/Blizzard/node-rdkafka/blob/master/README.md) file to the new version.
 
 ## Publishing new npm version

README.md

@@ -17,7 +17,7 @@ I am looking for *your* help to make this project even better! If you're interes
 
 The `node-rdkafka` library is a high-performance NodeJS client for [Apache Kafka](http://kafka.apache.org/) that wraps the native  [librdkafka](https://github.com/edenhill/librdkafka) library.  All the complexity of balancing writes across partitions and managing (possibly ever-changing) brokers should be encapsulated in the library.
 
-__This library currently uses `librdkafka` version `2.2.0`.__
+__This library currently uses `librdkafka` version `2.3.0`.__
 
 ## Reference Docs
 
@@ -60,7 +60,7 @@ Using Alpine Linux? Check out the [docs](https://github.com/Blizzard/node-rdkafk
 
 ### Windows
 
-Windows build **is not** compiled from `librdkafka` source but it is rather linked against the appropriate version of [NuGet librdkafka.redist](https://www.nuget.org/packages/librdkafka.redist/) static binary that gets downloaded from `https://globalcdn.nuget.org/packages/librdkafka.redist.2.2.0.nupkg` during installation. This download link can be changed using the environment variable `NODE_RDKAFKA_NUGET_BASE_URL` that defaults to `https://globalcdn.nuget.org/packages/` when it's no set.
+Windows build **is not** compiled from `librdkafka` source but it is rather linked against the appropriate version of [NuGet librdkafka.redist](https://www.nuget.org/packages/librdkafka.redist/) static binary that gets downloaded from `https://globalcdn.nuget.org/packages/librdkafka.redist.2.3.0.nupkg` during installation. This download link can be changed using the environment variable `NODE_RDKAFKA_NUGET_BASE_URL` that defaults to `https://globalcdn.nuget.org/packages/` when it's no set.
 
 Requirements:
  * [node-gyp for Windows](https://github.com/nodejs/node-gyp#on-windows)
@@ -97,7 +97,7 @@ const Kafka = require('node-rdkafka');
 
 ## Configuration
 
-You can pass many configuration options to `librdkafka`.  A full list can be found in `librdkafka`'s [Configuration.md](https://github.com/edenhill/librdkafka/blob/v2.2.0/CONFIGURATION.md)
+You can pass many configuration options to `librdkafka`.  A full list can be found in `librdkafka`'s [Configuration.md](https://github.com/edenhill/librdkafka/blob/v2.3.0/CONFIGURATION.md)
 
 Configuration keys that have the suffix `_cb` are designated as callbacks. Some
 of these keys are informational and you can choose to opt-in (for example, `dr_cb`). Others are callbacks designed to
@@ -132,7 +132,7 @@ You can also get the version of `librdkafka`
 const Kafka = require('node-rdkafka');
 console.log(Kafka.librdkafkaVersion);
 
-// #=> 2.2.0
+// #=> 2.3.0
 ```
 
 ## Sending Messages
@@ -145,7 +145,7 @@ const producer = new Kafka.Producer({
 });
 ```
 
-A `Producer` requires only `metadata.broker.list` (the Kafka brokers) to be created.  The values in this list are separated by commas.  For other configuration options, see the [Configuration.md](https://github.com/edenhill/librdkafka/blob/v2.2.0/CONFIGURATION.md) file described previously.
+A `Producer` requires only `metadata.broker.list` (the Kafka brokers) to be created.  The values in this list are separated by commas.  For other configuration options, see the [Configuration.md](https://github.com/edenhill/librdkafka/blob/v2.3.0/CONFIGURATION.md) file described previously.
 
 The following example illustrates a list with several `librdkafka` options set.
 

config.d.ts

@@ -1,4 +1,4 @@
-// ====== Generated from librdkafka 2.2.0 file CONFIGURATION.md ======
+// ====== Generated from librdkafka 2.3.0 file CONFIGURATION.md ======
 // Code that generated this is a derivative work of the code from Nam Nguyen
 // https://gist.github.com/ntgn81/066c2c8ec5b4238f85d1e9168a04e3fb
 
@@ -77,9 +77,9 @@ export interface GlobalConfig {
     "metadata.max.age.ms"?: number;
 
     /**
-     * When a topic loses its leader a new metadata request will be enqueued with this initial interval, exponentially increasing until the topic metadata has been refreshed. This is used to recover quickly from transitioning leader brokers.
+     * When a topic loses its leader a new metadata request will be enqueued immediately and then with this initial interval, exponentially increasing upto `retry.backoff.max.ms`, until the topic metadata has been refreshed. If not set explicitly, it will be defaulted to `retry.backoff.ms`. This is used to recover quickly from transitioning leader brokers.
      *
-     * @default 250
+     * @default 100
      */
     "topic.metadata.refresh.fast.interval.ms"?: number;
 
@@ -704,12 +704,19 @@ export interface ProducerGlobalConfig extends GlobalConfig {
     "retries"?: number;
 
     /**
-     * The backoff time in milliseconds before retrying a protocol request.
+     * The backoff time in milliseconds before retrying a protocol request, this is the first backoff time, and will be backed off exponentially until number of retries is exhausted, and it's capped by retry.backoff.max.ms.
      *
      * @default 100
      */
     "retry.backoff.ms"?: number;
 
+    /**
+     * The max backoff time in milliseconds before retrying a protocol request, this is the atmost backoff allowed for exponentially backed off requests.
+     *
+     * @default 1000
+     */
+    "retry.backoff.max.ms"?: number;
+
     /**
      * The threshold of outstanding not yet transmitted broker requests needed to backpressure the producer's message accumulator. If the number of not yet transmitted requests equals or exceeds this number, produce request creation that would have otherwise been triggered (for example, in accordance with linger.ms) will be delayed. A lower number yields larger and more effective batches. A higher value can improve latency when using compression on slow machines.
      *

deps/librdkafka

@@ -1,4 +1,4 @@
-// ====== Generated from librdkafka 2.2.0 file src-cpp/rdkafkacpp.h ======
+// ====== Generated from librdkafka 2.3.0 file src-cpp/rdkafkacpp.h ======
 export const CODES: { ERRORS: {
   /* Internal errors to rdkafka: */
   /** Begin internal error codes (**-200**) */
@@ -128,10 +128,8 @@ export const CODES: { ERRORS: {
   ERR__AUTO_OFFSET_RESET: number,
   /** Partition log truncation detected (**-139**) */
   ERR__LOG_TRUNCATION: number,
-
   /** End internal error codes (**-100**) */
   ERR__END: number,
-
   /* Kafka broker errors: */
   /** Unknown broker error (**-1**) */
   ERR_UNKNOWN: number,

errors.d.ts

@@ -27,7 +27,7 @@ LibrdKafkaError.wrap = errorWrap;
  * @enum {number}
  * @constant
  */
-// ====== Generated from librdkafka 2.2.0 file src-cpp/rdkafkacpp.h ======
+// ====== Generated from librdkafka 2.3.0 file src-cpp/rdkafkacpp.h ======
 LibrdKafkaError.codes = {
 
   /* Internal errors to rdkafka: */
@@ -158,10 +158,8 @@ LibrdKafkaError.codes = {
   ERR__AUTO_OFFSET_RESET: -140,
   /** Partition log truncation detected */
   ERR__LOG_TRUNCATION: -139,
-
   /** End internal error codes */
   ERR__END: -100,
-
   /* Kafka broker errors: */
   /** Unknown broker error */
   ERR_UNKNOWN: -1,