Update librdkafka from 1.5.3 -> 1.6.1 (#879)

Author: fabianschmitthenner

README.md

@@ -16,7 +16,7 @@ I am looking for *your* help to make this project even better! If you're interes
 
 The `node-rdkafka` library is a high-performance NodeJS client for [Apache Kafka](http://kafka.apache.org/) that wraps the native  [librdkafka](https://github.com/edenhill/librdkafka) library.  All the complexity of balancing writes across partitions and managing (possibly ever-changing) brokers should be encapsulated in the library.
 
-__This library currently uses `librdkafka` version `1.5.3`.__
+__This library currently uses `librdkafka` version `1.6.1`.__
 
 ## Reference Docs
 
@@ -59,7 +59,7 @@ Using Alpine Linux? Check out the [docs](https://github.com/Blizzard/node-rdkafk
 
 ### Windows
 
-Windows build **is not** compiled from `librdkafka` source but it is rather linked against the appropriate version of [NuGet librdkafka.redist](https://www.nuget.org/packages/librdkafka.redist/) static binary that gets downloaded from `https://globalcdn.nuget.org/packages/librdkafka.redist.1.5.3.nupkg` during installation. This download link can be changed using the environment variable `NODE_RDKAFKA_NUGET_BASE_URL` that defaults to `https://globalcdn.nuget.org/packages/` when it's no set.
+Windows build **is not** compiled from `librdkafka` source but it is rather linked against the appropriate version of [NuGet librdkafka.redist](https://www.nuget.org/packages/librdkafka.redist/) static binary that gets downloaded from `https://globalcdn.nuget.org/packages/librdkafka.redist.1.6.1.nupkg` during installation. This download link can be changed using the environment variable `NODE_RDKAFKA_NUGET_BASE_URL` that defaults to `https://globalcdn.nuget.org/packages/` when it's no set.
 
 Requirements:
  * [node-gyp for Windows](https://github.com/nodejs/node-gyp#on-windows)  (the easies way to get it: `npm install --global --production windows-build-tools`, if your node version is 6.x or below, pleasse use `npm install --global --production [email protected]`)
@@ -96,7 +96,7 @@ var Kafka = require('node-rdkafka');
 
 ## Configuration
 
-You can pass many configuration options to `librdkafka`.  A full list can be found in `librdkafka`'s [Configuration.md](https://github.com/edenhill/librdkafka/blob/v1.5.3/CONFIGURATION.md)
+You can pass many configuration options to `librdkafka`.  A full list can be found in `librdkafka`'s [Configuration.md](https://github.com/edenhill/librdkafka/blob/v1.6.1/CONFIGURATION.md)
 
 Configuration keys that have the suffix `_cb` are designated as callbacks. Some
 of these keys are informational and you can choose to opt-in (for example, `dr_cb`). Others are callbacks designed to
@@ -131,7 +131,7 @@ You can also get the version of `librdkafka`
 const Kafka = require('node-rdkafka');
 console.log(Kafka.librdkafkaVersion);
 
-// #=> 1.5.3
+// #=> 1.6.1
 ```
 
 ## Sending Messages
@@ -144,7 +144,7 @@ var producer = new Kafka.Producer({
 });
 ```
 
-A `Producer` requires only `metadata.broker.list` (the Kafka brokers) to be created.  The values in this list are separated by commas.  For other configuration options, see the [Configuration.md](https://github.com/edenhill/librdkafka/blob/v1.5.3/CONFIGURATION.md) file described previously.
+A `Producer` requires only `metadata.broker.list` (the Kafka brokers) to be created.  The values in this list are separated by commas.  For other configuration options, see the [Configuration.md](https://github.com/edenhill/librdkafka/blob/v1.6.1/CONFIGURATION.md) file described previously.
 
 The following example illustrates a list with several `librdkafka` options set.
 

config.d.ts

@@ -1,4 +1,4 @@
-// ====== Generated from librdkafka 1.5.3 file CONFIGURATION.md ======
+// ====== Generated from librdkafka 1.6.1 file CONFIGURATION.md ======
 // Code that generated this is a derivative work of the code from Nam Nguyen
 // https://gist.github.com/ntgn81/066c2c8ec5b4238f85d1e9168a04e3fb
 
@@ -261,7 +261,7 @@ export interface GlobalConfig {
     "log.thread.name"?: boolean;
 
     /**
-     * If enabled librdkafka will initialize the POSIX PRNG with srand(current_time.milliseconds) on the first invocation of rd_kafka_new(). If disabled the application must call srand() prior to calling rd_kafka_new().
+     * If enabled librdkafka will initialize the PRNG with srand(current_time.milliseconds) on the first invocation of rd_kafka_new() (required only if rand_r() is not available on your platform). If disabled the application must call srand() prior to calling rd_kafka_new().
      *
      * @default true
      */
@@ -402,7 +402,9 @@ export interface GlobalConfig {
     "ssl_certificate"?: any;
 
     /**
-     * File or directory path to CA certificate(s) for verifying the broker's key. Defaults: On Windows the system's CA certificates are automatically looked up in the Windows Root certificate store. On Mac OSX it is recommended to install openssl using Homebrew, to provide CA certificates. On Linux install the distribution's ca-certificates package. If OpenSSL is statically linked or `ssl.ca.location` is set to `probe` a list of standard paths will be probed and the first one found will be used as the default CA certificate location path. If OpenSSL is dynamically linked the OpenSSL library's default path will be used (see `OPENSSLDIR` in `openssl version -a`).
+     * File or directory path to CA certificate(s) for verifying the broker's key. Defaults: On Windows the system's CA certificates are automatically looked up in the Windows Root certificate store. On Mac OSX this configuration defaults to `probe`. It is recommended to install openssl using Homebrew, to provide CA certificates. On Linux install the distribution's ca-certificates package. If OpenSSL is statically linked or `ssl.ca.location` is set to `probe` a list of standard paths will be probed and the first one found will be used as the default CA certificate location path. If OpenSSL is dynamically linked the OpenSSL library's default path will be used (see `OPENSSLDIR` in `openssl version -a`).
+     *
+     * @default probe
      */
     "ssl.ca.location"?: string;
 
@@ -411,6 +413,13 @@ export interface GlobalConfig {
      */
     "ssl_ca"?: any;
 
+    /**
+     * Comma-separated list of Windows Certificate stores to load CA certificates from. Certificates will be loaded in the same order as stores are specified. If no certificates can be loaded from any of the specified stores an error is logged and the OpenSSL library's default CA location is used instead. Store names are typically one or more of: MY, Root, Trust, CA.
+     *
+     * @default Root
+     */
+    "ssl.ca.certificate.stores"?: string;
+
     /**
      * Path to CRL for verifying broker's certificate validity.
      */
@@ -669,6 +678,13 @@ export interface ProducerGlobalConfig extends GlobalConfig {
      * Delivery report callback (set with rd_kafka_conf_set_dr_msg_cb())
      */
     "dr_msg_cb"?: boolean;
+
+    /**
+     * Delay in milliseconds to wait to assign new sticky partitions for each topic. By default, set to double the time of linger.ms. To disable sticky behavior, set to 0. This behavior affects messages with the key NULL in all cases, and messages with key lengths of zero when the consistent_random partitioner is in use. These messages would otherwise be assigned randomly. A higher value allows for more effective batching of these messages.
+     *
+     * @default 10
+     */
+    "sticky.partitioning.linger.ms"?: number;
 }
 
 export interface ConsumerGlobalConfig extends GlobalConfig {
@@ -683,7 +699,7 @@ export interface ConsumerGlobalConfig extends GlobalConfig {
     "group.instance.id"?: string;
 
     /**
-     * Name of partition assignment strategy to use when elected group leader assigns partitions to group members.
+     * The name of one or more partition assignment strategies. The elected group leader will use a strategy supported by all members of the group to assign partitions to group members. If there is more than one eligible strategy, preference is determined by the order of this list (strategies earlier in the list have higher priority). Cooperative and non-cooperative (eager) strategies must not be mixed. Available strategies: range, roundrobin, cooperative-sticky.
      *
      * @default range,roundrobin
      */
@@ -704,7 +720,7 @@ export interface ConsumerGlobalConfig extends GlobalConfig {
     "heartbeat.interval.ms"?: number;
 
     /**
-     * Group protocol type
+     * Group protocol type. NOTE: Currently, the only supported group protocol type is `consumer`.
      *
      * @default consumer
      */
@@ -971,7 +987,7 @@ export interface ConsumerTopicConfig extends TopicConfig {
     "auto.commit.interval.ms"?: number;
 
     /**
-     * Action to take when there is no initial offset in offset store or the desired offset is out of range: 'smallest','earliest' - automatically reset the offset to the smallest offset, 'largest','latest' - automatically reset the offset to the largest offset, 'error' - trigger an error which is retrieved by consuming messages and checking 'message->err'.
+     * Action to take when there is no initial offset in offset store or the desired offset is out of range: 'smallest','earliest' - automatically reset the offset to the smallest offset, 'largest','latest' - automatically reset the offset to the largest offset, 'error' - trigger an error (ERR__AUTO_OFFSET_RESET) which is retrieved by consuming messages and checking 'message->err'.
      *
      * @default largest
      */

deps/librdkafka

@@ -1,4 +1,4 @@
-// ====== Generated from librdkafka 1.5.3 file src-cpp/rdkafkacpp.h ======
+// ====== Generated from librdkafka 1.6.1 file src-cpp/rdkafkacpp.h ======
 export const CODES: { ERRORS: {
   /* Internal errors to rdkafka: */
   /** Begin internal error codes (**-200**) */
@@ -120,6 +120,12 @@ export const CODES: { ERRORS: {
   ERR__FENCED: number,
   /** Application generated error (**-143**) */
   ERR__APPLICATION: number,
+  /** Assignment lost (**-142**) */
+  ERR__ASSIGNMENT_LOST: number,
+  /** No operation performed (**-141**) */
+  ERR__NOOP: number,
+  /** No offset to automatically reset to (**-140**) */
+  ERR__AUTO_OFFSET_RESET: number,
   /** End internal error codes (**-100**) */
   ERR__END: number,
   /* Kafka broker errors: */
@@ -309,10 +315,31 @@ export const CODES: { ERRORS: {
   ERR_ELECTION_NOT_NEEDED: number,
   /** No partition reassignment is in progress (**85**) */
   ERR_NO_REASSIGNMENT_IN_PROGRESS: number,
-  /** Deleting offsets of a topic while the consumer group is subscribed to it (**86**) */
+  /** Deleting offsets of a topic while the consumer group is
+  *  subscribed to it (**86**) */
   ERR_GROUP_SUBSCRIBED_TO_TOPIC: number,
   /** Broker failed to validate record (**87**) */
   ERR_INVALID_RECORD: number,
   /** There are unstable offsets that need to be cleared (**88**) */
   ERR_UNSTABLE_OFFSET_COMMIT: number,
+  /** Throttling quota has been exceeded (**89**) */
+  ERR_THROTTLING_QUOTA_EXCEEDED: number,
+  /** There is a newer producer with the same transactionalId
+  *  which fences the current one (**90**) */
+  ERR_PRODUCER_FENCED: number,
+  /** Request illegally referred to resource that does not exist (**91**) */
+  ERR_RESOURCE_NOT_FOUND: number,
+  /** Request illegally referred to the same resource twice (**92**) */
+  ERR_DUPLICATE_RESOURCE: number,
+  /** Requested credential would not meet criteria for acceptability (**93**) */
+  ERR_UNACCEPTABLE_CREDENTIAL: number,
+  /** Indicates that the either the sender or recipient of a
+  *  voter-only request is not one of the expected voters (**94**) */
+  ERR_INCONSISTENT_VOTER_SET: number,
+  /** Invalid update version (**95**) */
+  ERR_INVALID_UPDATE_VERSION: number,
+  /** Unable to update finalized features due to server error (**96**) */
+  ERR_FEATURE_UPDATE_FAILED: number,
+  /** Request principal deserialization failed during forwarding (**97**) */
+  ERR_PRINCIPAL_DESERIALIZATION_FAILURE: number,
 }}

errors.d.ts

@@ -27,7 +27,7 @@ LibrdKafkaError.wrap = errorWrap;
  * @enum {number}
  * @constant
  */
-// ====== Generated from librdkafka 1.5.3 file src-cpp/rdkafkacpp.h ======
+// ====== Generated from librdkafka 1.6.1 file src-cpp/rdkafkacpp.h ======
 LibrdKafkaError.codes = {
 
   /* Internal errors to rdkafka: */
@@ -150,6 +150,12 @@ LibrdKafkaError.codes = {
   ERR__FENCED: -144,
   /** Application generated error */
   ERR__APPLICATION: -143,
+  /** Assignment lost */
+  ERR__ASSIGNMENT_LOST: -142,
+  /** No operation performed */
+  ERR__NOOP: -141,
+  /** No offset to automatically reset to */
+  ERR__AUTO_OFFSET_RESET: -140,
   /** End internal error codes */
   ERR__END: -100,
   /* Kafka broker errors: */
@@ -339,12 +345,33 @@ LibrdKafkaError.codes = {
   ERR_ELECTION_NOT_NEEDED: 84,
   /** No partition reassignment is in progress */
   ERR_NO_REASSIGNMENT_IN_PROGRESS: 85,
-  /** Deleting offsets of a topic while the consumer group is subscribed to it */
+  /** Deleting offsets of a topic while the consumer group is
+  *  subscribed to it */
   ERR_GROUP_SUBSCRIBED_TO_TOPIC: 86,
   /** Broker failed to validate record */
   ERR_INVALID_RECORD: 87,
   /** There are unstable offsets that need to be cleared */
-  ERR_UNSTABLE_OFFSET_COMMIT: 88
+  ERR_UNSTABLE_OFFSET_COMMIT: 88,
+  /** Throttling quota has been exceeded */
+  ERR_THROTTLING_QUOTA_EXCEEDED: 89,
+  /** There is a newer producer with the same transactionalId
+  *  which fences the current one */
+  ERR_PRODUCER_FENCED: 90,
+  /** Request illegally referred to resource that does not exist */
+  ERR_RESOURCE_NOT_FOUND: 91,
+  /** Request illegally referred to the same resource twice */
+  ERR_DUPLICATE_RESOURCE: 92,
+  /** Requested credential would not meet criteria for acceptability */
+  ERR_UNACCEPTABLE_CREDENTIAL: 93,
+  /** Indicates that the either the sender or recipient of a
+  *  voter-only request is not one of the expected voters */
+  ERR_INCONSISTENT_VOTER_SET: 94,
+  /** Invalid update version */
+  ERR_INVALID_UPDATE_VERSION: 95,
+  /** Unable to update finalized features due to server error */
+  ERR_FEATURE_UPDATE_FAILED: 96,
+  /** Request principal deserialization failed during forwarding */
+  ERR_PRINCIPAL_DESERIALIZATION_FAILURE: 97
 };
 
 /**

lib/error.js

@@ -2,7 +2,7 @@
   "name": "node-rdkafka",
   "version": "v2.10.1",
   "description": "Node.js bindings for librdkafka",
-  "librdkafka": "1.5.3",
+  "librdkafka": "1.6.1",
   "main": "lib/index.js",
   "scripts": {
     "configure": "node-gyp configure",