Update librdkafka from 1.6.1 -> 1.7.0 (#915)

Author: GaryWilber

CONTRIBUTING.md

@@ -27,6 +27,8 @@ so if you feel something is missing feel free to send a pull request.
 [Debugging](#debugging)
   * [Debugging C++](#debugging-c)
 
+[Updating librdkafka version](#updating-librdkafka-version)
+
 ## What should I know before I get started?
 
 ### Contributor Agreement
@@ -190,3 +192,31 @@ gdb node
 ```
 
 You can add breakpoints and so on after that.
+
+## Updating librdkafka version
+
+The librdkafka should be periodically updated to the latest release in https://github.com/edenhill/librdkafka/releases
+
+Steps to update:
+1. Update the `librdkafka` property in [`package.json`](https://github.com/Blizzard/node-rdkafka/blob/master/package.json) to the desired version.
+
+1. Update the librdkafka git submodule to that versions release commit (example below)
+
+    ```bash
+    cd deps/librdkafka
+    git checkout 77a013b7a2611f7bdc091afa1e56b1a46d1c52f5 # for version 1.70
+    ```
+
+1. Update [`config.d.ts`](https://github.com/Blizzard/node-rdkafka/blob/master/config.d.ts) and [`errors.d.ts`](https://github.com/Blizzard/node-rdkafka/blob/master/errors.d.ts) TypeScript definitions by running:
+    ```bash
+    node ci/librdkafka-defs-generator.js
+    ```
+    Note: This is ran automatically during CI flows but it's good to run it during the version upgrade pull request.
+
+1. Run `npm install` to build with the new version and fix any build errors that occur.
+
+1. Run unit tests: `npm run test`
+
+1. Run end to end tests: `npm run test:e2e`. This requires running kafka & zookeeper locally.
+
+1. Update the version numbers referenced in the [`README.md`](https://github.com/Blizzard/node-rdkafka/blob/master/README.md) file to the new version.

README.md

@@ -5,7 +5,7 @@ Copyright (c) 2016 Blizzard Entertainment.
 
 [https://github.com/blizzard/node-rdkafka](https://github.com/blizzard/node-rdkafka)
 
-[![Build Status](https://travis-ci.com/Blizzard/node-rdkafka.svg?branch=master)](https://travis-ci.com/Blizzard/node-rdkafka)
+[![Build Status](https://app.travis-ci.com/Blizzard/node-rdkafka.svg?branch=master)](https://app.travis-ci.com/github/Blizzard/node-rdkafka)
 [![npm version](https://badge.fury.io/js/node-rdkafka.svg)](https://badge.fury.io/js/node-rdkafka)
 
 # Looking for Collaborators!
@@ -16,7 +16,7 @@ I am looking for *your* help to make this project even better! If you're interes
 
 The `node-rdkafka` library is a high-performance NodeJS client for [Apache Kafka](http://kafka.apache.org/) that wraps the native  [librdkafka](https://github.com/edenhill/librdkafka) library.  All the complexity of balancing writes across partitions and managing (possibly ever-changing) brokers should be encapsulated in the library.
 
-__This library currently uses `librdkafka` version `1.6.1`.__
+__This library currently uses `librdkafka` version `1.7.0`.__
 
 ## Reference Docs
 
@@ -59,7 +59,7 @@ Using Alpine Linux? Check out the [docs](https://github.com/Blizzard/node-rdkafk
 
 ### Windows
 
-Windows build **is not** compiled from `librdkafka` source but it is rather linked against the appropriate version of [NuGet librdkafka.redist](https://www.nuget.org/packages/librdkafka.redist/) static binary that gets downloaded from `https://globalcdn.nuget.org/packages/librdkafka.redist.1.6.1.nupkg` during installation. This download link can be changed using the environment variable `NODE_RDKAFKA_NUGET_BASE_URL` that defaults to `https://globalcdn.nuget.org/packages/` when it's no set.
+Windows build **is not** compiled from `librdkafka` source but it is rather linked against the appropriate version of [NuGet librdkafka.redist](https://www.nuget.org/packages/librdkafka.redist/) static binary that gets downloaded from `https://globalcdn.nuget.org/packages/librdkafka.redist.1.7.0.nupkg` during installation. This download link can be changed using the environment variable `NODE_RDKAFKA_NUGET_BASE_URL` that defaults to `https://globalcdn.nuget.org/packages/` when it's no set.
 
 Requirements:
  * [node-gyp for Windows](https://github.com/nodejs/node-gyp#on-windows)  (the easies way to get it: `npm install --global --production windows-build-tools`, if your node version is 6.x or below, pleasse use `npm install --global --production [email protected]`)
@@ -96,7 +96,7 @@ var Kafka = require('node-rdkafka');
 
 ## Configuration
 
-You can pass many configuration options to `librdkafka`.  A full list can be found in `librdkafka`'s [Configuration.md](https://github.com/edenhill/librdkafka/blob/v1.6.1/CONFIGURATION.md)
+You can pass many configuration options to `librdkafka`.  A full list can be found in `librdkafka`'s [Configuration.md](https://github.com/edenhill/librdkafka/blob/v1.7.0/CONFIGURATION.md)
 
 Configuration keys that have the suffix `_cb` are designated as callbacks. Some
 of these keys are informational and you can choose to opt-in (for example, `dr_cb`). Others are callbacks designed to
@@ -131,7 +131,7 @@ You can also get the version of `librdkafka`
 const Kafka = require('node-rdkafka');
 console.log(Kafka.librdkafkaVersion);
 
-// #=> 1.6.1
+// #=> 1.7.0
 ```
 
 ## Sending Messages
@@ -144,7 +144,7 @@ var producer = new Kafka.Producer({
 });
 ```
 
-A `Producer` requires only `metadata.broker.list` (the Kafka brokers) to be created.  The values in this list are separated by commas.  For other configuration options, see the [Configuration.md](https://github.com/edenhill/librdkafka/blob/v1.6.1/CONFIGURATION.md) file described previously.
+A `Producer` requires only `metadata.broker.list` (the Kafka brokers) to be created.  The values in this list are separated by commas.  For other configuration options, see the [Configuration.md](https://github.com/edenhill/librdkafka/blob/v1.7.0/CONFIGURATION.md) file described previously.
 
 The following example illustrates a list with several `librdkafka` options set.
 

ci/librdkafka-defs-generator.js

@@ -172,7 +172,7 @@ function updateErrorDefinitions(file) {
   // validate body
   const emptyCheck = body
     .replace(/((  \/\*)|(   ?\*)).*/g, '')
-    .replace(/  ERR_\w+: -?\d+,?\n/g, '')
+    .replace(/  ERR_\w+: -?\d+,?\r?\n/g, '')
     .trim()
   if (emptyCheck !== '') {
     throw new Error(`Fail to parse ${file}. It contains these extra details:\n${emptyCheck}`);
@@ -184,7 +184,7 @@ function updateErrorDefinitions(file) {
     .replace(/(\/\/.*\n)?LibrdKafkaError.codes = {[^}]+/g, `${getHeader(file)}\nLibrdKafkaError.codes = {\n${body}`)
 
   fs.writeFileSync(error_js_file, error_js);
-  fs.writeFileSync(path.resolve(__dirname, '../errors.d.ts'), `${getHeader(file)}\nexport const CODES: { ERRORS: {${body.replace(/[ \.]*(\*\/\n  \w+: )(-?\d+),?/g, ' (**$2**) $1number,')}}}`)
+  fs.writeFileSync(path.resolve(__dirname, '../errors.d.ts'), `${getHeader(file)}\nexport const CODES: { ERRORS: {${body.replace(/[ \.]*(\*\/\r?\n  \w+: )(-?\d+),?/g, ' (**$2**) $1number,')}}}`)
 }
 
 (async function updateTypeDefs() {

config.d.ts

@@ -1,4 +1,4 @@
-// ====== Generated from librdkafka 1.6.1 file CONFIGURATION.md ======
+// ====== Generated from librdkafka 1.7.0 file CONFIGURATION.md ======
 // Code that generated this is a derivative work of the code from Nam Nguyen
 // https://gist.github.com/ntgn81/066c2c8ec5b4238f85d1e9168a04e3fb
 
@@ -62,13 +62,6 @@ export interface GlobalConfig {
      */
     "max.in.flight"?: number;
 
-    /**
-     * Non-topic request timeout in milliseconds. This is for metadata requests, etc.
-     *
-     * @default 60000
-     */
-    "metadata.request.timeout.ms"?: number;
-
     /**
      * Period of time in milliseconds at which topic and broker metadata is refreshed in order to proactively discover any new brokers, topics, partitions or partition leader changes. Use -1 to disable the intervalled refresh (not recommended). If there are no locally referenced topics (no topic objects created, no messages produced, no subscription or no assignment) then only the broker list will be refreshed every interval but no more often than every 10s.
      *
@@ -184,6 +177,13 @@ export interface GlobalConfig {
      */
     "broker.address.family"?: 'any' | 'v4' | 'v6';
 
+    /**
+     * Close broker connections after the specified time of inactivity. Disable with 0. If this property is left at its default value some heuristics are performed to determine a suitable default value, this is currently limited to identifying brokers on Azure (see librdkafka issue #3109 for more info).
+     *
+     * @default 0
+     */
+    "connections.max.idle.ms"?: number;
+
     /**
      * **DEPRECATED** No longer used. See `reconnect.backoff.ms` and `reconnect.backoff.max.ms`.
      *
@@ -403,8 +403,6 @@ export interface GlobalConfig {
 
     /**
      * File or directory path to CA certificate(s) for verifying the broker's key. Defaults: On Windows the system's CA certificates are automatically looked up in the Windows Root certificate store. On Mac OSX this configuration defaults to `probe`. It is recommended to install openssl using Homebrew, to provide CA certificates. On Linux install the distribution's ca-certificates package. If OpenSSL is statically linked or `ssl.ca.location` is set to `probe` a list of standard paths will be probed and the first one found will be used as the default CA certificate location path. If OpenSSL is dynamically linked the OpenSSL library's default path will be used (see `OPENSSLDIR` in `openssl version -a`).
-     *
-     * @default probe
      */
     "ssl.ca.location"?: string;
 
@@ -435,6 +433,23 @@ export interface GlobalConfig {
      */
     "ssl.keystore.password"?: string;
 
+    /**
+     * Path to OpenSSL engine library. OpenSSL >= 1.1.0 required.
+     */
+    "ssl.engine.location"?: string;
+
+    /**
+     * OpenSSL engine id is the name used for loading engine.
+     *
+     * @default dynamic
+     */
+    "ssl.engine.id"?: string;
+
+    /**
+     * OpenSSL engine callback data (set with rd_kafka_conf_set_engine_callback_data()).
+     */
+    "ssl_engine_callback_data"?: any;
+
     /**
      * Enable OpenSSL's builtin broker (server) certificate verification. This verification can be extended by the application by implementing a certificate_verify_cb.
      *
@@ -708,7 +723,7 @@ export interface ConsumerGlobalConfig extends GlobalConfig {
     /**
      * Client group session and failure detection timeout. The consumer sends periodic heartbeats (heartbeat.interval.ms) to indicate its liveness to the broker. If no hearts are received by the broker for a group member within the session timeout, the broker will remove the consumer from the group and trigger a rebalance. The allowed range is configured with the **broker** configuration properties `group.min.session.timeout.ms` and `group.max.session.timeout.ms`. Also see `max.poll.interval.ms`.
      *
-     * @default 10000
+     * @default 45000
      */
     "session.timeout.ms"?: number;
 
@@ -966,14 +981,14 @@ export interface ProducerTopicConfig extends TopicConfig {
 
 export interface ConsumerTopicConfig extends TopicConfig {
     /**
-     * **DEPRECATED** [**LEGACY PROPERTY:** This property is used by the simple legacy consumer only. When using the high-level KafkaConsumer, the global `enable.auto.commit` property must be used instead]. If true, periodically commit offset of the last message handed to the application. This committed offset will be used when the process restarts to pick up where it left off. If false, the application will have to call `rd_kafka_offset_store()` to store an offset (optional). **NOTE:** There is currently no zookeeper integration, offsets will be written to broker or local file according to offset.store.method.
+     * **DEPRECATED** [**LEGACY PROPERTY:** This property is used by the simple legacy consumer only. When using the high-level KafkaConsumer, the global `enable.auto.commit` property must be used instead]. If true, periodically commit offset of the last message handed to the application. This committed offset will be used when the process restarts to pick up where it left off. If false, the application will have to call `rd_kafka_offset_store()` to store an offset (optional). Offsets will be written to broker or local file according to offset.store.method.
      *
      * @default true
      */
     "auto.commit.enable"?: boolean;
 
     /**
-     * **DEPRECATED** Alias for `auto.commit.enable`: [**LEGACY PROPERTY:** This property is used by the simple legacy consumer only. When using the high-level KafkaConsumer, the global `enable.auto.commit` property must be used instead]. If true, periodically commit offset of the last message handed to the application. This committed offset will be used when the process restarts to pick up where it left off. If false, the application will have to call `rd_kafka_offset_store()` to store an offset (optional). **NOTE:** There is currently no zookeeper integration, offsets will be written to broker or local file according to offset.store.method.
+     * **DEPRECATED** Alias for `auto.commit.enable`: [**LEGACY PROPERTY:** This property is used by the simple legacy consumer only. When using the high-level KafkaConsumer, the global `enable.auto.commit` property must be used instead]. If true, periodically commit offset of the last message handed to the application. This committed offset will be used when the process restarts to pick up where it left off. If false, the application will have to call `rd_kafka_offset_store()` to store an offset (optional). Offsets will be written to broker or local file according to offset.store.method.
      *
      * @default true
      */

deps/librdkafka

@@ -1,4 +1,4 @@
-// ====== Generated from librdkafka 1.6.1 file src-cpp/rdkafkacpp.h ======
+// ====== Generated from librdkafka 1.7.0 file src-cpp/rdkafkacpp.h ======
 export const CODES: { ERRORS: {
   /* Internal errors to rdkafka: */
   /** Begin internal error codes (**-200**) */
@@ -126,8 +126,10 @@ export const CODES: { ERRORS: {
   ERR__NOOP: number,
   /** No offset to automatically reset to (**-140**) */
   ERR__AUTO_OFFSET_RESET: number,
+
   /** End internal error codes (**-100**) */
   ERR__END: number,
+
   /* Kafka broker errors: */
   /** Unknown broker error (**-1**) */
   ERR_UNKNOWN: number,

errors.d.ts

@@ -27,7 +27,7 @@ LibrdKafkaError.wrap = errorWrap;
  * @enum {number}
  * @constant
  */
-// ====== Generated from librdkafka 1.6.1 file src-cpp/rdkafkacpp.h ======
+// ====== Generated from librdkafka 1.7.0 file src-cpp/rdkafkacpp.h ======
 LibrdKafkaError.codes = {
 
   /* Internal errors to rdkafka: */
@@ -156,8 +156,10 @@ LibrdKafkaError.codes = {
   ERR__NOOP: -141,
   /** No offset to automatically reset to */
   ERR__AUTO_OFFSET_RESET: -140,
+
   /** End internal error codes */
   ERR__END: -100,
+
   /* Kafka broker errors: */
   /** Unknown broker error */
   ERR_UNKNOWN: -1,

lib/error.js

@@ -2,12 +2,13 @@
   "name": "node-rdkafka",
   "version": "v2.11.0",
   "description": "Node.js bindings for librdkafka",
-  "librdkafka": "1.6.1",
+  "librdkafka": "1.7.0",
   "main": "lib/index.js",
   "scripts": {
     "configure": "node-gyp configure",
     "build": "node-gyp build",
     "test": "make test",
+    "test:e2e": "make e2e",
     "install": "node-gyp rebuild",
     "prepack": "node ./ci/prepublish.js"
   },