zero-downtime restart: add initial document

SUMMARY.md

@@ -57,6 +57,7 @@
   * [Linux Capability](deployment/linux-capability.md)
   * [Command Line Option](deployment/command-line-option.md)
   * [Source Only Mode](deployment/source-only-mode.md)
+  * [Zero-downtime restart](deployment/zero-downtime-restart.md)
 * [Container Deployment](container-deployment/README.md)
   * [Docker Image](container-deployment/install-by-docker.md)
   * [Docker Logging Driver](container-deployment/docker-logging-driver.md)

deployment/rpc.md

@@ -29,10 +29,16 @@ As evident from the output above, each endpoint returns a JSON object as its res
 | :--- | :---: | :---: |
 | `/api/processes.interruptWorkers` | [SIGINT](signals.md#sigint-or-sigterm) | Stops the daemon. |
 | `/api/processes.killWorkers` | [SIGTERM](signals.md#sigint-or-sigterm) | Stops the daemon. |
+| `/api/processes.zeroDowntimeRestart` | [SIGUSR2](signals.md#sigusr2) | Restarts Fluentd with zero-downtime. |
 | `/api/processes.flushBuffersAndKillWorkers` | [SIGUSR1](signals.md#sigusr1) and [SIGTERM](signals.md#sigint-or-sigterm) | Flushes buffer and stops the daemon. |
 | `/api/plugins.flushBuffers` | [SIGUSR1](signals.md#sigusr1) | Flushes the buffered messages. |
-| `/api/config.gracefulReload` | [SIGUSR2](signals.md#sigusr2) | Reloads configuration. |
 | `/api/config.reload` | [SIGHUP](signals.md#sighup) | Reloads configuration. |
+| `/api/config.gracefulReload` | --- | Reloads configuration. |
+
+Appendix:
+
+* `/api/processes.zeroDowntimeRestart`: This is supported since v1.18.0 on non-Windows.
+* `/api/config.gracefulReload`: This is the replacement of `SIGUSR2` before v1.18.0. Please use `/api/processes.zeroDowntimeRestart` or `/api/config.reload` unless there is a special reason. See [SIGUSR2](signals.md#sigusr2) for details.
 
 If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License.
 

deployment/signals.md

@@ -18,15 +18,56 @@ Forces the buffered messages to be flushed and reopens Fluentd's log. Fluentd wi
 
 ### SIGUSR2
 
+Since v1.18, it has two features: Zero-downtime restart and Graceful reload.
+
+Non-Windows:
+
+| process    | feature                                    | version      |
+| :---       | :---                                       | :---         |
+| Supervisor | Zero-downtime restart                      | v1.18.0 ~    |
+| Supervisor | Graceful reload (forwarded to all workers) | v1.9 ~ v1.17 |
+| Worker     | Graceful reload                            | v1.9 ~       |
+
+Windows:
+
+| process    | feature                                    | version |
+| :---       | :---                                       | :---    |
+| Supervisor | Graceful reload (forwarded to all workers) | v1.9 ~  |
+| Worker     | Graceful reload                            | v1.9 ~  |
+
+#### Zero-downtime restart
+
+This feature allows a complete restart of Fluentd without bringing down some input plugins, such as `in_udp` or `in_tcp`.
+
+See [Zero-downtime restart](zero-downtime-restart.md) for details.
+
+**Comparison with SIGHUP**
+
+`SIGHUP` gracefully restarting the worker process to reload.
+
+This method does not cause socket downtime, so if there is no need to restart the supervisor, `SIGHUP` is a lighter zero-downtime restart method.
+
+**Comparison with Graceful reload**
+
+You can still use Graceful reload feature by sending `SIGUSR2` directly to the worker processor or using [RPC](rpc.md) even after v1.18.0.
+
+This allows you to reload without restarting the process, but there are some limitations.
+Please use zero-downtime restart or `SIGHUP` unless there is a special reason.
+
+#### Graceful reload
+
 Reloads the configuration file by gracefully re-constructing the data pipeline. Fluentd will try to flush the entire memory buffer at once, but will not retry if the flush fails. Fluentd will not flush the file buffer; the logs are persisted on the disk by default.
 
-This signal has been supported since v1.9.0.
+Limitations:
+
+* A change to System Configuration (`<system>`) is ignored.
+* All plugins must not use class variable when restarting.
 
 ### SIGHUP
 
 Reloads the configuration file by gracefully restarting the worker process. Fluentd will try to flush the entire memory buffer at once, but will not retry if the flush fails. Fluentd will not flush the file buffer; the logs are persisted on the disk by default.
 
-If you use fluentd v1.9.0 or later, use `SIGUSR2` instead.
+This does not cause socket downtime because the supervisor process keeps the normal sockets, as long as the socket is provided as a shared socket by [server_helper](../plugin-helper-overview/api-plugin-helper-server.md).
 
 ### SIGCONT
 

deployment/zero-downtime-restart.md

@@ -0,0 +1,48 @@
+# Zero-downtime restart
+
+This feature allows a complete restart of Fluentd without bringing down some input plugins.
+
+Supported standard input plugins are as follows.
+
+| supported input plugin | version |
+| :---                   | :---    |
+| in_udp                 | v1.18.0 |
+| in_tcp                 | v1.18.0 |
+| in_syslog              | v1.18.0 |
+
+If these input plugins are down, client applications may fail to send data.
+If that client does not have a resend feature, the data will be lost.
+
+You can use this feature to completely restart Fluentd without losing data for these plugins.
+
+## How to use this feature
+
+You can use this feature in the following ways.
+
+* [Signals - SIGUSR2](signals.md#sigusr2)
+* [RPC](rpc.md)
+
+## Mechanism of zero-downtime restart
+
+![zero-downtime restart mechanism](../.gitbook/assets/fluentd-zero-downtime-restart-mechanism.png)
+
+1. The old supervisor receives `SIGUSR2`.
+2. Spawn a new supervisor.
+3. Take over shared sockets.
+4. Launch new workers, and stop old processes in parallel.
+   * Launch new workers with [Source Only Mode](source-only-mode.md).
+     * In addition to source-only mode, further limit the starting pluings to only those that support this feature.
+     * Data received by the new workers are stored in the temporary buffer of source-only mode.
+   * Send `SIGTERM` to the old supervisor after `10s` delay.
+5. The old supervisor stops and sends `SIGWINCH` to the new one.
+6. The new workers starts to run fully.
+   * The temporary buffer of source-only mode starts to load.
+
+You can configure the temporary buffer.
+See [Source Only Mode](source-only-mode.md) for details.
+
+## Plugins: how to support this feature
+
+See [How to Write Input Plugin - zero_downtime_restart_ready?](../plugin-development/api-plugin-input.md#zero_downtime_restart_ready).
+
+If this article is incorrect or outdated, or omits critical information, please [let us know](https://github.com/fluent/fluentd-docs-gitbook/issues?state=open). [Fluentd](http://www.fluentd.org/) is an open-source project under [Cloud Native Computing Foundation \(CNCF\)](https://cncf.io/). All components are available under the Apache 2 License.

plugin-development/api-plugin-input.md

@@ -102,7 +102,31 @@ router.emit(tag, time, {:foo => 'bar'})
 
 ## Methods
 
-There are no specific methods for the Input plugins.
+### zero_downtime_restart_ready?
+
+To support [Zero-downtime restart](../deployment/zero-downtime-restart.md), you can override this method to return `true`.
+
+```ruby
+def zero_downtime_restart_ready?
+  true
+end
+```
+
+To do this, the following condition must be met:
+
+* This plugin can run in parallel with another Fluentd.
+
+This is because there is a period when the old process and the new process run in parallel during a zero-downtime restart.
+
+After addressing the following considerations and ensuring there are no issues, override this method.
+Then, the plugin will not experience downtime with zero-downtime restart.
+
+* Handling Files
+  * When handling files, there is a possibility of conflict.
+  * Basically, input plugins that handle files should not support Zero-downtime restart.
+* Handling Sockets
+  * A socket provided as a shared socket by [server plugin helper](../plugin-helper-overview/api-plugin-helper-server.md) is shared between the old and new processes. So, such a plugin can support Zero-downtime restart.
+  * When handling sockets on your own, be careful to avoid conflicts.
 
 ## Writing Tests