diff --git a/pipeline/filters/throttle.md b/pipeline/filters/throttle.md index c45e726d5..65aace039 100644 --- a/pipeline/filters/throttle.md +++ b/pipeline/filters/throttle.md @@ -12,6 +12,8 @@ The plugin supports the following configuration parameters: | `Window` | `Integer` | Amount of intervals to calculate average over. Default: `5`. | | `Interval` | `String` | Time interval, expressed in `sleep` format. For example, `3s`, `1.5m`, `0.5h`. | | `Print_Status` | `Bool` | Whether to print status messages with current rate and the limits to information logs. | +| `Retain` | `Bool` | Whether to whether or not to drop logs if rate limit is exceeded. Default: `false`.| + ## Functional description @@ -63,7 +65,15 @@ will become: +---------+ ``` -The last pane of the window was overwritten and 1 message was dropped. +The last pane of the window was overwritten and one message was dropped. If you can accept the cost of latency for collector messages, use the `Retain` parameter to retain all the logs. + +### Do not drop messages + +When `Retain` isn't set or is set to `false`, if the rate limit is exceeded, throttle drops the messages. + +In cases when Fluent Bit first runs, and there is a input with huge messages which exceeded the throttle's (`window * rate * interval`), then only the first (`window * rate * interval`) records will be collected. Other records collected before Fluent Bit runs will be dropped. + +If `Retain` is set to `true`, all messages will be collected without dropping. This collection has a cost of some latency for collecting all messages, which depends on the account of collected target input. ### `Interval` versus `Window` size @@ -147,11 +157,12 @@ pipeline: Path lines.txt [FILTER] - Name throttle - Match * - Rate 1000 - Window 300 - Interval 1s + Name throttle + Match * + Rate 1000 + Window 300 + Interval 1s + Retain false [OUTPUT] Name stdout