Skip to content

throttle: support not drop messages #796

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

throttle: support not drop messages #796

Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
eda6f1b
throttle: support not drop messages
wangzhuzhen Apr 25, 2022
7d4dad0
Merge branch 'master' into throttle_retain
esmerel Oct 27, 2025
f03bc8f
Update pipeline/filters/throttle.md
esmerel Oct 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 17 additions & 6 deletions pipeline/filters/throttle.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,8 @@
| `Window` | `Integer` | Amount of intervals to calculate average over. Default: `5`. |
| `Interval` | `String` | Time interval, expressed in `sleep` format. For example, `3s`, `1.5m`, `0.5h`. |
| `Print_Status` | `Bool` | Whether to print status messages with current rate and the limits to information logs. |
| `Retain` | `Bool` | Whether to whether or not to drop logs if rate limit is exceeded. Default: `false`.|


## Functional description

Expand Down Expand Up @@ -63,7 +65,15 @@
+---------+
```

The last pane of the window was overwritten and 1 message was dropped.
The last pane of the window was overwritten and one message was dropped. If you can accept the cost of latency for collector messages, use the `Retain` parameter to retain all the logs.

### Do not drop messages

Check warning on line 70 in pipeline/filters/throttle.md

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[FluentBit.Contractions] Feel free to use 'don't' instead of 'Do not'.

Raw Output:
{"message": "[FluentBit.Contractions] Feel free to use 'don't' instead of 'Do not'.", "location": {"path": "pipeline/filters/throttle.md", "range": {"start": {"line": 70, "column": 5}}}, "severity": "INFO"}

When `Retain` isn't set or is set to `false`, if therate limit is exceeded, throttle drops the messages.

Check warning on line 72 in pipeline/filters/throttle.md

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[FluentBit.Spelling] Spelling check: 'therate'?

Raw Output:
{"message": "[FluentBit.Spelling] Spelling check: 'therate'?", "location": {"path": "pipeline/filters/throttle.md", "range": {"start": {"line": 72, "column": 50}}}, "severity": "INFO"}

In cases when Fluent Bit first runs, and there is a input with huge messages which exceeded the throttle's (`window * rate * interval`), then only the first (`window * rate * interval`) records will be collected. Other records collected before Fluent Bit runs will be dropped.

If `Retain` is set to `true`, all messages will be collected without dropping. This collection has a cost of some latency for collecting all messages, which depends on the account of collected target input.

### `Interval` versus `Window` size

Expand Down Expand Up @@ -147,11 +157,12 @@
Path lines.txt

[FILTER]
Name throttle
Match *
Rate 1000
Window 300
Interval 1s
Name throttle
Match *
Rate 1000
Window 300
Interval 1s
Retain false

[OUTPUT]
Name stdout
Expand Down