You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Plugins that interact with AWS services fetch credentials from the following providers
4
-
in order. Only the first provider that provides credentials is used.
3
+
Plugins that interact with AWS services fetch credentials from the following providers in order. Only the first provider that provides credentials is used.
5
4
6
5
-[Environment variables](#environment-variables)
7
6
-[Shared configuration and credentials files](#shared-configuration-and-credentials-files)
8
7
-[EKS Web Identity Token (OIDC)](#eks-web-identity-token-oidc)
All AWS plugins additionally support a `role_arn` (or `AWS_ROLE_ARN`, for
13
-
[Elasticsearch](../pipeline/outputs/elasticsearch.md)) configuration parameter. If
14
-
specified, the fetched credentials are used to assume the given role.
11
+
All AWS plugins additionally support a `role_arn` (or `AWS_ROLE_ARN`, for [Elasticsearch](../pipeline/outputs/elasticsearch.md)) configuration parameter. If specified, the fetched credentials are used to assume the given role.
15
12
16
13
## Environment variables
17
14
18
-
Plugins use the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` (and optionally
19
-
`AWS_SESSION_TOKEN`) environment variables if set.
15
+
Plugins use the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` (and optionally `AWS_SESSION_TOKEN`) environment variables if set.
20
16
21
17
## Shared configuration and credentials files
22
18
23
-
Plugins read the shared `config` file at `$AWS_CONFIG_FILE` (or `$HOME/.aws/config`),
24
-
and the shared credentials file at `$AWS_SHARED_CREDENTIALS_FILE` (or
25
-
`$HOME/.aws/credentials`) to fetch the credentials for the profile named
26
-
`$AWS_PROFILE` or `$AWS_DEFAULT_PROFILE` (or "default"). See
27
-
[Configuration and credential file settings in the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html).
19
+
Plugins read the shared `config` file at `$AWS_CONFIG_FILE` (or `$HOME/.aws/config`), and the shared credentials file at `$AWS_SHARED_CREDENTIALS_FILE` (or `$HOME/.aws/credentials`) to fetch the credentials for the profile named `$AWS_PROFILE` or `$AWS_DEFAULT_PROFILE` (or "default"). See [Configuration and credential file settings in the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html).
28
20
29
21
The shared settings evaluate in the following order:
30
22
@@ -37,22 +29,16 @@ No other settings are supported.
37
29
38
30
## EKS Web Identity Token (OIDC)
39
31
40
-
Credentials are fetched using a signed web identity token for a Kubernetes service account.
41
-
See [IAM roles for service accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html).
32
+
Credentials are fetched using a signed web identity token for a Kubernetes service account. See [IAM roles for service accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html).
42
33
43
34
## ECS HTTP credentials endpoint
44
35
45
-
Credentials are fetched for the ECS task's role. See
46
-
[Amazon ECS task IAM role](https://docs.aws.amazon.com/AmazonECS/latest/userguide/task-iam-roles.html).
36
+
Credentials are fetched for the ECS task's role. See [Amazon ECS task IAM role](https://docs.aws.amazon.com/AmazonECS/latest/userguide/task-iam-roles.html).
47
37
48
38
## EKS Pod Identity credentials
49
39
50
-
Credentials are fetched using a pod identity endpoint. See
51
-
[Learn how EKS Pod Identity grants pods access to AWS services](https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html).
40
+
Credentials are fetched using a pod identity endpoint. See [Learn how EKS Pod Identity grants pods access to AWS services](https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html).
52
41
53
42
## EC2 instance profile credentials (IMDS)
54
43
55
-
Fetches credentials for the EC2 instance profile's role. See
56
-
[IAM roles for Amazon EC2](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html).
57
-
As of Fluent Bit version 1.8.8, IMDSv2 is used by default and IMDSv1 might be disabled.
58
-
Prior versions of Fluent Bit require enabling IMDSv1 on EC2.
44
+
Fetches credentials for the EC2 instance profile's role. See [IAM roles for Amazon EC2](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html). As of Fluent Bit version 1.8.8, IMDSv2 is used by default and IMDSv1 might be disabled. Prior versions of Fluent Bit require enabling IMDSv1 on EC2.
a `[info] [input] {input name or alias} resume (mem buf overlimit)` message.
32
-
33
-
Mitigate the risk of data loss by configuring secondary storage on the filesystem
34
-
using the `storage.type` of `filesystem` (as described in [Buffering &
35
-
Storage](buffering-and-storage.md)). Initially, logs will be buffered to both memory
36
-
and the filesystem. When the `storage.max_chunks_up` limit is reached, all new data
37
-
will be stored in the filesystem. Fluent Bit stops queueing new data in memory and
38
-
buffers only to the filesystem. When `storage.type filesystem` is set, the
39
-
`Mem_Buf_Limit` setting no longer has any effect. Instead, the `[SERVICE]` level
40
-
`storage.max_chunks_up` setting controls the size of the memory buffer.
5
+
It's possible for logs or data to be ingested or created faster than the ability toflush it to some destinations. A common scenario is when reading from big log files, especially with a large backlog, and dispatching the logs to a backend over the network, which takes time to respond. This generates _backpressure_, leading to high memory consumption in the service.
6
+
7
+
To avoid backpressure, Fluent Bit implements a mechanism in the engine that restricts the amount of data an input plugin can ingest. Restriction is done through the configuration parameters `Mem_Buf_Limit` and `storage.Max_Chunks_Up`.
8
+
9
+
As described in the [Buffering](../concepts/buffering.md) concepts section, Fluent Bit offers two modes for data handling: in-memory only (default) and in-memory and filesystem (optional).
10
+
11
+
The default `storage.type memory` buffer can be restricted with `Mem_Buf_Limit`. If memory reaches this limit and you reach a backpressure scenario, you won't be able to ingest more data until the data chunks that are in memory can be flushed. The input pauses and Fluent Bit [emits](https://github.com/fluent/fluent-bit/blob/v2.0.0/src/flb_input_chunk.c#L1334) a `[warn] [input] {input name or alias} paused (mem buf overlimit)` log message.
12
+
13
+
Depending on the input plugin in use, this might cause incoming data to be discarded (for example, TCP input plugin). The tail plugin can handle pauses without data ingloss, storing its current file offset and resuming reading later. When buffer memory is available, the input resumes accepting logs. Fluent Bit [emits](https://github.com/fluent/fluent-bit/blob/v2.0.0/src/flb_input_chunk.c#L1277) a `[info] [input] {input name or alias} resume (mem buf overlimit)` message.
14
+
15
+
Mitigate the risk of data loss by configuring secondary storage on the filesystem using the `storage.type` of `filesystem` (as described in [Buffering & Storage](buffering-and-storage.md)). Initially, logs will be buffered to both memory and the filesystem. When the `storage.max_chunks_up` limit is reached, all new data will be stored in the filesystem. Fluent Bit stops queueing new data in memory and buffers only to the filesystem. When `storage.type filesystem` is set, the `Mem_Buf_Limit` setting no longer has any effect. Instead, the `[SERVICE]` level `storage.max_chunks_up` setting controls the size of the memory buffer.
41
16
42
17
## `Mem_Buf_Limit`
43
18
44
-
`Mem_Buf_Limit` applies only with the default `storage.type memory`. This option is
45
-
disabled by default and can be applied to all input plugins.
19
+
`Mem_Buf_Limit` applies only with the default `storage.type memory`. This option is disabled by default and can be applied to all input plugins.
46
20
47
21
As an example situation:
48
22
@@ -53,21 +27,14 @@ As an example situation:
53
27
- Engine scheduler retries the flush after 10 seconds.
54
28
- The input plugin tries to append 500 KB.
55
29
56
-
In this situation, the engine allows appending those 500 KB of data into the memory,
57
-
with a total of 1.2 MB of data buffered. The limit is permissive and will
58
-
allow a single write past the limit. When the limit is exceeded, the following
59
-
actions are taken:
30
+
In this situation, the engine allows appending those 500 KB of data into the memory, with a total of 1.2 MB of data buffered. The limit is permissive and will allow a single write past the limit. When the limit is exceeded, the following actions are taken:
60
31
61
32
- Block local buffers for the input plugin (can't append more data).
62
33
- Notify the input plugin, invoking a `pause` callback.
63
34
64
-
The engine protects itself and won't append more data coming from the input plugin in
65
-
question. It's the responsibility of the plugin to keep state and decide what to do
66
-
in a `paused` state.
35
+
The engine protects itself and won't append more data coming from the input plugin in question. It's the responsibility of the plugin to keep state and decide what to do in a `paused` state.
67
36
68
-
In a few seconds, if the scheduler was able to flush the initial 700 KB of data or it
69
-
has given up after retrying, that amount of memory is released and the following
70
-
actions occur:
37
+
In a few seconds, if the scheduler was able to flush the initial 700 KB of data or it has given up after retrying, that amount of memory is released and the following actions occur:
71
38
72
39
- Upon data buffer release (700 KB), the internal counters get updated.
73
40
- Counters now are set at 500 KB.
@@ -77,42 +44,28 @@ actions occur:
77
44
78
45
## `storage.max_chunks_up`
79
46
80
-
The `[SERVICE]` level `storage.max_chunks_up` setting controls the size of the memory
81
-
buffer. When `storage.type filesystem` is set, the `Mem_Buf_Limit` setting no longer
82
-
has an effect.
47
+
The `[SERVICE]` level `storage.max_chunks_up` setting controls the size of the memory buffer. When `storage.type filesystem` is set, the `Mem_Buf_Limit` setting no longer has an effect.
83
48
84
-
The setting behaves similar to the `Mem_Buf_Limit` scenario when the non-default
85
-
`storage.pause_on_chunks_overlimit` is enabled.
49
+
The setting behaves similar to the `Mem_Buf_Limit` scenario when the non-default `storage.pause_on_chunks_overlimit` is enabled.
86
50
87
-
When (default) `storage.pause_on_chunks_overlimit` is disabled, the input won't pause
88
-
when the memory limit is reached. Instead, it switches to buffering logs only in
89
-
the filesystem. Limit the disk spaced used for filesystem buffering with
90
-
`storage.total_limit_size`.
51
+
When (default) `storage.pause_on_chunks_overlimit` is disabled, the input won't pause when the memory limit is reached. Instead, it switches to buffering logs only in the filesystem. Limit the disk spaced used for filesystem buffering with `storage.total_limit_size`.
91
52
92
53
See [Buffering & Storage](buffering-and-storage.md) docs for more information.
93
54
94
55
## About pause and resume callbacks
95
56
96
-
Each plugin is independent and not all of them implement `pause` and `resume`
97
-
callbacks. These callbacks are a notification mechanism for the plugin.
57
+
Each plugin is independent and not all of them implement `pause` and `resume` callbacks. These callbacks are a notification mechanism for the plugin.
98
58
99
-
One example of a plugin that implements these callbacks and keeps state correctly is
100
-
the [Tail Input](../pipeline/inputs/tail.md) plugin. When the `pause` callback
101
-
triggers, it pauses its collectors and stops appending data. Upon `resume`, it
102
-
resumes the collectors and continues ingesting data. Tail tracks the current file
103
-
offset when it pauses, and resumes at the same position. If the file hasn't been
104
-
deleted or moved, it can still be read.
59
+
One example of a plugin that implements these callbacks and keeps state correctly is the [Tail Input](../pipeline/inputs/tail.md) plugin. When the `pause` callback triggers, it pauses its collectors and stops appending data. Upon `resume`, it resumes the collectors and continues ingesting data. Tail tracks the current file offset when it pauses, and resumes at the same position. If the file hasn't been deleted or moved, it can still be read.
105
60
106
-
With the default `storage.type memory` and `Mem_Buf_Limit`, the following log
107
-
messages emit for `pause` and `resume`:
61
+
With the default `storage.type memory` and `Mem_Buf_Limit`, the following log messages emit for `pause` and `resume`:
108
62
109
63
```text
110
64
[warn] [input] {input name or alias} paused (mem buf overlimit)
111
65
[info] [input] {input name or alias} resume (mem buf overlimit)
112
66
```
113
67
114
-
With `storage.type filesystem` and `storage.max_chunks_up`, the following log
115
-
messages emit for `pause` and `resume`:
68
+
With `storage.type filesystem` and `storage.max_chunks_up`, the following log messages emit for `pause` and `resume`:
116
69
117
70
```text
118
71
[input] {input name or alias} paused (storage buf overlimit)
0 commit comments