You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: pipeline/outputs/s3.md
+7-9Lines changed: 7 additions & 9 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -19,7 +19,7 @@ for details about fetching AWS credentials.
19
19
20
20
{% hint style="warning" %}
21
21
22
-
The [Prometheus success/retry/error metrics values](administration/monitoring.md) output by the built-in HTTP server in Fluent Bit are meaningless for S3 output. S3 has its own buffering and retry mechanisms. The Fluent Bit AWS S3 maintainers apologize for this feature gap; you can [track issue progress on GitHub](https://github.com/fluent/fluent-bit/issues/6141).
22
+
The [Prometheus success/retry/error metrics values](administration/monitoring.md) output by the built-in HTTP server in Fluent Bit are meaningless for S3 output. S3 has its own buffering and retry mechanisms. The Fluent Bit AWS S3 maintainers acknowlege this feature gap, and you can [track issue progress on GitHub](https://github.com/fluent/fluent-bit/issues/6141).
23
23
24
24
{% endhint %}
25
25
@@ -58,8 +58,7 @@ The [Prometheus success/retry/error metrics values](administration/monitoring.md
58
58
59
59
## TLS / SSL
60
60
61
-
To skip TLS verification, set `tls.verify` as `false`. For more details about the properties available and general configuration, refer to
To skip TLS verification, set `tls.verify` as `false`. For more details about the properties available and general configuration, refer to [TLS/SSL](../../administration/transport-security.md).
63
62
64
63
## Permissions
65
64
@@ -84,7 +83,7 @@ The S3 output plugin is used to upload large files to an Amazon S3 bucket, while
84
83
85
84
When Fluent Bit receives logs, it stores them in chunks, either in memory or the filesystem depending on your settings. Chunks are usually around 2 MB in size. Fluent Bit sends chunks, in order, to each output that matches their tag. Most outputs then send the chunk immediately to their destination. A chunk is sent to the output's `flush` callback function, which must return one of `FLB_OK`, `FLB_RETRY`, or `FLB_ERROR`. Fluent Bit keeps count of the return values from each output's `flush` callback function. These counters are the data source for Fluent Bit error, retry, and success metrics available in Prometheus format through its monitoring interface.
86
85
87
-
The S3 output plugin conforms to the Fluent Bit output plugin specification. Since S3's use case is to upload large files (over 2 MB), its behavior is different. S3's `flush` callback function buffers the incoming chunk to the filesystem, and returns an `FLB_OK`. This means Prometheus metrics available from the Fluent Bit HTTP server are meaningless for S3. In addition, the `storage.total_limit_size` parameter isn't meaningful for S3 since it has its own buffering system in the `store_dir`. Instead, use `store_dir_limit_size`. S3 requires a writeable filesystem. Running Fluent Bit on a read-only filesystem won't work with the S3 output.
86
+
The S3 output plugin conforms to the Fluent Bit output plugin specification. Because S3's use case is to upload large files (over 2 MB), its behavior is different. S3's `flush` callback function buffers the incoming chunk to the filesystem, and returns an `FLB_OK`. This means Prometheus metrics available from the Fluent Bit HTTP server are meaningless for S3. In addition, the `storage.total_limit_size` parameter isn't meaningful for S3 since it has its own buffering system in the `store_dir`. Instead, use `store_dir_limit_size`. S3 requires a writeable filesystem. Running Fluent Bit on a read-only filesystem won't work with the S3 output.
88
87
89
88
S3 uploads primarily initiate using the S3 [`timer`](https://docs.aws.amazon.com/iotevents/latest/apireference/API_iotevents-data_Timer.html) callback function, which runs separately from its `flush`.
90
89
@@ -103,7 +102,7 @@ In Fluent Bit, all logs have an associated tag. The `s3_key_format` option lets
103
102
-`$TAG`: The full tag.
104
103
-`$TAG[n]`: The nth part of the tag (index starting at zero). This syntax is copied from the rewrite tag filter. By default, tag parts are separated with dots, but you can change this with `s3_key_format_tag_delimiters`.
105
104
106
-
In the following example, assume the date is `January 1st, 2020 00:00:00` and the tag associated with the logs in question is `my_app_name-logs.prod`.
105
+
In the following example, assume the date is `January 1st, 2020 00:00:00` and the tag associated with the logs in question is `my_app_name-logs.prod`:
107
106
108
107
{% tabs %}
109
108
{% tab title="fluent-bit.yaml" %}
@@ -158,7 +157,7 @@ The `PutObject` API is used in these cases:
158
157
- On startup when the S3 output finds old buffer files in the `store_dir` from a previous run and attempts to send all of them at once.
159
158
- On shutdown. To prevent data loss the S3 output attempts to send all currently buffered data at once.
160
159
161
-
You should always specify `$UUID` somewhere in your S3 key format. Otherwise, if the `PutObject` API is used, S3 appends a random eight-character UUID to the end of your S3 key. This means that a file extension set at the end of an S3 key will have the random UUID appended to it. Disabled this with `static_file_path On`.
160
+
You should always specify `$UUID` somewhere in your S3 key format. Otherwise, if the `PutObject` API is used, S3 appends a random eight-character UUID to the end of your S3 key. This means that a file extension set at the end of an S3 key will have the random UUID appended to it. Disable this with `static_file_path On`.
162
161
163
162
This example attempts to set a `.gz` extension without specifying `$UUID`:
164
163
@@ -287,7 +286,7 @@ To disable this behavior, use one of the following methods:
287
286
288
287
The `store_dir` is used to temporarily store data before upload. If Fluent Bit stops suddenly, it will try to send all data and complete all uploads before it shuts down. If it can not send some data, on restart it will look in the `store_dir` for existing data and try to send it.
289
288
290
-
Multipart uploads are ideal for most use cases because they allow the plugin to upload data in small chunks over time. For example, 1 GB file can be created from 200 5 MB chunks. While the file size in S3 will be 1 GB, only 5 MB will be buffered on disk at any one point in time.
289
+
Multipart uploads are ideal for most use cases because they allow the plugin to upload data in small chunks over time. For example, 1 GB file can be created from 200 5 MB chunks. Although the file size in S3 will be 1 GB, only 5 MB will be buffered on disk at any one point in time.
291
290
292
291
One drawback to multipart uploads is that the file and data aren't visible in S3 until the upload is completed with a [CompleteMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html) call. The plugin attempts to make this call whenever Fluent Bit is shut down to ensure your data is available in S3. It also stores metadata about each upload in the `store_dir`, ensuring that uploads can be completed when Fluent Bit restarts (assuming it has access to persistent disk and the `store_dir` files will still be present on restart).
293
292
@@ -359,8 +358,7 @@ The S3 API allows the last part in an upload to be less than the 5,242,880 byte
359
358
360
359
### `upload_timeout` constrains total multipart upload time for a single file
361
360
362
-
The `upload_timeout` evaluated against the [CreateMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html)
363
-
time. A multipart upload will be completed after `upload_timeout` elapses, even if the desired size hasn't yet been reached.
361
+
The `upload_timeout` evaluated against the [CreateMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html) time. A multipart upload will be completed after `upload_timeout` elapses, even if the desired size hasn't yet been reached.
0 commit comments