Skip to content

Commit 1d8026a

Browse files
esmerelcnorris-cs
andauthored
Apply suggestions from code review
Co-authored-by: Craig Norris <[email protected]> Signed-off-by: Lynette Miles <[email protected]>
1 parent b06d2bd commit 1d8026a

File tree

1 file changed

+7
-9
lines changed

1 file changed

+7
-9
lines changed

pipeline/outputs/s3.md

Lines changed: 7 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ for details about fetching AWS credentials.
1919

2020
{% hint style="warning" %}
2121

22-
The [Prometheus success/retry/error metrics values](administration/monitoring.md) output by the built-in HTTP server in Fluent Bit are meaningless for S3 output. S3 has its own buffering and retry mechanisms. The Fluent Bit AWS S3 maintainers apologize for this feature gap; you can [track issue progress on GitHub](https://github.com/fluent/fluent-bit/issues/6141).
22+
The [Prometheus success/retry/error metrics values](administration/monitoring.md) output by the built-in HTTP server in Fluent Bit are meaningless for S3 output. S3 has its own buffering and retry mechanisms. The Fluent Bit AWS S3 maintainers acknowlege this feature gap, and you can [track issue progress on GitHub](https://github.com/fluent/fluent-bit/issues/6141).
2323

2424
{% endhint %}
2525

@@ -58,8 +58,7 @@ The [Prometheus success/retry/error metrics values](administration/monitoring.md
5858

5959
## TLS / SSL
6060

61-
To skip TLS verification, set `tls.verify` as `false`. For more details about the properties available and general configuration, refer to
62-
[TLS/SSL](../../administration/transport-security.md).
61+
To skip TLS verification, set `tls.verify` as `false`. For more details about the properties available and general configuration, refer to [TLS/SSL](../../administration/transport-security.md).
6362

6463
## Permissions
6564

@@ -84,7 +83,7 @@ The S3 output plugin is used to upload large files to an Amazon S3 bucket, while
8483

8584
When Fluent Bit receives logs, it stores them in chunks, either in memory or the filesystem depending on your settings. Chunks are usually around 2&nbsp;MB in size. Fluent Bit sends chunks, in order, to each output that matches their tag. Most outputs then send the chunk immediately to their destination. A chunk is sent to the output's `flush` callback function, which must return one of `FLB_OK`, `FLB_RETRY`, or `FLB_ERROR`. Fluent Bit keeps count of the return values from each output's `flush` callback function. These counters are the data source for Fluent Bit error, retry, and success metrics available in Prometheus format through its monitoring interface.
8685

87-
The S3 output plugin conforms to the Fluent Bit output plugin specification. Since S3's use case is to upload large files (over 2&nbsp;MB), its behavior is different. S3's `flush` callback function buffers the incoming chunk to the filesystem, and returns an `FLB_OK`. This means Prometheus metrics available from the Fluent Bit HTTP server are meaningless for S3. In addition, the `storage.total_limit_size` parameter isn't meaningful for S3 since it has its own buffering system in the `store_dir`. Instead, use `store_dir_limit_size`. S3 requires a writeable filesystem. Running Fluent Bit on a read-only filesystem won't work with the S3 output.
86+
The S3 output plugin conforms to the Fluent Bit output plugin specification. Because S3's use case is to upload large files (over 2&nbsp;MB), its behavior is different. S3's `flush` callback function buffers the incoming chunk to the filesystem, and returns an `FLB_OK`. This means Prometheus metrics available from the Fluent Bit HTTP server are meaningless for S3. In addition, the `storage.total_limit_size` parameter isn't meaningful for S3 since it has its own buffering system in the `store_dir`. Instead, use `store_dir_limit_size`. S3 requires a writeable filesystem. Running Fluent Bit on a read-only filesystem won't work with the S3 output.
8887

8988
S3 uploads primarily initiate using the S3 [`timer`](https://docs.aws.amazon.com/iotevents/latest/apireference/API_iotevents-data_Timer.html) callback function, which runs separately from its `flush`.
9089

@@ -103,7 +102,7 @@ In Fluent Bit, all logs have an associated tag. The `s3_key_format` option lets
103102
- `$TAG`: The full tag.
104103
- `$TAG[n]`: The nth part of the tag (index starting at zero). This syntax is copied from the rewrite tag filter. By default, tag parts are separated with dots, but you can change this with `s3_key_format_tag_delimiters`.
105104

106-
In the following example, assume the date is `January 1st, 2020 00:00:00` and the tag associated with the logs in question is `my_app_name-logs.prod`.
105+
In the following example, assume the date is `January 1st, 2020 00:00:00` and the tag associated with the logs in question is `my_app_name-logs.prod`:
107106

108107
{% tabs %}
109108
{% tab title="fluent-bit.yaml" %}
@@ -158,7 +157,7 @@ The `PutObject` API is used in these cases:
158157
- On startup when the S3 output finds old buffer files in the `store_dir` from a previous run and attempts to send all of them at once.
159158
- On shutdown. To prevent data loss the S3 output attempts to send all currently buffered data at once.
160159

161-
You should always specify `$UUID` somewhere in your S3 key format. Otherwise, if the `PutObject` API is used, S3 appends a random eight-character UUID to the end of your S3 key. This means that a file extension set at the end of an S3 key will have the random UUID appended to it. Disabled this with `static_file_path On`.
160+
You should always specify `$UUID` somewhere in your S3 key format. Otherwise, if the `PutObject` API is used, S3 appends a random eight-character UUID to the end of your S3 key. This means that a file extension set at the end of an S3 key will have the random UUID appended to it. Disable this with `static_file_path On`.
162161

163162
This example attempts to set a `.gz` extension without specifying `$UUID`:
164163

@@ -287,7 +286,7 @@ To disable this behavior, use one of the following methods:
287286

288287
The `store_dir` is used to temporarily store data before upload. If Fluent Bit stops suddenly, it will try to send all data and complete all uploads before it shuts down. If it can not send some data, on restart it will look in the `store_dir` for existing data and try to send it.
289288

290-
Multipart uploads are ideal for most use cases because they allow the plugin to upload data in small chunks over time. For example, 1&nbsp;GB file can be created from 200 5&nbsp;MB chunks. While the file size in S3 will be 1&nbsp;GB, only 5&nbsp;MB will be buffered on disk at any one point in time.
289+
Multipart uploads are ideal for most use cases because they allow the plugin to upload data in small chunks over time. For example, 1&nbsp;GB file can be created from 200 5&nbsp;MB chunks. Although the file size in S3 will be 1&nbsp;GB, only 5&nbsp;MB will be buffered on disk at any one point in time.
291290

292291
One drawback to multipart uploads is that the file and data aren't visible in S3 until the upload is completed with a [CompleteMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html) call. The plugin attempts to make this call whenever Fluent Bit is shut down to ensure your data is available in S3. It also stores metadata about each upload in the `store_dir`, ensuring that uploads can be completed when Fluent Bit restarts (assuming it has access to persistent disk and the `store_dir` files will still be present on restart).
293292

@@ -359,8 +358,7 @@ The S3 API allows the last part in an upload to be less than the 5,242,880 byte
359358

360359
### `upload_timeout` constrains total multipart upload time for a single file
361360

362-
The `upload_timeout` evaluated against the [CreateMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html)
363-
time. A multipart upload will be completed after `upload_timeout` elapses, even if the desired size hasn't yet been reached.
361+
The `upload_timeout` evaluated against the [CreateMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CreateMultipartUpload.html) time. A multipart upload will be completed after `upload_timeout` elapses, even if the desired size hasn't yet been reached.
364362

365363
### Completing uploads
366364

0 commit comments

Comments
 (0)