Skip to content

Commit a59ef20

Browse files
pipeline: outputs: kinesis: general cleanup
Signed-off-by: Alexa Kreizinger <[email protected]>
1 parent 44919d1 commit a59ef20

File tree

3 files changed

+48
-52
lines changed

3 files changed

+48
-52
lines changed

pipeline/outputs/kinesis.md

Lines changed: 46 additions & 52 deletions
Original file line numberDiff line numberDiff line change
@@ -4,45 +4,45 @@ description: Send logs to Amazon Kinesis Streams
44

55
# Amazon Kinesis Data Streams
66

7-
The Amazon Kinesis Data Streams output plugin allows to ingest your records into the [Kinesis](https://aws.amazon.com/kinesis/data-streams/) service.
7+
The Amazon Kinesis Data Streams output plugin lets you ingest your records into the [Kinesis](https://aws.amazon.com/kinesis/data-streams/) service.
88

9-
This is the documentation for the core Fluent Bit Kinesis plugin written in C. It has all the core features of the [aws/amazon-kinesis-streams-for-fluent-bit](https://github.com/aws/amazon-kinesis-streams-for-fluent-bit) Golang Fluent Bit plugin released in 2019. The Golang plugin was named `kinesis`; this new high performance and highly efficient kinesis plugin is called `kinesis_streams` to prevent conflicts/confusion.
9+
This is the documentation for the core Fluent Bit Kinesis plugin written in C. It has all the core features of the [aws/amazon-kinesis-streams-for-fluent-bit](https://github.com/aws/amazon-kinesis-streams-for-fluent-bit) Golang Fluent Bit plugin released in 2019. The original Golang plugin is named `kinesis`, and this new high performance and highly efficient Kinesis plugin is called `kinesis_streams` to prevent conflicts or confusion.
1010

11-
Currently, this `kinesis_streams` plugin will always use a random partition key when uploading records to kinesis via the [PutRecords API](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecords.html).
11+
This `kinesis_streams` plugin always uses a random partition key when uploading records to Kinesis through the [PutRecords API](https://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecords.html).
1212

13-
See [here](https://github.com/fluent/fluent-bit-docs/tree/43c4fe134611da471e706b0edb2f9acd7cdfdbc3/administration/aws-credentials.md) for details on how AWS credentials are fetched.
13+
For information about how AWS credentials are fetched, see [AWS credentials](../administration/aws-credentials).
1414

15-
## Configuration Parameters
15+
## Configuration parameters
1616

17-
| Key | Description |
18-
| :--- | :--- |
19-
| region | The AWS region. |
20-
| stream | The name of the Kinesis Streams Delivery stream that you want log records sent to. |
21-
| time\_key | Add the timestamp to the record under this key. By default the timestamp from Fluent Bit will not be added to records sent to Kinesis. |
22-
| time\_key\_format | strftime compliant format string for the timestamp; for example, the default is '%Y-%m-%dT%H:%M:%S'. Supports millisecond precision with '%3N' and supports nanosecond precision with '%9N' and '%L'; for example, adding '%3N' to support millisecond '%Y-%m-%dT%H:%M:%S.%3N'. This option is used with time\_key. |
23-
| log\_key | By default, the whole log record will be sent to Kinesis. If you specify a key name with this option, then only the value of that key will be sent to Kinesis. For example, if you are using the Fluentd Docker log driver, you can specify `log_key log` and only the log message will be sent to Kinesis. |
24-
| role\_arn | ARN of an IAM role to assume \(for cross account access\). |
25-
| endpoint | Specify a custom endpoint for the Kinesis API. |
26-
| port | TCP port of the Kinesis Streams service. Defaults to port `443`. |
27-
| sts\_endpoint | Custom endpoint for the STS API. |
28-
| auto\_retry\_requests | Immediately retry failed requests to AWS services once. This option does not affect the normal Fluent Bit retry mechanism with backoff. Instead, it enables an immediate retry with no delay for networking errors, which may help improve throughput when there are transient/random networking issues. This option defaults to `true`. |
29-
| external\_id | Specify an external ID for the STS API, can be used with the role_arn parameter if your role requires an external ID. |
30-
| profile | AWS profile name to use. Defaults to `default`. |
31-
| workers | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. Default: `1`. |
17+
| Key | Description | Default |
18+
| --- | ----------- | ------- |
19+
| `region` | The AWS region. | _none_ |
20+
| `stream` | The name of the Kinesis Streams Delivery stream that you want log records sent to. | _none_ |
21+
| `time_key` | Add the timestamp to the record under this key. | `false` |
22+
| `time_key_format` | The strftime compliant format string for the timestamp. Supports millisecond precision with `%3N` and supports nanosecond precision with `%9N` and `%L`. For example, adding `%3N` to support millisecond `%Y-%m-%dT%H:%M:%S.%3N`. This option is used with `time_key`. | `%Y-%m-%dT%H:%M:%S` |
23+
| `log_key` | By default, the whole log record will be sent to Kinesis. If you specify a key name with this option, then only the value of that key will be sent to Kinesis. For example, if you are using the Fluentd Docker log driver, you can specify `log_key log` and only the log message will be sent to Kinesis. | _none_ |
24+
| `role_arn` | ARN of an IAM role to assume (for cross account access). | _none_ |
25+
| `endpoint` | Specify a custom endpoint for the Kinesis API. | _none_ |
26+
| `port` | TCP port of the Kinesis Streams service. | `443` |
27+
| `sts_endpoint` | Custom endpoint for the STS API. | _none_ |
28+
| `auto_retry_requests` | Immediately retry failed requests to AWS services once. This option doesn't affect the normal Fluent Bit retry mechanism with backoff. Instead, it enables an immediate retry with no delay for networking errors, which might help improve throughput when there are transient/random networking issues. | `true` |
29+
| `external_id` | Specify an external ID for the STS API, can be used with the `role_arn` parameter if your role requires an external ID. | _none_ |
30+
| `profile` | AWS profile name to use. | `default` |
31+
| `workers` | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `1` |
3232

33-
## Getting Started
33+
## Get started
3434

35-
In order to send records into Amazon Kinesis Data Streams, you can run the plugin from the command line or through the configuration file:
35+
To send records into Amazon Kinesis Data Streams, you can run the plugin from the command line or through the configuration file.
3636

37-
### Command Line
37+
### Command line
3838

39-
The **kinesis\_streams** plugin, can read the parameters from the command line through the **-p** argument \(property\), e.g:
39+
The *`kinesis_streams` plugin can read the parameters from the command line through the `-p` (property) argument. For example:
4040

4141
```shell
4242
fluent-bit -i cpu -o kinesis_streams -p stream=my-stream -p region=us-west-2 -m '*' -f 1
4343
```
4444

45-
### Configuration File
45+
### Configuration file
4646

4747
In your main configuration file append the following:
4848

@@ -51,7 +51,7 @@ In your main configuration file append the following:
5151

5252
```yaml
5353
pipeline:
54-
54+
5555
outputs:
5656
- name: kinesis_steams
5757
match: '*'
@@ -78,9 +78,9 @@ pipeline:
7878
The following AWS IAM permissions are required to use this plugin:
7979

8080
```json
81-
{
82-
"Version": "2012-10-17",
83-
"Statement": [{
81+
{
82+
"Version": "2012-10-17",
83+
"Statement": [{
8484
"Effect": "Allow",
8585
"Action": [
8686
"kinesis:PutRecords"
@@ -90,48 +90,42 @@ The following AWS IAM permissions are required to use this plugin:
9090
}
9191
```
9292

93-
### AWS for Fluent Bit
93+
## Container images
9494

9595
Amazon distributes a container image with Fluent Bit and these plugins.
9696

97-
#### GitHub
98-
99-
[github.com/aws/aws-for-fluent-bit](https://github.com/aws/aws-for-fluent-bit)
97+
### GitHub
10098

101-
#### Amazon ECR Public Gallery
99+
The [aws-for-fluent-bit](https://github.com/aws/aws-for-fluent-bit) container image is available on GitHub.
102100

103-
[aws-for-fluent-bit](https://gallery.ecr.aws/aws-observability/aws-for-fluent-bit)
101+
### Amazon ECR Public Gallery
104102

105-
Our images are available in Amazon ECR Public Gallery. You can download images with different tags by following command:
103+
The [aws-for-fluent-bit](https://gallery.ecr.aws/aws-observability/aws-for-fluent-bit) container image is available on the Amazon ECR Public Gallery. You can use the following command to download images with different tags:
106104

107105
```shell
108-
docker pull public.ecr.aws/aws-observability/aws-for-fluent-bit:<tag>
106+
docker pull public.ecr.aws/aws-observability/aws-for-fluent-bit:TAG
109107
```
110108

111-
For example, you can pull the image with latest version by:
112-
113-
```shell
114-
docker pull public.ecr.aws/aws-observability/aws-for-fluent-bit:latest
115-
```
109+
Replace `TAG` with the tag of the image you want to download. You can also use the value `latest` to download the latest image.
116110

117-
If you see errors for image pull limits, try log into public ECR with your AWS credentials:
111+
If you see errors for image pull limits, try to log into the gallery with your AWS credentials:
118112

119113
```shell
120114
aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws
121115
```
122116

123-
You can check the [Amazon ECR Public official doc](https://docs.aws.amazon.com/AmazonECR/latest/public/get-set-up-for-amazon-ecr.html) for more details.
117+
You can use the Fluent Bit SSM Public Parameters to find the Amazon ECR image URI in your region:
124118

125-
#### Docker Hub
119+
```shell
120+
aws ssm get-parameters-by-path --path /aws/service/aws-for-fluent-bit/
121+
```
126122

127-
[amazon/aws-for-fluent-bit](https://hub.docker.com/r/amazon/aws-for-fluent-bit/tags)
123+
For more details, see the [Amazon ECR Public official doc](https://docs.aws.amazon.com/AmazonECR/latest/public/get-set-up-for-amazon-ecr.html).
128124

129-
#### Amazon ECR
125+
### Docker Hub
130126

131-
You can use our SSM Public Parameters to find the Amazon ECR image URI in your region:
127+
The [aws-for-fluent-bit](https://hub.docker.com/r/amazon/aws-for-fluent-bit/tags) container image is available on Docker Hub.
132128

133-
```shell
134-
aws ssm get-parameters-by-path --path /aws/service/aws-for-fluent-bit/
135-
```
129+
### More information
136130

137-
For more see [the AWS for Fluent Bit github repo](https://github.com/aws/aws-for-fluent-bit#public-images).
131+
For more information, see the [aws-for-fluent-bit README](https://github.com/aws/aws-for-fluent-bit#public-images) on GitHub.

vale-styles/FluentBit/Acronyms.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -92,6 +92,7 @@ exceptions:
9292
- SQL
9393
- SSH
9494
- SSL
95+
- SSM
9596
- SSO
9697
- STS
9798
- SVG

vale-styles/FluentBit/Headings.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ exceptions:
1111
- Amazon CloudWatch
1212
- Amazon ECR Public Gallery
1313
- Amazon Kinesis Data Firehose
14+
- Amazon Kinesis Data Streams
1415
- Amazon Kinesis Firehose
1516
- Amazon Kinesis Streams
1617
- Amazon OpenSearch Service

0 commit comments

Comments
 (0)