Skip to content

Commit 412b444

Browse files
Merge branch 'master' into master
Signed-off-by: Tanmaya Panda <[email protected]>
2 parents 37abfb2 + 15e6bf3 commit 412b444

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

63 files changed

+2338
-1269
lines changed

.gitbook/assets/4.0.png

1.49 MB
Loading

.gitbook/assets/image (1).png

1.09 MB
Loading

.gitbook/assets/image.png

438 KB
Loading

.gitbook/includes/untitled.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
---
2+
title: Untitled
3+
---
4+
5+
{% embed url="https://o11y-workshops.gitlab.io/workshop-fluentbit/lab01.html" fullWidth="false" %}
6+
Lab 1 - Introduction to Fluent Bit
7+
{% endembed %}

.vale.ini

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,12 +1,12 @@
11
StylesPath = vale-styles
22
MinAlertLevel = suggestion # suggestion, warning or error
3+
IgnoredScopes = tt, code
34

45
[formats]
56
mdx = md
67

78
[*.{md,markdown,txt,htm,html}]
89
BasedOnStyles = FluentBit
9-
IgnoredScopes = tt, code
1010

1111
FluentBit.Terms = YES
1212
FluentBit.Titles = YES

CONTRIBUTING.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -113,11 +113,11 @@ request may suggest changes accordingly.
113113

114114
### URLs
115115

116-
When cross-linking to a page in this repository, use a full relative path whenever
116+
When cross-linking to a page in this repository, use a full absolute path whenever
117117
possible. For example:
118118

119119
```text
120-
[LTSV](pipeline/parsers/ltsv.md) and [Logfmt](pipeline/parsers/logfmt.md)
120+
[LTSV](../pipeline/parsers/ltsv.md) and [Logfmt](../pipeline/parsers/logfmt.md)
121121
```
122122

123123
### Vale

README.md

Lines changed: 7 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -2,21 +2,17 @@
22
description: High Performance Telemetry Agent for Logs, Metrics and Traces
33
---
44

5-
# Fluent Bit v3.2 Documentation
5+
# Fluent Bit v4.0 Documentation
66

7-
<figure><img src=".gitbook/assets/3.2.png" alt=""><figcaption></figcaption></figure>
7+
<figure><img src=".gitbook/assets/4.0.png" alt=""><figcaption></figcaption></figure>
88

9-
[Fluent Bit](http://fluentbit.io) is a fast and lightweight telemetry agent
10-
for logs, metrics, and traces for Linux, macOS, Windows, and BSD family
11-
operating systems. Fluent Bit has been made with a strong focus on performance to allow
12-
the collection and processing of telemetry data from different sources without
13-
complexity.![](https://static.scarf.sh/a.png?x-pxid=71f0e011-761f-4c6f-9a89-38817887faae)
9+
[Fluent Bit](http://fluentbit.io) is a fast and lightweight telemetry agent for logs, metrics, and traces for Linux, macOS, Windows, and BSD family operating systems. Fluent Bit has been made with a strong focus on performance to allow the collection and processing of telemetry data from different sources without complexity.![](https://static.scarf.sh/a.png?x-pxid=71f0e011-761f-4c6f-9a89-38817887faae)
1410

1511
## Features
1612

1713
- High performance: High throughput with low resources consumption
1814
- Data parsing
19-
- Convert your unstructured messages using our parsers:
15+
- Convert your unstructured messages using Fluent Bit parsers:
2016
[JSON](pipeline/parsers/json.md),
2117
[Regex](pipeline/parsers/regular-expression.md),
2218
[LTSV](pipeline/parsers/ltsv.md) and [Logfmt](pipeline/parsers/logfmt.md)
@@ -42,15 +38,11 @@ complexity.![](https://static.scarf.sh/a.png?x-pxid=71f0e011-761f-4c6f-9a89-3881
4238
and transformation using simple SQL queries
4339
- Create new streams of data using query results
4440
- Aggregation windows
45-
- Data analysis and prediction: Timeseries forecasting
41+
- Data analysis and prediction: Time series forecasting
4642
- Portable: Runs on Linux, macOS, Windows and BSD systems
4743

4844
## Fluent Bit, Fluentd and CNCF
4945

50-
[Fluent Bit](http://fluentbit.io) is a [CNCF](https://cncf.io) graduated sub-project
51-
under the umbrella of [Fluentd](http://fluentd.org). Fluent Bit is licensed under
52-
the terms of the [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0).
46+
[Fluent Bit](http://fluentbit.io) is a [CNCF](https://cncf.io) graduated sub-project under the umbrella of [Fluentd](http://fluentd.org). Fluent Bit is licensed under the terms of the [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0).
5347

54-
Fluent Bit was originally created by [Eduardo Silva](https://www.linkedin.com/in/edsiper/)
55-
and is now sponsored by [Chronosphere](https://chronosphere.io/). As a
56-
CNCF-hosted project, it is a fully vendor-neutral and community-driven project.
48+
Fluent Bit was originally created by [Eduardo Silva](https://www.linkedin.com/in/edsiper/) and is now sponsored by [Chronosphere](https://chronosphere.io/). As a CNCF-hosted project, it's a fully vendor-neutral and community-driven project.

SUMMARY.md

Lines changed: 12 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,14 @@
11
# Table of contents
22

3-
* [Fluent Bit v3.2 Documentation](README.md)
3+
* [Fluent Bit v4.0 Documentation](README.md)
44

55
## About
66

77
* [What is Fluent Bit?](about/what-is-fluent-bit.md)
88
* [A Brief History of Fluent Bit](about/history.md)
99
* [Fluentd & Fluent Bit](about/fluentd-and-fluent-bit.md)
1010
* [License](about/license.md)
11+
* [Sandbox and Lab Resources](about/sandbox-and-lab-resources.md)
1112

1213
## Concepts
1314

@@ -55,7 +56,7 @@
5556
* [Multiline Parsers](administration/configuring-fluent-bit/yaml/multiline-parsers-section.md)
5657
* [Pipeline](administration/configuring-fluent-bit/yaml/pipeline-section.md)
5758
* [Plugins](administration/configuring-fluent-bit/yaml/plugins-section.md)
58-
* [Upstream Servers](administration/configuring-fluent-bit/yaml/upstream-servers-section.md))
59+
* [Upstream Servers](administration/configuring-fluent-bit/yaml/upstream-servers-section.md)
5960
* [Environment Variables](administration/configuring-fluent-bit/yaml/environment-variables-section.md)
6061
* [Includes](administration/configuring-fluent-bit/yaml/includes-section.md)
6162
* [Classic mode](administration/configuring-fluent-bit/classic-mode/README.md)
@@ -79,6 +80,7 @@
7980
* [Hot Reload](administration/hot-reload.md)
8081
* [Troubleshooting](administration/troubleshooting.md)
8182
* [Performance Tips](administration/performance.md)
83+
* [AWS credentials](administration/aws-credentials.md)
8284

8385
## Local Testing
8486

@@ -145,6 +147,9 @@
145147
* [Metrics Selector](pipeline/processors/metrics-selector.md)
146148
* [OpenTelemetry Envelope](pipeline/processors/opentelemetry-envelope.md)
147149
* [SQL](pipeline/processors/sql.md)
150+
* [Traces](pipeline/processors/traces.md)
151+
* [Filters as processors](pipeline/processors/filters.md)
152+
* [Conditional processing](pipeline/processors/conditional-processing.md)
148153
* [Filters](pipeline/filters/README.md)
149154
* [AWS Metadata](pipeline/filters/aws-metadata.md)
150155
* [CheckList](pipeline/filters/checklist.md)
@@ -173,8 +178,8 @@
173178
* [Amazon Kinesis Data Firehose](pipeline/outputs/firehose.md)
174179
* [Amazon Kinesis Data Streams](pipeline/outputs/kinesis.md)
175180
* [Amazon S3](pipeline/outputs/s3.md)
176-
* [Azure Blob](pipeline/outputs/azure\_blob.md)
177-
* [Azure Data Explorer](pipeline/outputs/azure\_kusto.md)
181+
* [Azure Blob](pipeline/outputs/azure_blob.md)
182+
* [Azure Data Explorer](pipeline/outputs/azure_kusto.md)
178183
* [Azure Log Analytics](pipeline/outputs/azure.md)
179184
* [Azure Logs Ingestion API](pipeline/outputs/azure_logs_ingestion.md)
180185
* [Counter](pipeline/outputs/counter.md)
@@ -194,12 +199,12 @@
194199
* [Kafka REST Proxy](pipeline/outputs/kafka-rest-proxy.md)
195200
* [LogDNA](pipeline/outputs/logdna.md)
196201
* [Loki](pipeline/outputs/loki.md)
197-
* [Microsoft Fabric](pipeline/outputs/azure\_kusto.md)
202+
* [Microsoft Fabric](pipeline/outputs/azure_kusto.md)
198203
* [NATS](pipeline/outputs/nats.md)
199204
* [New Relic](pipeline/outputs/new-relic.md)
200205
* [NULL](pipeline/outputs/null.md)
201206
* [Observe](pipeline/outputs/observe.md)
202-
* [OpenObserve](pipeline/inputs/openobserve.md)
207+
* [OpenObserve](pipeline/outputs/openobserve.md)
203208
* [OpenSearch](pipeline/outputs/opensearch.md)
204209
* [OpenTelemetry](pipeline/outputs/opentelemetry.md)
205210
* [Oracle Log Analytics](pipeline/outputs/oci-logging-analytics.md)
@@ -229,7 +234,7 @@
229234

230235
## Fluent Bit for Developers <a href="#development" id="development"></a>
231236

232-
* [C Library API](development/library\_api.md)
237+
* [C Library API](development/library_api.md)
233238
* [Ingest Records Manually](development/ingest-records-manually.md)
234239
* [Golang Output Plugins](development/golang-output-plugins.md)
235240
* [WASM Filter Plugins](development/wasm-filter-plugins.md)

about/sandbox-and-lab-resources.md

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
---
2+
description: >-
3+
The following page gives an overview of free public resources for Sandbox and
4+
Labs for learning how to best operate, use, and have success with Fluent Bit.
5+
---
6+
7+
# Sandbox and Lab Resources
8+
9+
## Fluent Bit Sandbox - sign-up required
10+
11+
The following are labs that can run in your browser however require email sign-up
12+
13+
<div data-full-width="true"><figure><img src="../.gitbook/assets/image (1).png" alt=""><figcaption></figcaption></figure></div>
14+
15+
* [Fluent Bit 101 Sandbox - Getting Started with configuration and routing](https://play.instruqt.com/Fluent/invite/nuys5ifhsprt)
16+
17+
## Open Source Labs - environment required
18+
19+
The following are open source labs where you will need to spin up resources to run through the lab in details
20+
21+
### O11y Workshops by Chronosphere
22+
23+
These workshops, open source, provided by Chronosphere can be found here: [https://o11y-workshops.gitlab.io/](https://o11y-workshops.gitlab.io/). The OSS repository can be found here: [https://gitlab.com/o11y-workshops/workshop-fluentbit](https://gitlab.com/o11y-workshops/workshop-fluentbit)
24+
25+
The cards below include links to each of the labs in the workshop&#x20;
26+
27+
{% embed url="https://o11y-workshops.gitlab.io/workshop-fluentbit/" %}
28+
Fluent Bit Workshop for Getting Started with Cloud Native Telemetry Pipelines
29+
{% endembed %}
30+
31+
1. [Lab 1 - Introduction to Fluent Bit](https://o11y-workshops.gitlab.io/workshop-fluentbit/lab01.html)
32+
2. [Lab 2 - Installing Fluent Bit](https://o11y-workshops.gitlab.io/workshop-fluentbit/#/4)
33+
3. [Lab 3 - Exploring First Pipelines](https://o11y-workshops.gitlab.io/workshop-fluentbit/#/5)
34+
4. [Lab 4 - Exploring More Pipelines](https://o11y-workshops.gitlab.io/workshop-fluentbit/#/6)
35+
5. [Lab 5 - Understanding Backpressure](https://o11y-workshops.gitlab.io/workshop-fluentbit/#/7)
36+
6. [Lab 6 - Avoid Telemetry Data Loss](https://o11y-workshops.gitlab.io/workshop-fluentbit/#/8)
37+
7. [Lab 7 - Pipeline Integration with OpenTelemetry](https://o11y-workshops.gitlab.io/workshop-fluentbit/#/9)
38+
39+
### Logging with Fluent Bit and Amazon OpenSearch workshop by Amazon
40+
41+
This workshop by Amazon goes through common Kubernetes logging patterns and routing data to OpenSearch and visualizing with OpenSearch dashboards
42+
43+
{% embed url="https://archive.eksworkshop.com/intermediate/230_logging/" %}

administration/aws-credentials.md

Lines changed: 37 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -1,39 +1,53 @@
11
# AWS Credentials
22

3-
Plugins that interact with AWS services will fetch credentials from various providers in the following order.
4-
Only the first provider that is able to provide credentials will be used.
3+
Plugins that interact with AWS services fetch credentials from the following providers
4+
in order. Only the first provider that provides credentials is used.
55

6-
All AWS plugins additionally support a `role_arn` (or `AWS_ROLE_ARN`, for [Elasticsearch](../pipeline/outputs/elasticsearch.md)) configuration parameter. If specified, the fetched credentials will then be used to assume the given role.
6+
- [Environment variables](#environment-variables)
7+
- [Shared configuration and credentials files](#shared-configuration-and-credentials-files)
8+
- [EKS Web Identity Token (OIDC)](#eks-web-identity-token-oidc)
9+
- [ECS HTTP credentials endpoint](#ecs-http-credentials-endpoint)
10+
- [EC2 Instance Profile Credentials (IMDS)](#ec2-instance-profile-credentials-imds)
711

8-
## 1. Environment Variables
12+
All AWS plugins additionally support a `role_arn` (or `AWS_ROLE_ARN`, for
13+
[Elasticsearch](../pipeline/outputs/elasticsearch.md)) configuration parameter. If
14+
specified, the fetched credentials are used to assume the given role.
915

10-
Uses the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` (and optionally `AWS_SESSION_TOKEN`) environment variables if set.
16+
## Environment variables
1117

12-
## 2. Shared Configuration and Credentials Files
18+
Plugins use the `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` (and optionally
19+
`AWS_SESSION_TOKEN`) environment variables if set.
1320

14-
Reads the shared config file at `$AWS_CONFIG_FILE` (or `$HOME/.aws/config`) and the shared credentials file at `$AWS_SHARED_CREDENTIALS_FILE` (or `$HOME/.aws/credentials`) to fetch the credentials for the profile named `$AWS_PROFILE` or `$AWS_DEFAULT_PROFILE` (or "default"). See https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html.
21+
## Shared configuration and credentials files
1522

16-
The shared settings will be evaluated in the following order.
23+
Plugins read the shared `config` file at `$AWS_CONFIG_FILE` (or `$HOME/.aws/config`),
24+
and the shared credentials file at `$AWS_SHARED_CREDENTIALS_FILE` (or
25+
`$HOME/.aws/credentials`) to fetch the credentials for the profile named
26+
`$AWS_PROFILE` or `$AWS_DEFAULT_PROFILE` (or "default"). See
27+
[Configuration and credential file settings in the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html).
1728

18-
Setting|File|Description
19-
---|---|---
20-
`credential_process`|config| See https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sourcing-external.html.<br/>Supported on Linux only.
21-
`aws_access_key_id`<br/>`aws_secret_access_key`<br/>*`aws_session_token`*|credentials|Access key ID and secret key to use to authenticate.<br/>The session token must be set for temporary credentials.
29+
The shared settings evaluate in the following order:
2230

23-
At this time, no other settings are supported.
31+
| Setting | File | Description |
32+
|---|---|---|
33+
| `credential_process` | `config` | Linux only. See [Sourcing credentials with an external process in the AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-sourcing-external.html). |
34+
| `aws_access_key_id`<br />`aws_secret_access_key`<br />`aws_session_token` | `credentials` | Access key ID and secret key to use to authenticate. The session token must be set for temporary credentials. |
2435

25-
## 3. EKS Web Identity Token (OIDC)
36+
No other settings are supported.
2637

27-
Fetches credentials via a signed web identity token for a Kubernetes service account.
28-
See https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html.
38+
## EKS Web Identity Token (OIDC)
2939

30-
## 4. ECS HTTP Credentials Endpoint
40+
Credentials are fetched using a signed web identity token for a Kubernetes service account.
41+
See [IAM roles for service accounts](https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html).
3142

32-
Fetches credentials for the ECS task's role.
33-
See https://docs.aws.amazon.com/AmazonECS/latest/userguide/task-iam-roles.html.
43+
## ECS HTTP credentials endpoint
3444

35-
## 5. EC2 Instance Profile Credentials (IMDS)
45+
Credentials are fetched for the ECS task's role. See
46+
[Amazon ECS task IAM role](https://docs.aws.amazon.com/AmazonECS/latest/userguide/task-iam-roles.html).
3647

37-
Fetches credentials for the EC2 instance profile's role.
38-
See https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html.
39-
As of Fluent Bit version 1.8.8, IMDSv2 is used by default and IMDSv1 may be disabled. Prior versions of Fluent Bit require enabling IMDSv1 on EC2.
48+
## EC2 instance profile credentials (IMDS)
49+
50+
Fetches credentials for the EC2 instance profile's role. See
51+
[IAM roles for Amazon EC2](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html).
52+
As of Fluent Bit version 1.8.8, IMDSv2 is used by default and IMDSv1 might be disabled.
53+
Prior versions of Fluent Bit require enabling IMDSv1 on EC2.

0 commit comments

Comments
 (0)