You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: about/fluentd-and-fluent-bit.md
+7-21Lines changed: 7 additions & 21 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,42 +4,28 @@ description: The production grade telemetry ecosystem
4
4
5
5
# Fluentd and Fluent Bit
6
6
7
-
Telemetry data processing can be complex, especially at scale. That's why
8
-
[Fluentd](https://www.fluentd.org) was created. Fluentd is more than a simple tool,
9
-
it's grown into a fullscale ecosystem that contains SDKs for different languages
10
-
and subprojects like [Fluent Bit](https://fluentbit.io).
7
+
Telemetry data processing can be complex, especially at scale. That's why [Fluentd](https://www.fluentd.org) was created. Fluentd is more than a simple tool, it's grown into a fullscale ecosystem that contains SDKs for different languages and subprojects like [Fluent Bit](https://fluentbit.io).
11
8
12
-
Here, we describe the relationship between the [Fluentd](http://fluentd.org)
13
-
and [Fluent Bit](http://fluentbit.io) open source projects.
9
+
Here, we describe the relationship between the [Fluentd](http://fluentd.org) and [Fluent Bit](http://fluentbit.io) open source projects.
14
10
15
11
Both projects are:
16
12
17
-
- Licensed under the terms of Apache License v2.0.
18
-
- Graduated hosted projects by the [Cloud Native Computing Foundation (CNCF)](https://cncf.io).
19
-
- Production grade solutions: Deployed millions of times every single day.
20
-
- Vendor neutral and community driven.
21
-
- Widely adopted by the industry: Trusted by major companies like AWS, Microsoft,
22
-
Google Cloud, and hundreds of others.
13
+
- Licensed under the terms of Apache License v2.0. - Graduated hosted projects by the [Cloud Native Computing Foundation (CNCF)](https://cncf.io). - Production grade solutions: Deployed millions of times every single day. - Vendor neutral and community driven. - Widely adopted by the industry: Trusted by major companies like AWS, Microsoft, Google Cloud, and hundreds of others.
23
14
24
-
The projects have many similarities: [Fluent Bit](https://fluentbit.io) is
25
-
designed and built on top of the best ideas of [Fluentd](https://www.fluentd.org)
26
-
architecture and general design. Which one you choose depends on your end-users' needs.
15
+
The projects have many similarities: [Fluent Bit](https://fluentbit.io) is designed and built on top of the best ideas of [Fluentd](https://www.fluentd.org) architecture and general design. Which one you choose depends on your end-users' needs.
27
16
28
17
The following table describes a comparison of different areas of the projects:
Both [Fluentd](https://www.fluentd.org) and [Fluent Bit](https://fluentbit.io)
41
-
can work as Aggregators or Forwarders, and can complement each other or be used
42
-
as standalone solutions.
29
+
Both [Fluentd](https://www.fluentd.org) and [Fluent Bit](https://fluentbit.io) can work as Aggregators or Forwarders, and can complement each other or be used as standalone solutions.
43
30
44
-
In the recent years, cloud providers have switched from Fluentd to Fluent Bit for
45
-
performance and compatibility. Fluent Bit is now considered the next-generation solution.
31
+
In the recent years, cloud providers have switched from Fluentd to Fluent Bit for performance and compatibility. Fluent Bit is now considered the next-generation solution.
In 2014, the [Fluentd](https://www.fluentd.org/) team at [Treasure Data](https://www.treasuredata.com/) was forecasting the need for a lightweight log processor for constraint environments like embedded Linux and gateways. The project aimed to be part of the Fluentd ecosystem. At that moment, Eduardo Silva created [Fluent Bit](https://fluentbit.io/), a new open source solution, written from scratch and available under the terms of the [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0).
15
9
16
-
After the project matured, it gained traction for normal Linux systems. With the
17
-
new containerized world, the Cloud Native community asked to extend the
18
-
project scope to support more sources, filters, and destinations. Not long after,
19
-
Fluent Bit became one of the preferred solutions to solve the logging challenges
20
-
in Cloud environments.
10
+
After the project matured, it gained traction for normal Linux systems. With the new containerized world, the Cloud Native community asked to extend the project scope to support more sources, filters, and destinations. Not long after, Fluent Bit became one of the preferred solutions to solve the logging challenges in Cloud environments.
[Fluent Bit](http://fluentbit.io), including its core, plugins, and tools are distributed under the terms of the [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0):
Copy file name to clipboardExpand all lines: about/what-is-fluent-bit.md
+3-16Lines changed: 3 additions & 16 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,24 +4,11 @@ description: Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
4
4
5
5
# What is Fluent Bit?
6
6
7
-
[Fluent Bit](https://fluentbit.io) is an open source telemetry agent specifically
8
-
designed to efficiently handle the challenges of collecting and processing telemetry
9
-
data across a wide range of environments, from constrained systems to complex cloud
10
-
infrastructures. Managing telemetry data from various sources and formats can be a
11
-
constant challenge, particularly when performance is a critical factor.
7
+
[Fluent Bit](https://fluentbit.io) is an open source telemetry agent specifically designed to efficiently handle the challenges of collecting and processing telemetry data across a wide range of environments, from constrained systems to complex cloud infrastructures. Managing telemetry data from various sources and formats can be a constant challenge, particularly when performance is a critical factor.
12
8
13
-
Rather than serving as a drop-in replacement, Fluent Bit enhances the observability
14
-
strategy for your infrastructure by adapting and optimizing your existing logging
15
-
layer, and adding metrics and traces processing. Fluent Bit supports a
16
-
vendor-neutral approach, seamlessly integrating with other ecosystems such as
17
-
Prometheus and OpenTelemetry. Trusted by major cloud providers, banks, and companies
18
-
in need of a ready-to-use telemetry agent solution, Fluent Bit effectively manages
19
-
diverse data sources and formats while maintaining optimal performance and keeping
20
-
resource consumption low.
9
+
Rather than serving as a drop-in replacement, Fluent Bit enhances the observability strategy for your infrastructure by adapting and optimizing your existing logging layer, and adding metrics and traces processing. Fluent Bit supports a vendor-neutral approach, seamlessly integrating with other ecosystems such as Prometheus and OpenTelemetry. Trusted by major cloud providers, banks, and companies in need of a ready-to-use telemetry agent solution, Fluent Bit effectively manages diverse data sources and formats while maintaining optimal performance and keeping resource consumption low.
21
10
22
-
Fluent Bit can be deployed as an edge agent for localized telemetry data handling or
23
-
utilized as a central aggregator/collector for managing telemetry data across
24
-
multiple sources and environments.
11
+
Fluent Bit can be deployed as an edge agent for localized telemetry data handling or utilized as a central aggregator/collector for managing telemetry data across multiple sources and environments.
Copy file name to clipboardExpand all lines: concepts/buffering.md
+6-21Lines changed: 6 additions & 21 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,29 +4,14 @@ description: Performance and data safety
4
4
5
5
# Buffering
6
6
7
-
When [Fluent Bit](https://fluentbit.io) processes data, it uses the system memory
8
-
(heap) as a primary and temporary place to store the record logs before they get
9
-
delivered. The records are processed in this private memory area.
7
+
When [Fluent Bit](https://fluentbit.io) processes data, it uses the system memory (heap) as a primary and temporary place to store the record logs before they get delivered. The records are processed in this private memory area.
10
8
11
-
Buffering is the ability to store the records, and continue storing incoming data
12
-
while previous data is processed and delivered. Buffering in memory is the fastest
13
-
mechanism, but there are scenarios requiring special strategies to deal with
14
-
[backpressure](../administration/backpressure.md), data safety, or to reduce memory
15
-
consumption by the service in constrained environments.
9
+
Buffering is the ability to store the records, and continue storing incoming data while previous data is processed and delivered. Buffering in memory is the fastest mechanism, but there are scenarios requiring special strategies to deal with [backpressure](../administration/backpressure.md), data safety, or to reduce memory consumption by the service in constrained environments.
16
10
17
-
Network failures or latency in third party service is common. When data can't be
18
-
delivered fast enough and new data to process arrives, the system can face
19
-
backpressure.
11
+
Network failures or latency in third party service is common. When data can't be delivered fast enough and new data to process arrives, the system can face backpressure.
20
12
21
-
Fluent Bit buffering strategies are designed to solve problems associated with
22
-
backpressure and general delivery failures. Fluent Bit offers a primary buffering
23
-
mechanism in memory and an optional secondary one using the file system. With
24
-
this hybrid solution you can accommodate any use case safely and keep a high
25
-
performance while processing your data.
13
+
Fluent Bit buffering strategies are designed to solve problems associated with backpressure and general delivery failures. Fluent Bit offers a primary buffering mechanism in memory and an optional secondary one using the file system. With this hybrid solution you can accommodate any use case safely and keep a high performance while processing your data.
26
14
27
-
These mechanisms aren't mutually exclusive. When data is ready to be processed or
28
-
delivered it's always be in memory, while other data in the queue might be in
29
-
the file system until is ready to be processed and moved up to memory.
15
+
These mechanisms aren't mutually exclusive. When data is ready to be processed or delivered it's always be in memory, while other data in the queue might be in the file system until is ready to be processed and moved up to memory.
30
16
31
-
To learn more about the buffering configuration in Fluent Bit, see
Copy file name to clipboardExpand all lines: concepts/data-pipeline/buffer.md
+3-7Lines changed: 3 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,12 +4,9 @@ description: Data processing with reliability
4
4
5
5
# Buffer
6
6
7
-
The [`buffer`](../buffering.md) phase in the pipeline aims to provide a unified and
8
-
persistent mechanism to store your data, using the primary in-memory model or the
9
-
file system-based mode.
7
+
The [`buffer`](../buffering.md) phase in the pipeline aims to provide a unified and persistent mechanism to store your data, using the primary in-memory model or the file system-based mode.
10
8
11
-
The `buffer` phase contains the data in an immutable state, meaning that no other
12
-
filter can be applied.
9
+
The `buffer` phase contains the data in an immutable state, meaning that no other filter can be applied.
13
10
14
11
```mermaid
15
12
graph LR
@@ -27,5 +24,4 @@ graph LR
27
24
28
25
Buffered data uses the Fluent Bit internal binary representation, which isn't raw text.
29
26
30
-
Fluent Bit offers a buffering mechanism in the file system that acts as a backup
31
-
system to avoid data loss in case of system failures.
27
+
Fluent Bit offers a buffering mechanism in the file system that acts as a backup system to avoid data loss in case of system failures.
Copy file name to clipboardExpand all lines: concepts/data-pipeline/filter.md
+5-10Lines changed: 5 additions & 10 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,8 +4,7 @@ description: Modify, enrich or drop your records
4
4
5
5
# Filter
6
6
7
-
In production environments you need full control of the data you're collecting.
8
-
Filtering lets you alter the collected data before delivering it to a destination.
7
+
In production environments you need full control of the data you're collecting. Filtering lets you alter the collected data before delivering it to a destination.
9
8
10
9
```mermaid
11
10
graph LR
@@ -21,14 +20,10 @@ graph LR
21
20
style C stroke:darkred,stroke-width:2px;
22
21
```
23
22
24
-
Filtering is implemented through plugins. Each available filter can be used to
25
-
match, exclude, or enrich your logs with specific metadata.
23
+
Filtering is implemented through plugins. Each available filter can be used to match, exclude, or enrich your logs with specific metadata.
26
24
27
-
Fluent Bit support many filters. A common use case for filtering is Kubernetes
28
-
deployments. Every pod log needs the proper metadata associated with it.
25
+
Fluent Bit support many filters. A common use case for filtering is Kubernetes deployments. Every pod log needs the proper metadata associated with it.
29
26
30
-
Like input plugins, filters run in an instance context, which has its own independent
31
-
configuration. Configuration keys are often called _properties_.
27
+
Like input plugins, filters run in an instance context, which has its own independent configuration. Configuration keys are often called _properties_.
32
28
33
-
For more details about the Filters available and their usage, see
0 commit comments