Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 5 additions & 5 deletions SUMMARY.md
Original file line number Diff line number Diff line change
@@ -1,20 +1,20 @@
# Table of contents

* [Fluent Bit Documentation](README.md)
* [Fluent Bit documentation](README.md)

## About

* [What is Fluent Bit?](about/what-is-fluent-bit.md)
* [A Brief History of Fluent Bit](about/history.md)
* [A brief history of Fluent Bit](about/history.md)
* [Fluentd and Fluent Bit](about/fluentd-and-fluent-bit.md)
* [License](about/license.md)
* [Sandbox and Lab Resources](about/sandbox-and-lab-resources.md)
* [Sandbox and lab resources](about/sandbox-and-lab-resources.md)

## Concepts

* [Key Concepts](concepts/key-concepts.md)
* [Key concepts](concepts/key-concepts.md)
* [Buffering](concepts/buffering.md)
* [Data Pipeline](concepts/data-pipeline/README.md)
* [Data pipeline](concepts/data-pipeline/README.md)
* [Input](concepts/data-pipeline/input.md)
* [Parser](concepts/data-pipeline/parser.md)
* [Filter](concepts/data-pipeline/filter.md)
Expand Down
28 changes: 7 additions & 21 deletions about/fluentd-and-fluent-bit.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,42 +4,28 @@ description: The production grade telemetry ecosystem

# Fluentd and Fluent Bit

Telemetry data processing can be complex, especially at scale. That's why
[Fluentd](https://www.fluentd.org) was created. Fluentd is more than a simple tool,
it's grown into a fullscale ecosystem that contains SDKs for different languages
and subprojects like [Fluent Bit](https://fluentbit.io).
Telemetry data processing can be complex, especially at scale. That's why [Fluentd](https://www.fluentd.org) was created. Fluentd is more than a simple tool, it's grown into a fullscale ecosystem that contains SDKs for different languages and subprojects like [Fluent Bit](https://fluentbit.io).

Here, we describe the relationship between the [Fluentd](http://fluentd.org)
and [Fluent Bit](http://fluentbit.io) open source projects.
Here, we describe the relationship between the [Fluentd](http://fluentd.org) and [Fluent Bit](http://fluentbit.io) open source projects.

Both projects are:

- Licensed under the terms of Apache License v2.0.
- Graduated hosted projects by the [Cloud Native Computing Foundation (CNCF)](https://cncf.io).
- Production grade solutions: Deployed millions of times every single day.
- Vendor neutral and community driven.
- Widely adopted by the industry: Trusted by major companies like AWS, Microsoft,
Google Cloud, and hundreds of others.
- Licensed under the terms of Apache License v2.0. - Graduated hosted projects by the [Cloud Native Computing Foundation (CNCF)](https://cncf.io). - Production grade solutions: Deployed millions of times every single day. - Vendor neutral and community driven. - Widely adopted by the industry: Trusted by major companies like AWS, Microsoft, Google Cloud, and hundreds of others.

The projects have many similarities: [Fluent Bit](https://fluentbit.io) is
designed and built on top of the best ideas of [Fluentd](https://www.fluentd.org)
architecture and general design. Which one you choose depends on your end-users' needs.
The projects have many similarities: [Fluent Bit](https://fluentbit.io) is designed and built on top of the best ideas of [Fluentd](https://www.fluentd.org) architecture and general design. Which one you choose depends on your end-users' needs.

The following table describes a comparison of different areas of the projects:

| Attribute | Fluentd | Fluent Bit |
| ------------ | --------------------- | --------------------- |
| Scope | Containers / Servers | Embedded Linux / Containers / Servers |
| Language | C & Ruby | C |
| Language | C and Ruby | C |
| Memory | Greater than 60 MB | Approximately 1 MB |
| Performance | Medium Performance | High Performance |
| Dependencies | Built as a Ruby Gem, depends on other gems. | Zero dependencies, unless required by a plugin. |
| Plugins | Over 1,000 external plugins available. | Over 100 built-in plugins available. |
| License | [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0) | [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0) |

Both [Fluentd](https://www.fluentd.org) and [Fluent Bit](https://fluentbit.io)
can work as Aggregators or Forwarders, and can complement each other or be used
as standalone solutions.
Both [Fluentd](https://www.fluentd.org) and [Fluent Bit](https://fluentbit.io) can work as Aggregators or Forwarders, and can complement each other or be used as standalone solutions.

In the recent years, cloud providers have switched from Fluentd to Fluent Bit for
performance and compatibility. Fluent Bit is now considered the next-generation solution.
In the recent years, cloud providers have switched from Fluentd to Fluent Bit for performance and compatibility. Fluent Bit is now considered the next-generation solution.
14 changes: 2 additions & 12 deletions about/history.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,16 +5,6 @@ description: Every project has a story

# A brief history of Fluent Bit

In 2014, the [Fluentd](https://www.fluentd.org/) team at
[Treasure Data](https://www.treasuredata.com/) was forecasting the need for a
lightweight log processor for constraint environments like embedded Linux and
gateways. The project aimed to be part of the Fluentd ecosystem. At that moment,
Eduardo Silva created [Fluent Bit](https://fluentbit.io/), a new open source solution,
written from scratch and available under the terms of the
[Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0).
In 2014, the [Fluentd](https://www.fluentd.org/) team at [Treasure Data](https://www.treasuredata.com/) was forecasting the need for a lightweight log processor for constraint environments like embedded Linux and gateways. The project aimed to be part of the Fluentd ecosystem. At that moment, Eduardo Silva created [Fluent Bit](https://fluentbit.io/), a new open source solution, written from scratch and available under the terms of the [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0).

After the project matured, it gained traction for normal Linux systems. With the
new containerized world, the Cloud Native community asked to extend the
project scope to support more sources, filters, and destinations. Not long after,
Fluent Bit became one of the preferred solutions to solve the logging challenges
in Cloud environments.
After the project matured, it gained traction for normal Linux systems. With the new containerized world, the Cloud Native community asked to extend the project scope to support more sources, filters, and destinations. Not long after, Fluent Bit became one of the preferred solutions to solve the logging challenges in Cloud environments.
4 changes: 1 addition & 3 deletions about/license.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,7 @@ description: Fluent Bit license description

# License

[Fluent Bit](http://fluentbit.io), including its core, plugins, and tools are
distributed under the terms of the
[Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0):
[Fluent Bit](http://fluentbit.io), including its core, plugins, and tools are distributed under the terms of the [Apache License v2.0](http://www.apache.org/licenses/LICENSE-2.0):

```text
Apache License
Expand Down
2 changes: 1 addition & 1 deletion about/sandbox-and-lab-resources.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ description: >-
Labs for learning how to best operate, use, and have success with Fluent Bit.
---

# Sandbox and Lab Resources
# Sandbox and lab resources

## Fluent Bit Sandbox - sign-up required

Expand Down
19 changes: 3 additions & 16 deletions about/what-is-fluent-bit.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,24 +4,11 @@ description: Fluent Bit is a CNCF sub-project under the umbrella of Fluentd

# What is Fluent Bit?

[Fluent Bit](https://fluentbit.io) is an open source telemetry agent specifically
designed to efficiently handle the challenges of collecting and processing telemetry
data across a wide range of environments, from constrained systems to complex cloud
infrastructures. Managing telemetry data from various sources and formats can be a
constant challenge, particularly when performance is a critical factor.
[Fluent Bit](https://fluentbit.io) is an open source telemetry agent specifically designed to efficiently handle the challenges of collecting and processing telemetry data across a wide range of environments, from constrained systems to complex cloud infrastructures. Managing telemetry data from various sources and formats can be a constant challenge, particularly when performance is a critical factor.

Rather than serving as a drop-in replacement, Fluent Bit enhances the observability
strategy for your infrastructure by adapting and optimizing your existing logging
layer, and adding metrics and traces processing. Fluent Bit supports a
vendor-neutral approach, seamlessly integrating with other ecosystems such as
Prometheus and OpenTelemetry. Trusted by major cloud providers, banks, and companies
in need of a ready-to-use telemetry agent solution, Fluent Bit effectively manages
diverse data sources and formats while maintaining optimal performance and keeping
resource consumption low.
Rather than serving as a drop-in replacement, Fluent Bit enhances the observability strategy for your infrastructure by adapting and optimizing your existing logging layer, and adding metrics and traces processing. Fluent Bit supports a vendor-neutral approach, seamlessly integrating with other ecosystems such as Prometheus and OpenTelemetry. Trusted by major cloud providers, banks, and companies in need of a ready-to-use telemetry agent solution, Fluent Bit effectively manages diverse data sources and formats while maintaining optimal performance and keeping resource consumption low.

Fluent Bit can be deployed as an edge agent for localized telemetry data handling or
utilized as a central aggregator/collector for managing telemetry data across
multiple sources and environments.
Fluent Bit can be deployed as an edge agent for localized telemetry data handling or utilized as a central aggregator/collector for managing telemetry data across multiple sources and environments.


{% embed url="https://www.youtube.com/watch?v=3ELc1helke4" %}
27 changes: 6 additions & 21 deletions concepts/buffering.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,29 +4,14 @@ description: Performance and data safety

# Buffering

When [Fluent Bit](https://fluentbit.io) processes data, it uses the system memory
(heap) as a primary and temporary place to store the record logs before they get
delivered. The records are processed in this private memory area.
When [Fluent Bit](https://fluentbit.io) processes data, it uses the system memory (heap) as a primary and temporary place to store the record logs before they get delivered. The records are processed in this private memory area.

Buffering is the ability to store the records, and continue storing incoming data
while previous data is processed and delivered. Buffering in memory is the fastest
mechanism, but there are scenarios requiring special strategies to deal with
[backpressure](../administration/backpressure.md), data safety, or to reduce memory
consumption by the service in constrained environments.
Buffering is the ability to store the records, and continue storing incoming data while previous data is processed and delivered. Buffering in memory is the fastest mechanism, but there are scenarios requiring special strategies to deal with [backpressure](../administration/backpressure.md), data safety, or to reduce memory consumption by the service in constrained environments.

Network failures or latency in third party service is common. When data can't be
delivered fast enough and new data to process arrives, the system can face
backpressure.
Network failures or latency in third party service is common. When data can't be delivered fast enough and new data to process arrives, the system can face backpressure.

Fluent Bit buffering strategies are designed to solve problems associated with
backpressure and general delivery failures. Fluent Bit offers a primary buffering
mechanism in memory and an optional secondary one using the file system. With
this hybrid solution you can accommodate any use case safely and keep a high
performance while processing your data.
Fluent Bit buffering strategies are designed to solve problems associated with backpressure and general delivery failures. Fluent Bit offers a primary buffering mechanism in memory and an optional secondary one using the file system. With this hybrid solution you can accommodate any use case safely and keep a high performance while processing your data.

These mechanisms aren't mutually exclusive. When data is ready to be processed or
delivered it's always be in memory, while other data in the queue might be in
the file system until is ready to be processed and moved up to memory.
These mechanisms aren't mutually exclusive. When data is ready to be processed or delivered it's always be in memory, while other data in the queue might be in the file system until is ready to be processed and moved up to memory.

To learn more about the buffering configuration in Fluent Bit, see
[Buffering & Storage](../administration/buffering-and-storage.md).
To learn more about the buffering configuration in Fluent Bit, see [Buffering & Storage](../administration/buffering-and-storage.md).
3 changes: 1 addition & 2 deletions concepts/data-pipeline/README.md
Original file line number Diff line number Diff line change
@@ -1,2 +1 @@
# Data Pipeline

# Data pipeline
10 changes: 3 additions & 7 deletions concepts/data-pipeline/buffer.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,9 @@ description: Data processing with reliability

# Buffer

The [`buffer`](../buffering.md) phase in the pipeline aims to provide a unified and
persistent mechanism to store your data, using the primary in-memory model or the
file system-based mode.
The [`buffer`](../buffering.md) phase in the pipeline aims to provide a unified and persistent mechanism to store your data, using the primary in-memory model or the file system-based mode.

The `buffer` phase contains the data in an immutable state, meaning that no other
filter can be applied.
The `buffer` phase contains the data in an immutable state, meaning that no other filter can be applied.

```mermaid
graph LR
Expand All @@ -27,5 +24,4 @@ graph LR

Buffered data uses the Fluent Bit internal binary representation, which isn't raw text.

Fluent Bit offers a buffering mechanism in the file system that acts as a backup
system to avoid data loss in case of system failures.
Fluent Bit offers a buffering mechanism in the file system that acts as a backup system to avoid data loss in case of system failures.
15 changes: 5 additions & 10 deletions concepts/data-pipeline/filter.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,7 @@ description: Modify, enrich or drop your records

# Filter

In production environments you need full control of the data you're collecting.
Filtering lets you alter the collected data before delivering it to a destination.
In production environments you need full control of the data you're collecting. Filtering lets you alter the collected data before delivering it to a destination.

```mermaid
graph LR
Expand All @@ -21,14 +20,10 @@ graph LR
style C stroke:darkred,stroke-width:2px;
```

Filtering is implemented through plugins. Each available filter can be used to
match, exclude, or enrich your logs with specific metadata.
Filtering is implemented through plugins. Each available filter can be used to match, exclude, or enrich your logs with specific metadata.

Fluent Bit support many filters. A common use case for filtering is Kubernetes
deployments. Every pod log needs the proper metadata associated with it.
Fluent Bit support many filters. A common use case for filtering is Kubernetes deployments. Every pod log needs the proper metadata associated with it.

Like input plugins, filters run in an instance context, which has its own independent
configuration. Configuration keys are often called _properties_.
Like input plugins, filters run in an instance context, which has its own independent configuration. Configuration keys are often called _properties_.

For more details about the Filters available and their usage, see
[Filters](https://docs.fluentbit.io/manual/pipeline/filters).
For more details about the Filters available and their usage, see [Filters](https://docs.fluentbit.io/manual/pipeline/filters).
11 changes: 3 additions & 8 deletions concepts/data-pipeline/input.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,7 @@ description: The way to gather data from your sources

# Input

[Fluent Bit](http://fluentbit.io) provides input plugins to gather information from
different sources. Some plugins collect data from log files, while others can
gather metrics information from the operating system. There are many plugins to suit
different needs.
[Fluent Bit](http://fluentbit.io) provides input plugins to gather information from different sources. Some plugins collect data from log files, while others can gather metrics information from the operating system. There are many plugins to suit different needs.

```mermaid
graph LR
Expand All @@ -23,10 +20,8 @@ graph LR
style A stroke:darkred,stroke-width:2px;
```

When an input plugin loads, an internal _instance_ is created. Each instance has its
own independent configuration. Configuration keys are often called _properties_.
When an input plugin loads, an internal _instance_ is created. Each instance has its own independent configuration. Configuration keys are often called _properties_.

Every input plugin has its own documentation section that specifies how to use it
and what properties are available.
Every input plugin has its own documentation section that specifies how to use it and what properties are available.

For more details, see [Input Plugins](https://docs.fluentbit.io/manual/pipeline/inputs).
8 changes: 2 additions & 6 deletions concepts/data-pipeline/output.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,7 @@ description: Learn about destinations for your data, such as databases and cloud

# Output

The output interface lets you define destinations for your data. Common destinations
are remote services, local file systems, or other standard interfaces. Outputs are
implemented as plugins.
The output interface lets you define destinations for your data. Common destinations are remote services, local file systems, or other standard interfaces. Outputs are implemented as plugins.

```mermaid
graph LR
Expand All @@ -24,9 +22,7 @@ graph LR
style H stroke:darkred,stroke-width:2px;
```

When an output plugin is loaded, an internal _instance_ is created. Every instance
has its own independent configuration. Configuration keys are often called
_properties_.
When an output plugin is loaded, an internal _instance_ is created. Every instance has its own independent configuration. Configuration keys are often called _properties_.

Every output plugin has its own documentation section specifying how it can be used and what properties are available.

Expand Down
16 changes: 4 additions & 12 deletions concepts/data-pipeline/parser.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,9 +4,7 @@ description: Convert unstructured messages to structured messages

# Parser

Dealing with raw strings or unstructured messages is difficult. Having a structure
makes data more usable. Set a structure to the incoming data by using input
plugins as data is collected:
Dealing with raw strings or unstructured messages is difficult. Having a structure makes data more usable. Set a structure to the incoming data by using input plugins as data is collected:

```mermaid
graph LR
Expand All @@ -22,17 +20,13 @@ graph LR
style B stroke:darkred,stroke-width:2px;
```

The parser converts unstructured data to structured data. As an example, consider the
following Apache (HTTP Server) log entry:
The parser converts unstructured data to structured data. As an example, consider the following Apache (HTTP Server) log entry:

```text
192.168.2.20 - - [28/Jul/2006:10:27:10 -0300] "GET /cgi-bin/try/ HTTP/1.0" 200 3395
```

This log line is a raw string without format. Structuring the log makes it easier
to process the data later. If the
[regular expression parser](pipeline/parsers/regular-expression) is used, the log
entry could be converted to:
This log line is a raw string without format. Structuring the log makes it easier to process the data later. If the [regular expression parser](pipeline/parsers/regular-expression) is used, the log entry could be converted to:

```javascript
{
Expand All @@ -47,6 +41,4 @@ entry could be converted to:
}
```

Parsers are fully configurable and are independently and optionally handled by each
input plugin. For more details, see
[Parsers](https://docs.fluentbit.io/manual/pipeline/parsers).
Parsers are fully configurable and are independently and optionally handled by each input plugin. For more details, see [Parsers](https://docs.fluentbit.io/manual/pipeline/parsers).
Loading