-
Notifications
You must be signed in to change notification settings - Fork 540
fluent: docs: updating key concepts for style #1482
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from 5 commits
3e81480
4a1f678
e33f6c6
30b6a95
30d9050
63f3489
5b7e1f1
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change | ||||
|---|---|---|---|---|---|---|
| @@ -1,25 +1,28 @@ | ||||||
| --- | ||||||
| description: >- | ||||||
| There are a few key concepts that are really important to understand how | ||||||
| Fluent Bit operates. | ||||||
| description: Learn these key concepts to understand how Fluent Bit operates. | ||||||
| --- | ||||||
|
|
||||||
| # Key Concepts | ||||||
| # Key concepts | ||||||
|
|
||||||
| Before diving into [Fluent Bit](https://fluentbit.io) it’s good to get acquainted with some of the key concepts of the service. This document provides a gentle introduction to those concepts and common [Fluent Bit](https://fluentbit.io) terminology. We’ve provided a list below of all the terms we’ll cover, but we recommend reading this document from start to finish to gain a more general understanding of our log and stream processor. | ||||||
| Before diving into [Fluent Bit](https://fluentbit.io) you might want to get acquainted | ||||||
| with some of the key concepts of the service. This document provides an | ||||||
| introduction to those concepts and common [Fluent Bit](https://fluentbit.io) | ||||||
| terminology. Reading this document will help you gain a more general understanding of the | ||||||
| log and stream processor. | ||||||
|
|
||||||
| * Event or Record | ||||||
| * Filtering | ||||||
| * Tag | ||||||
| * Timestamp | ||||||
| * Match | ||||||
| * Structured Message | ||||||
| - Event or Record | ||||||
| - Filtering | ||||||
| - Tag | ||||||
| - Timestamp | ||||||
| - Match | ||||||
| - Structured Message | ||||||
|
|
||||||
| ## Event or Record | ||||||
|
|
||||||
| Every incoming piece of data that belongs to a log or a metric that is retrieved by Fluent Bit is considered an Event or a Record. | ||||||
| Every incoming piece of data that belongs to a log or a metric that's retrieved by | ||||||
| Fluent Bit is considered an _Event_ or a _Record_. | ||||||
esmerel marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
|
||||||
| As an example consider the following content of a Syslog file: | ||||||
| As an example, consider the following content of a Syslog file: | ||||||
|
|
||||||
| ```text | ||||||
| Jan 18 12:52:16 flb systemd[2222]: Starting GNOME Terminal Server | ||||||
|
|
@@ -28,30 +31,31 @@ Jan 18 12:52:16 flb systemd[2222]: Started GNOME Terminal Server. | |||||
| Jan 18 12:52:16 flb gsd-media-keys[2640]: # watch_fast: "/org/gnome/terminal/legacy/" (establishing: 0, active: 0) | ||||||
| ``` | ||||||
|
|
||||||
| It contains four lines and all of them represents **four** independent Events. | ||||||
| It contains four lines, representing four independent Events. | ||||||
esmerel marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||||||
|
|
||||||
| Internally an Event is comprised of: | ||||||
| Internally, an Event is comprised of: | ||||||
esmerel marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||||||
|
|
||||||
| * timestamp | ||||||
| * key/value metadata (since v2.1.0) | ||||||
| * payload | ||||||
| - timestamp | ||||||
| - key/value metadata (v2.1.0 and greater) | ||||||
| - payload | ||||||
|
|
||||||
| ### Event format | ||||||
|
|
||||||
| The Fluent Bit wire protocol represents an Event as a 2-element array | ||||||
| with a nested array as the first element: | ||||||
esmerel marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||||||
|
|
||||||
| ```javascript | ||||||
| ```javascript copy | ||||||
| [[TIMESTAMP, METADATA], MESSAGE] | ||||||
| ``` | ||||||
|
|
||||||
| where | ||||||
|
|
||||||
| * TIMESTAMP is a timestamp in seconds as an integer or floating point value (not a string); | ||||||
| * METADATA is a possibly-empty object containing event metadata; and | ||||||
| * MESSAGE is an object containing the event body. | ||||||
| - _`TIMESTAMP`_ is a timestamp in seconds as an integer or floating point value | ||||||
| (not a string). | ||||||
| - _`METADATA`_ is a possibly empty object containing event metadata. | ||||||
esmerel marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||||||
| - _`MESSAGE`_ is an object containing the event body. | ||||||
|
|
||||||
| Fluent Bit versions prior to v2.1.0 instead used: | ||||||
| Fluent Bit versions prior to v2.1.0 used: | ||||||
|
|
||||||
| ```javascript | ||||||
| [TIMESTAMP, MESSAGE] | ||||||
|
|
@@ -62,74 +66,79 @@ streams. | |||||
|
|
||||||
| ## Filtering | ||||||
|
|
||||||
| In some cases it is required to perform modifications on the Events content, the process to alter, enrich or drop Events is called Filtering. | ||||||
| You might need to perform modifications on the Event's content. The process to alter, | ||||||
esmerel marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||||||
| append to, or drop Events is called [_filtering_](data-pipeline/filter.md). | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. append [stuff] to? |
||||||
|
|
||||||
| There are many use cases when Filtering is required like: | ||||||
| Use filtering to: | ||||||
|
|
||||||
| * Append specific information to the Event like an IP address or metadata. | ||||||
| * Select a specific piece of the Event content. | ||||||
| * Drop Events that matches certain pattern. | ||||||
| - Append specific information to the Event like an IP address or metadata. | ||||||
| - Select a specific piece of the Event content. | ||||||
| - Drop Events that match a certain pattern. | ||||||
|
|
||||||
| ## Tag | ||||||
|
|
||||||
| Every Event that gets into Fluent Bit gets assigned a Tag. This tag is an internal string that is used in a later stage by the Router to decide which Filter or Output phase it must go through. | ||||||
| Every Event ingested by Fluent Bit is assigned a Tag. This tag is an internal string | ||||||
esmerel marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
| used in a later stage by the Router to decide which Filter or | ||||||
| [Output](data-pipeline/output.md) phase it must go through. | ||||||
|
|
||||||
| Most of the tags are assigned manually in the configuration. If a tag is not specified, Fluent Bit will assign the name of the Input plugin instance from where that Event was generated from. | ||||||
| Most tags are assigned manually in the configuration. If a tag isn't specified, | ||||||
| Fluent Bit assigns the name of the [Input](data-pipeline/input.md) plugin | ||||||
| instance where that Event was generated from. | ||||||
|
|
||||||
| {% hint style="info" %} | ||||||
| The only input plugin that **does NOT** assign tags is [Forward](../pipeline/inputs/forward.md) input. This plugin speaks the Fluentd wire protocol called Forward where every Event already comes with a Tag associated. Fluent Bit will always use the incoming Tag set by the client. | ||||||
| The [Forward](../pipeline/inputs/forward.md) input plugin doesn't assign tags. This | ||||||
| plugin speaks the Fluentd wire protocol called Forward where every Event already | ||||||
| comes with a Tag associated. Fluent Bit will always use the incoming Tag set by the | ||||||
| client. | ||||||
| {% endhint %} | ||||||
|
|
||||||
| A Tagged record must always have a Matching rule. To learn more about Tags and Matches check the [Routing](data-pipeline/router.md) section. | ||||||
| A tagged record must always have a Matching rule. To learn more about Tags and | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
not sure if a code block is the best approach here but I feel like something is needed because otherwise the capitalization by itself sticks out |
||||||
| Matches, see [Routing](data-pipeline/router.md). | ||||||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
|
||||||
|
|
||||||
| ## Timestamp | ||||||
|
|
||||||
| The Timestamp represents the _time_ when an Event was created. Every Event contains a Timestamp associated. The Timestamp is a numeric fractional integer in the format: | ||||||
| The timestamp represents the time an Event was created. Every Event contains a | ||||||
| timestamp associated. A timestamp always exists, and is set by the Input plugin or | ||||||
| discovered through a data parsing process. | ||||||
esmerel marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||||||
|
|
||||||
| The timestamp is a numeric fractional integer in the format: | ||||||
|
|
||||||
| ```javascript | ||||||
| SECONDS.NANOSECONDS | ||||||
| ``` | ||||||
|
|
||||||
| ### Seconds | ||||||
|
|
||||||
| It is the number of seconds that have elapsed since the _Unix epoch._ | ||||||
|
|
||||||
| ### Nanoseconds | ||||||
| where: | ||||||
|
|
||||||
| Fractional second or one thousand-millionth of a second. | ||||||
|
|
||||||
| {% hint style="info" %} | ||||||
| A timestamp always exists, either set by the Input plugin or discovered through a data parsing process. | ||||||
| {% endhint %} | ||||||
| - `_Seconds_` is the number of seconds that have elapsed since the Unix epoch. | ||||||
| - `_Nanoseconds_` is a fractional second or one thousand-millionth of a second. | ||||||
esmerel marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||||||
|
|
||||||
| ## Match | ||||||
|
|
||||||
| Fluent Bit allows to deliver your collected and processed Events to one or multiple destinations, this is done through a routing phase. A Match represent a simple rule to select Events where it Tags matches a defined rule. | ||||||
| Fluent Bit lets you deliver your collected and processed Events to one or multiple | ||||||
| destinations through a routing phase. A _Match_ represents a rule to select Events | ||||||
esmerel marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||||||
| where a Tag matches a defined rule. | ||||||
|
|
||||||
| To learn more about Tags and Matches check the [Routing](data-pipeline/router.md) section. | ||||||
| To learn more about Tags and Matches, see [Routing](data-pipeline/router.md). | ||||||
|
|
||||||
| ## Structured Messages | ||||||
| ## Structured messages | ||||||
|
|
||||||
| Source events can have or not have a structure. A structure defines a set of _keys_ and _values_ inside the Event message. As an example consider the following two messages: | ||||||
| Source events can have a structure. A structure defines a set of `keys` and `values` | ||||||
| inside the Event message to implement faster operations on data modifications. | ||||||
| Fluent Bit handles every Event message as a structured message. | ||||||
esmerel marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||||||
|
|
||||||
| ### No structured message | ||||||
| Consider the following two messages: | ||||||
|
|
||||||
| ```javascript | ||||||
| "Project Fluent Bit created on 1398289291" | ||||||
| ``` | ||||||
|
|
||||||
| ### Structured Message | ||||||
| - No structured message | ||||||
esmerel marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
|
||||||
| ```javascript | ||||||
| {"project": "Fluent Bit", "created": 1398289291} | ||||||
| ``` | ||||||
| ```javascript | ||||||
| "Project Fluent Bit created on 1398289291" | ||||||
| ``` | ||||||
|
|
||||||
| At a low level both are just an array of bytes, but the Structured message defines _keys_ and _values_, having a structure helps to implement faster operations on data modifications. | ||||||
| - With a structured message | ||||||
esmerel marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
|
||||||
| {% hint style="info" %} | ||||||
| Fluent Bit **always** handles every Event message as a structured message. | ||||||
| For performance reasons, we use a binary serialization data format called [MessagePack](https://msgpack.org/). | ||||||
|
|
||||||
| Consider [MessagePack](https://msgpack.org/) as a binary version of JSON on steroids. | ||||||
| {% endhint %} | ||||||
| ```javascript | ||||||
| {"project": "Fluent Bit", "created": 1398289291} | ||||||
| ``` | ||||||
|
|
||||||
| For performance reasons, Fluent Bit uses a binary serialization data format called | ||||||
| [MessagePack](https://msgpack.org/). | ||||||
Uh oh!
There was an error while loading. Please reload this page.