diff --git a/concepts/key-concepts.md b/concepts/key-concepts.md index c3b70801a..fc9841f30 100644 --- a/concepts/key-concepts.md +++ b/concepts/key-concepts.md @@ -1,25 +1,28 @@ --- -description: >- - There are a few key concepts that are really important to understand how - Fluent Bit operates. +description: Learn these key concepts to understand how Fluent Bit operates. --- -# Key Concepts +# Key concepts -Before diving into [Fluent Bit](https://fluentbit.io) it’s good to get acquainted with some of the key concepts of the service. This document provides a gentle introduction to those concepts and common [Fluent Bit](https://fluentbit.io) terminology. We’ve provided a list below of all the terms we’ll cover, but we recommend reading this document from start to finish to gain a more general understanding of our log and stream processor. +Before diving into [Fluent Bit](https://fluentbit.io) you might want to get acquainted +with some of the key concepts of the service. This document provides an +introduction to those concepts and common [Fluent Bit](https://fluentbit.io) +terminology. Reading this document will help you gain a more general understanding of the +following topics: -* Event or Record -* Filtering -* Tag -* Timestamp -* Match -* Structured Message +- Event or Record +- Filtering +- Tag +- Timestamp +- Match +- Structured Message ## Event or Record -Every incoming piece of data that belongs to a log or a metric that is retrieved by Fluent Bit is considered an Event or a Record. +Every incoming piece of data that belongs to a log or a metric that's retrieved by +Fluent Bit is considered an _Event_ or a _Record_. -As an example consider the following content of a Syslog file: +As an example, consider the following content of a Syslog file: ```text Jan 18 12:52:16 flb systemd[2222]: Starting GNOME Terminal Server @@ -28,30 +31,31 @@ Jan 18 12:52:16 flb systemd[2222]: Started GNOME Terminal Server. Jan 18 12:52:16 flb gsd-media-keys[2640]: # watch_fast: "/org/gnome/terminal/legacy/" (establishing: 0, active: 0) ``` -It contains four lines and all of them represents **four** independent Events. +It contains four lines that represent four independent Events. -Internally an Event is comprised of: +An Event is comprised of: -* timestamp -* key/value metadata (since v2.1.0) -* payload +- timestamp +- key/value metadata (v2.1.0 and greater) +- payload ### Event format -The Fluent Bit wire protocol represents an Event as a 2-element array +The Fluent Bit wire protocol represents an Event as a two-element array with a nested array as the first element: -```javascript +```javascript copy [[TIMESTAMP, METADATA], MESSAGE] ``` where -* TIMESTAMP is a timestamp in seconds as an integer or floating point value (not a string); -* METADATA is a possibly-empty object containing event metadata; and -* MESSAGE is an object containing the event body. +- _`TIMESTAMP`_ is a timestamp in seconds as an integer or floating point value + (not a string). +- _`METADATA`_ is an object containing event metadata, and might be empty. +- _`MESSAGE`_ is an object containing the event body. -Fluent Bit versions prior to v2.1.0 instead used: +Fluent Bit versions prior to v2.1.0 used: ```javascript [TIMESTAMP, MESSAGE] @@ -62,74 +66,79 @@ streams. ## Filtering -In some cases it is required to perform modifications on the Events content, the process to alter, enrich or drop Events is called Filtering. +You might need to perform modifications on an Event's content. The process to alter, +append to, or drop Events is called [_filtering_](data-pipeline/filter.md). -There are many use cases when Filtering is required like: +Use filtering to: -* Append specific information to the Event like an IP address or metadata. -* Select a specific piece of the Event content. -* Drop Events that matches certain pattern. +- Append specific information to the Event like an IP address or metadata. +- Select a specific piece of the Event content. +- Drop Events that match a certain pattern. ## Tag -Every Event that gets into Fluent Bit gets assigned a Tag. This tag is an internal string that is used in a later stage by the Router to decide which Filter or Output phase it must go through. +Every Event ingested by Fluent Bit is assigned a Tag. This tag is an internal string +used in a later stage by the Router to decide which Filter or +[Output](data-pipeline/output.md) phase it must go through. -Most of the tags are assigned manually in the configuration. If a tag is not specified, Fluent Bit will assign the name of the Input plugin instance from where that Event was generated from. +Most tags are assigned manually in the configuration. If a tag isn't specified, +Fluent Bit assigns the name of the [Input](data-pipeline/input.md) plugin +instance where that Event was generated from. {% hint style="info" %} -The only input plugin that **does NOT** assign tags is [Forward](../pipeline/inputs/forward.md) input. This plugin speaks the Fluentd wire protocol called Forward where every Event already comes with a Tag associated. Fluent Bit will always use the incoming Tag set by the client. +The [Forward](../pipeline/inputs/forward.md) input plugin doesn't assign tags. This +plugin speaks the Fluentd wire protocol called Forward where every Event already +comes with a Tag associated. Fluent Bit will always use the incoming Tag set by the +client. {% endhint %} -A Tagged record must always have a Matching rule. To learn more about Tags and Matches check the [Routing](data-pipeline/router.md) section. +A tagged record must always have a Matching rule. To learn more about Tags and +Matches, see [Routing](data-pipeline/router.md). ## Timestamp -The Timestamp represents the _time_ when an Event was created. Every Event contains a Timestamp associated. The Timestamp is a numeric fractional integer in the format: +The timestamp represents the time an Event was created. Every Event contains an +associated timestamps. All events have timestamps, and they're set by the input plugin or +discovered through a data parsing process. + +The timestamp is a numeric fractional integer in the format: ```javascript SECONDS.NANOSECONDS ``` -### Seconds - -It is the number of seconds that have elapsed since the _Unix epoch._ - -### Nanoseconds +where: -Fractional second or one thousand-millionth of a second. - -{% hint style="info" %} -A timestamp always exists, either set by the Input plugin or discovered through a data parsing process. -{% endhint %} +- `_SECONDS_` is the number of seconds that have elapsed since the Unix epoch. +- `_NANOSECONDS_` is a fractional second or one thousand-millionth of a second. ## Match -Fluent Bit allows to deliver your collected and processed Events to one or multiple destinations, this is done through a routing phase. A Match represent a simple rule to select Events where it Tags matches a defined rule. +Fluent Bit lets you route your collected and processed Events to one or multiple +destinations. A _Match_ represents a rule to select Events +where a Tag matches a defined rule. -To learn more about Tags and Matches check the [Routing](data-pipeline/router.md) section. +To learn more about Tags and Matches, see [Routing](data-pipeline/router.md). -## Structured Messages +## Structured messages -Source events can have or not have a structure. A structure defines a set of _keys_ and _values_ inside the Event message. As an example consider the following two messages: +Source events can have a structure. A structure defines a set of `keys` and `values` +inside the Event message to implement faster operations on data modifications. +Fluent Bit treats every Event message as a structured message. -### No structured message +Consider the following two messages: -```javascript -"Project Fluent Bit created on 1398289291" -``` - -### Structured Message +- No structured message -```javascript -{"project": "Fluent Bit", "created": 1398289291} -``` + ```javascript + "Project Fluent Bit created on 1398289291" + ``` -At a low level both are just an array of bytes, but the Structured message defines _keys_ and _values_, having a structure helps to implement faster operations on data modifications. +- With a structured message -{% hint style="info" %} -Fluent Bit **always** handles every Event message as a structured message. -For performance reasons, we use a binary serialization data format called [MessagePack](https://msgpack.org/). - -Consider [MessagePack](https://msgpack.org/) as a binary version of JSON on steroids. -{% endhint %} + ```javascript + {"project": "Fluent Bit", "created": 1398289291} + ``` +For performance reasons, Fluent Bit uses a binary serialization data format called +[MessagePack](https://msgpack.org/).