diff --git a/development/developer-guide.md b/development/developer-guide.md index d21ebb452..d8151d370 100644 --- a/development/developer-guide.md +++ b/development/developer-guide.md @@ -14,10 +14,10 @@ For cryptographic support, Fluent Bit uses the system installed version of OpenS When you write Fluent Bit code, you'll use the Fluent Bit versions of the standard C functions for working with memory: -* [`flb_malloc()`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_mem.h): Equivalent to `malloc`, allocates memory. -* [`flb_calloc()`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_mem.h): Equivalent to `calloc`, allocates memory and initializes it to zero. -* [`flb_realloc()`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_mem.h): Equivalent to `realloc`. -* [`flb_free()`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_mem.h): Equivalent to `free`, releases allocated memory. +- [`flb_malloc()`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_mem.h): Equivalent to `malloc`, allocates memory. +- [`flb_calloc()`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_mem.h): Equivalent to `calloc`, allocates memory and initializes it to zero. +- [`flb_realloc()`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_mem.h): Equivalent to `realloc`. +- [`flb_free()`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_mem.h): Equivalent to `free`, releases allocated memory. {% hint style="info" %} Many types have specialized create and destroy functions, like [`flb_sds_create()` and `flb_sds_destroy()`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_sds.h). @@ -33,9 +33,9 @@ In general, you should use SDS strings in any string processing code. SDS string Fluent Bit has its own network connection library. The key types and functions are defined in the following header files: -* [`flb_upstream.h`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_upstream.h) -* [`flb_http_client.h`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_http_client.h) -* [`flb_io.h`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_io.h) +- [`flb_upstream.h`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_upstream.h) +- [`flb_http_client.h`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_http_client.h) +- [`flb_io.h`](https://github.com/fluent/fluent-bit/blob/master/include/fluent-bit/flb_io.h) The following code demonstrates an HTTP request in Fluent Bit: @@ -318,7 +318,8 @@ The [stdout plugin](https://github.com/fluent/fluent-bit/tree/master/plugins/out Fluent Bit provides a standalone environment for development. Developers who use different operating systems or distributions can develop on a basic, common stack. The development environment provides the required libraries and tools for you. Development environments are provided for: -- [Devcontainer](https://github.com/fluent/fluent-bit/blob/master/DEVELOPER_GUIDE.md#devcontainer) + +- [Development container](https://github.com/fluent/fluent-bit/blob/master/DEVELOPER_GUIDE.md#devcontainer) - [Vagrant](https://github.com/fluent/fluent-bit/blob/master/DEVELOPER_GUIDE.md#vagrant). ## Testing diff --git a/development/library_api.md b/development/library_api.md index ac94466f1..f5a8f16d1 100644 --- a/development/library_api.md +++ b/development/library_api.md @@ -6,13 +6,13 @@ Fluent Bit is written in C and can be used from any C or C++ application. Fluent Bit runs as a service, which means that the exposed API provides interfaces to create and manage contexts, specify inputs and outputs, set configuration parameters, and set routing paths for events or records. A typical usage of this library involves: -* Creating library instance and contexts and setting their properties. -* Enabling input plugins and setting their properties. -* Enabling output plugins and setting their properties. -* Starting the library runtime. -* Optionally ingesting records manually. -* Stopping the library runtime. -* Destroying library instances and contexts. +- Creating library instance and contexts and setting their properties. +- Enabling input plugins and setting their properties. +- Enabling output plugins and setting their properties. +- Starting the library runtime. +- Optionally ingesting records manually. +- Stopping the library runtime. +- Destroying library instances and contexts. ## Data types diff --git a/development/msgpack-format.md b/development/msgpack-format.md index 54bfba643..d61197045 100644 --- a/development/msgpack-format.md +++ b/development/msgpack-format.md @@ -1,15 +1,15 @@ -# Msgpack format +# MessagePack format -Fluent Bit **always** handles every Event message as a structured message using a binary serialization data format called [MessagePack](https://msgpack.org/). +Fluent Bit always handles every Event message as a structured message using a binary serialization data format called [MessagePack](https://msgpack.org/). ## Fluent Bit usage MessagePack is a standard and well-defined format, refer to the official documentation for full details. This section provides an overview of the specific types used by Fluent Bit within the format to help anyone consuming it. -* The data structure used by Fluent Bit is a 2-length [`fixarray`](https://github.com/msgpack/msgpack/blob/master/spec.md#array-format-family) of the timestamp and the data. -* The timestamp comes from [`flb_time_append_to_msgpack`])(https://github.com/fluent/fluent-bit/blob/2138cee8f4878733956d42d82f6dcf95f0aa9339/src/flb_time.c#L197), so it's either a `uint64`, a `float64`, or a [`fixext`](https://github.com/msgpack/msgpack/blob/master/spec.md#ext-format-family) where the 4 MSBs are the seconds (big-endian `uint32`) and 4 LSBs are nanoseconds. -* The data itself is just a [`msgpack` map](https://github.com/msgpack/msgpack/blob/master/spec.md#map-format-family) with the keys as strings. +- The data structure used by Fluent Bit is a 2-length [`fixarray`](https://github.com/msgpack/msgpack/blob/master/spec.md#array-format-family) of the timestamp and the data. +- The timestamp comes from [`flb_time_append_to_msgpack`](https://github.com/fluent/fluent-bit/blob/2138cee8f4878733956d42d82f6dcf95f0aa9339/src/flb_time.c#L197), so it's either a `uint64`, a `float64`, or a [`fixext`](https://github.com/msgpack/msgpack/blob/master/spec.md#ext-format-family) where the 4 MSBs are the seconds (big-endian `uint32`) and 4 LSBs are nanoseconds. +- The data itself is a [`msgpack` map](https://github.com/msgpack/msgpack/blob/master/spec.md#map-format-family) with the keys as strings. ## Example @@ -17,11 +17,10 @@ Set up Fluent Bit to send in `msgpack` format to a specific port. ```bash docker run --rm -it --network=host fluent/fluent-bit /fluent-bit/bin/fluent-bit -i cpu -o tcp://127.0.0.1:5170 -p format=msgpack -v - ``` -We could send this to stdout but as it is a serialized format you would end up with strange output. -As an example we use the [Python msgpack library](https://msgpack.org/#languages) to deal with it: +You could send this to stdout but as it's a serialized format you would end up with strange output. +As an example, use the [Python `msgpack` library](https://msgpack.org/#languages) to handle it: ```python #Python3 diff --git a/development/wasm-filter-plugins.md b/development/wasm-filter-plugins.md index 8cfcbc0b7..84e474068 100644 --- a/development/wasm-filter-plugins.md +++ b/development/wasm-filter-plugins.md @@ -12,7 +12,7 @@ There are no additional requirements to execute Wasm plugins. #### Build `flb-wamrc` (optional) -`flb-wamrc` is a `flb`-prefixed AOT (ahead of time) compiler that's provided from [`wasm-micro-runtime`](https://github.com/bytecodealliance/wasm-micro-runtime). +`flb-wamrc` is a `flb`-prefixed Ahead of Time (AOT) compiler that's provided from [`wasm-micro-runtime`](https://github.com/bytecodealliance/wasm-micro-runtime). For `flb-wamrc` support, you must install the LLVM infrastructure and some additional libraries (`libmlir`, `libPolly`, `libedit`, and `libpfm`). For example: @@ -22,13 +22,13 @@ For `flb-wamrc` support, you must install the LLVM infrastructure and some addit ### For Wasm programs -Currently, Fluent Bit supports the following Wasm tool chains: +Fluent Bit supports the following Wasm tool chains: -* Rust on `wasm32-unknown-unknown` - * rustc 1.62.1 (e092d0b6b 2022-07-16) or later -* [TinyGo](https://github.com/tinygo-org/tinygo) on `wasm32-wasi` - * v0.24.0 or later -* [WASI SDK](https://github.com/WebAssembly/wasi-sdk) 13 or later. +- Rust on `wasm32-unknown-unknown` + - rustc 1.62.1 (`e092d0b6b` 2022-07-16) or later +- [TinyGo](https://github.com/tinygo-org/tinygo) on `wasm32-wasi` + - v0.24.0 or later +- [WASI SDK](https://github.com/WebAssembly/wasi-sdk) 13 or later. ## Get started @@ -44,8 +44,13 @@ To support AOT-compiled Wasm execution as filter plugins, build Fluent Bit with Once compiled, you can see new plugins that handle `wasm`, for example: +```shell +bin/fluent-bit -h +``` + +Which returns results similar to: + ```text -$ bin/fluent-bit -h Usage: fluent-bit [OPTION] Inputs # ... other input plugin stuffs @@ -60,9 +65,9 @@ Filters The Fluent Bit Wasm filter assumes C ABI, also known as `wasm32-unknown-unknown` on Rust target and `wasm32-wasi` on TinyGo target. -### To Install Additional Components +### Install additional components -TinyGo and WASI SDK support Wasm target by default. When using Rust's `wasm32-unknown-unknown` target, you must install `wasm32-unknown-unknown` by using [rustup](https://rustup.rs/). Then, install the target components as follows: +TinyGo and WASI SDK support Wasm target by default. When using Rust's `wasm32-unknown-unknown` target, you must install `wasm32-unknown-unknown` by using [`rustup`](https://rustup.rs/). Then, install the target components as follows: ```text rustup target add wasm32-unknown-unknown @@ -116,7 +121,7 @@ pipeline: function_name: super_awesome_filter # Note: run Fluent Bit from the 'wasm_path' location. accessible_paths: /path/to/fluent-bit - + outputs: - name: stdout match: '*' @@ -152,7 +157,8 @@ For example, one of the sample [Rust Wasm filters](https://github.com/fluent/flu [0] dummy.local: [1666270589.270348000, {"lang"=>"Rust", "message"=>"dummy", "original"=>"{"message":"dummy"}", "tag"=>"dummy.local", "time"=>"2022-10-20T12:56:29.270348000 +0000"}] [0] dummy.local: [1666270590.271107000, {"lang"=>"Rust", "message"=>"dummy", "original"=>"{"message":"dummy"}", "tag"=>"dummy.local", "time"=>"2022-10-20T12:56:30.271107000 +0000"}] ``` -Another example of a Rust Wasm filter is the [flb_filter_iis](https://github.com/kenriortega/flb_filter_iis) filter. + +Another example of a Rust Wasm filter is the [`flb_filter_iis`](https://github.com/kenriortega/flb_filter_iis) filter. This filter takes the [Internet Information Services (IIS)](https://learn.microsoft.com/en-us/iis/manage/provisioning-and-managing-iis/configure-logging-in-iis) [w3c logs](https://learn.microsoft.com/en-us/iis/manage/provisioning-and-managing-iis/configure-logging-in-iis#select-w3c-fields-to-log) (with some custom modifications) and transforms the raw string into a standard Fluent Bit JSON structured record. @@ -172,7 +178,7 @@ pipeline: wasm_path: /plugins/flb_filter_iis_wasm.wasm function_name: flb_filter_log_iis_w3c_custom accessible_paths: . - + outputs: - name: stdout match: 'iis.*' @@ -213,6 +219,7 @@ The output after the filter logic will be: ``` This filter approach offers several advantages inherent to programming languages. For example: + - It can be extended by adding type conversion to fields, such as `sc_bytes`, `cs_bytes`, and `time_taken`. You can use this to validate data results. - It allows for the use of conditions to apply more descriptive filters. For example, you can get only all logs that contain status codes higher than `4xx` or `5xx`. - It can be used to define allow lists and deny lists using a data structure array or a file to store predefined IP addresses. @@ -254,6 +261,7 @@ AOT compiling should generate CPU architecture-dependent objects. If you want to ### Further examples -* [C filter](https://github.com/fluent/fluent-bit/tree/master/examples/filter_wasm_c) -* [Rust Filter](https://github.com/fluent/fluent-bit/tree/master/examples/filter_rust) -* [TinyGo filter](https://github.com/fluent/fluent-bit/tree/master/examples/filter_wasm_go) \ No newline at end of file +- [C filter](https://github.com/fluent/fluent-bit/tree/master/examples/filter_wasm_c) +- [Rust Filter](https://github.com/fluent/fluent-bit/tree/master/examples/filter_rust) +- [TinyGo + filter](https://github.com/fluent/fluent-bit/tree/master/examples/filter_wasm_go) diff --git a/development/wasm-input-plugins.md b/development/wasm-input-plugins.md index 7abb60709..b838bd2b7 100644 --- a/development/wasm-input-plugins.md +++ b/development/wasm-input-plugins.md @@ -12,13 +12,13 @@ There are no additional requirements to execute Wasm plugins. ### For Wasm programs -Fluent Bit supports the following Wasm toolchains: +Fluent Bit supports the following Wasm tool chains: -* Rust on `wasm32-unknown-unknown` - * rustc 1.62.1 (e092d0b6b 2022-07-16) or later -* [TinyGo](https://github.com/tinygo-org/tinygo) on `wasm32-wasi` - * v0.24.0 or later -* [WASI SDK](https://github.com/WebAssembly/wasi-sdk) 13 or later +- Rust on `wasm32-unknown-unknown` + - rustc 1.62.1 (`e092d0b6b` 2022-07-16) or later +- [TinyGo](https://github.com/tinygo-org/tinygo) on `wasm32-wasi` + - v0.24.0 or later +- [WASI SDK](https://github.com/WebAssembly/wasi-sdk) 13 or later ## Get started @@ -50,7 +50,7 @@ Wasm input in Fluent Bit assumes WASI ABI, also known as `wasm32-wasi` on Rust t ### Install additional components -TinyGo and WASI SDK support Wasm target by default. When using Rust's `wasm32-wasi` target, you must install `wasm32-wasi` by using [rustup](https://rustup.rs/). Then, install the target components as: +TinyGo and WASI SDK support Wasm target by default. When using Rust's `wasm32-wasi` target, you must install `wasm32-wasi` by using [`rustup`](https://rustup.rs/). Then, install the target components as: ```text rustup target add wasm32-wasi @@ -74,10 +74,10 @@ pipeline: tag: exec.wasi.local wasi_path: /path/to/wasi_built_json.wasm # For security reasons, WASM/WASI program cannot access its outer world - # without accessible permissions. Uncomment below 'accessible_paths' and + # without accessible permissions. Uncomment below 'accessible_paths' and # run Fluent Bit from the 'wasi_path' location: # accessible_paths /path/to/fluent-bit - + outputs: - name: stdout match: '*' @@ -92,7 +92,7 @@ pipeline: Tag exec.wasi.local WASI_Path /path/to/wasi_built_json.wasm # For security reasons, WASM/WASI program cannot access its outer world - # without accessible permissions. Uncomment below 'accessible_paths' and + # without accessible permissions. Uncomment below 'accessible_paths' and # run Fluent Bit from the 'wasi_path' location: # accessible_paths /path/to/fluent-bit @@ -104,4 +104,5 @@ pipeline: {% endtab %} {% endtabs %} -For an example that handles structured logs, see the [Rust `serde-json` example](https://github.com/fluent/fluent-bit/tree/master/examples/wasi_serde_json). \ No newline at end of file +For an example that handles structured logs, see the [Rust `serde-json` +example](https://github.com/fluent/fluent-bit/tree/master/examples/wasi_serde_json). diff --git a/vale-styles/FluentBit/Headings.yml b/vale-styles/FluentBit/Headings.yml index 9ca3b1305..de8b40021 100644 --- a/vale-styles/FluentBit/Headings.yml +++ b/vale-styles/FluentBit/Headings.yml @@ -54,6 +54,7 @@ exceptions: - GCP - GitLab - GitHub + - Go - Google - Google Cloud - Google Cloud BigQuery @@ -82,6 +83,7 @@ exceptions: - LTSV - macOS - Marketplace + - MessagePack - Microsoft Fabric - MongoDB - NATS @@ -119,5 +121,6 @@ exceptions: - Visual - Vivo Exporter - VS + - Wasm - WebSocket - Windows