Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
102 changes: 48 additions & 54 deletions development/wasm-filter-plugins.md
Original file line number Diff line number Diff line change
@@ -1,51 +1,48 @@
# WASM Filter Plugins
# Wasm filter plugins

[WebAssembly](https://webassembly.org/) is binary instruction format for stack based virtual machine.
[WebAssembly](https://webassembly.org/) (Wasm) is a binary instruction format for stack-based virtual machines.

Fluent Bit currently supports integration of wasm plugins built as wasm/wasi objects for input and filter plugins only.
The interface for the WASM filter plugins is currently under development but is functional.
Fluent Bit supports integration of Wasm plugins built as Wasm/WASI objects for input and filter plugins only. The interface for Wasm filter plugins is currently under development but is functional.

## Prerequisites

### Building Fluent Bit
### Build Fluent Bit

There are no additional requirements to execute WASM plugins.
There are no additional requirements to execute Wasm plugins.

#### Building flb-wamrc (optional)
#### Build `flb-wamrc` (optional)

`flb-wamrc` is just `flb-` prefixed AOT (Ahead Of Time) compiler that is provided from [wasm-micro-runtime](https://github.com/bytecodealliance/wasm-micro-runtime).
`flb-wamrc` is a `flb`-prefixed AOT (ahead of time) compiler that's provided from [`wasm-micro-runtime`](https://github.com/bytecodealliance/wasm-micro-runtime).

For `flb-wamrc` support, users have to install llvm infrastructure and some additional libraries (`libmlir`, `libPolly`, `libedit`, and `libpfm`), e.g:
For `flb-wamrc` support, you must install the LLVM infrastructure and some additional libraries (`libmlir`, `libPolly`, `libedit`, and `libpfm`). For example:

```text
# apt install -y llvm libmlir-14-dev libclang-common-14-dev libedit-dev libpfm4-dev
```

### For Build WASM programs
### For Wasm programs

Currently, Fluent Bit supports the following WASM toolchains:
Currently, Fluent Bit supports the following Wasm tool chains:

* Rust on `wasm32-unknown-unknown`.
* Rust on `wasm32-unknown-unknown`
* rustc 1.62.1 (e092d0b6b 2022-07-16) or later
* [TinyGo](https://github.com/tinygo-org/tinygo) on wasm32-wasi
* [TinyGo](https://github.com/tinygo-org/tinygo) on `wasm32-wasi`
* v0.24.0 or later
* [WASI SDK](https://github.com/WebAssembly/wasi-sdk) 13 or later.

## Getting Started
## Get started

As described in general options in [source installation](../installation/sources/build-and-install.md),
WASM support is enabled by default.
Compile Fluent Bit with WASM support, e.g:
As described in the general options in the [source installation](../installation/sources/build-and-install.md) guide, Wasm support is enabled by default. Compile Fluent Bit with Wasm support, for example:

```text
$ cd build/
$ cmake .. [-DFLB_WAMRC=On]
$ make
```

To support AOT compiled WASM execution as filter plugins, users have to built Fluent Bit with `-DFLB_WAMRC=On`.
To support AOT-compiled Wasm execution as filter plugins, build Fluent Bit with `-DFLB_WAMRC=On`.

Once compiled, we can see new plugins in which handles _wasm_, e.g:
Once compiled, you can see new plugins that handle `wasm`, for example:

```text
$ bin/fluent-bit -h
Expand All @@ -59,31 +56,30 @@ Filters
wasm WASM program filter
```

## Build a WASM Filter for Filter Plugin
## Build a Wasm filter for filter plugin

Currently, Fluent Bit's WASM filter assumes C ABI that is also known as `wasm32-unknown-unknown` on Rust target and `wasm32-wasi` on TinyGo target.
The Fluent Bit Wasm filter assumes C ABI, also known as `wasm32-unknown-unknown` on Rust target and `wasm32-wasi` on TinyGo target.

### To Install Additional Components

TinyGo and WASI SDK support wasm target by default.
When using Rust's `wasm32-unknown-unknown` target, users must install `wasm32-unknown-unknown` via [rustup](https://rustup.rs/). Then, installing that target components as:
TinyGo and WASI SDK support Wasm target by default. When using Rust's `wasm32-unknown-unknown` target, you must install `wasm32-unknown-unknown` by using [rustup](https://rustup.rs/). Then, install the target components as follows:

```text
$ rustup target add wasm32-unknown-unknown
```

### Requirements of WASM programs
### Requirements of Wasm programs

WASM filter plugins execute the function that has the following signagure.
Wasm filter plugins execute the function that has the following signature.

For C:

```text
// We can use an arbitrary function name for filter operations w/ WASM.
// We can use an arbitrary function name for filter operations w/ Wasm.
char* c_filter(char*, int, uint32_t, uint32_t, char*, int);
```

For Go(TinyGo):
For Go (TinyGo):

```text
//export go_filter
Expand All @@ -95,14 +91,14 @@ For Rust:

```text
// #[no_mangle] attribute is needed for preventing mangles and align C ABI.
// Also we can use an arbitrary function name for filter operations w/ WASM.
// Also we can use an arbitrary function name for filter operations w/ Wasm.
#[no_mangle]
pub extern "C" fn rust_filter(tag: *const c_char, tag_len: u32, time_sec: u32, time_nsec: u32, record: *const c_char, record_len: u32)
```

Note that `//export XXX` on TinyGo and `#[no_mangle]` attributes on Rust are required. This is because TinyGo and Rust will mangle their function names if they are not specified.
The `//export XXX` attribute on TinyGo and `#[no_mangle]` attribute on Rust are required. This is because TinyGo and Rust will mangle their function names if they aren't specified.

Once built, a WASM program will be available. Then, that built program can be executed with the following Fluent Bit configuration:
Once built, a Wasm program will be available. Then you can execute that built program with the following Fluent Bit configuration:

```text
[INPUT]
Expand All @@ -121,14 +117,14 @@ Once built, a WASM program will be available. Then, that built program can be ex
Match *
```

For example, one of the examples of [Rust WASM filter](https://github.com/fluent/fluent-bit/tree/master/examples/filter_rust) should generate its filtered logs as follows:
For example, one of the sample [Rust Wasm filters](https://github.com/fluent/fluent-bit/tree/master/examples/filter_rust) should generate its filtered logs as follows:

```text
[0] dummy.local: [1666270588.271704000, {"lang"=>"Rust", "message"=>"dummy", "original"=>"{"message":"dummy"}", "tag"=>"dummy.local", "time"=>"2022-10-20T12:56:28.271704000 +0000"}]
[0] dummy.local: [1666270589.270348000, {"lang"=>"Rust", "message"=>"dummy", "original"=>"{"message":"dummy"}", "tag"=>"dummy.local", "time"=>"2022-10-20T12:56:29.270348000 +0000"}]
[0] dummy.local: [1666270590.271107000, {"lang"=>"Rust", "message"=>"dummy", "original"=>"{"message":"dummy"}", "tag"=>"dummy.local", "time"=>"2022-10-20T12:56:30.271107000 +0000"}]
```
Another example of a Rust WASM filter is the [flb_filter_iis](https://github.com/kenriortega/flb_filter_iis) filter.
Another example of a Rust Wasm filter is the [flb_filter_iis](https://github.com/kenriortega/flb_filter_iis) filter.
This filter takes the [Internet Information Services (IIS)](https://learn.microsoft.com/en-us/iis/manage/provisioning-and-managing-iis/configure-logging-in-iis) [w3c logs](https://learn.microsoft.com/en-us/iis/manage/provisioning-and-managing-iis/configure-logging-in-iis#select-w3c-fields-to-log) (with some custom modifications) and transforms the raw string into a standard Fluent Bit JSON structured record.

```text
Expand Down Expand Up @@ -158,32 +154,33 @@ The output after the filter logic will be:
```text
[0] iis.*: [[1692131925.559486675, {}], {"c_authorization_header"=>"-", "c_ip"=>"::1", "cs_bytes"=>756, "cs_cookie"=>"-", "cs_host"=>"localhost", "cs_method"=>"GET", "cs_referer"=>"-", "cs_uri_query"=>"-", "cs_uri_stem"=>"/", "cs_user_agent"=>"Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/115.0.0.0+Safari/537.36+Edg/115.0.1901.200", "date"=>"2023-08-11 19:56:44", "s_computername"=>"WIN-PC1", "s_ip"=>"::1", "s_port"=>"80", "s_sitename"=>"W3SVC1", "sc_bytes"=>142, "sc_status"=>"304", "source"=>"LogEntryIIS", "tag"=>"iis.*", "time"=>"2023-08-15T20:38:45.559486675 +0000", "time_taken"=>1078}]
```
This filter approach provides us with several powerful advantages inherent to programming languages.
For instance, it:
- Can be extended by adding type conversion to fields such as `sc_bytes, cs_bytes, time_taken`. This is particularly useful when we need to validate our data results.
- Allows for the use of conditions to apply more descriptive filters, for example, "get only all logs that contain status codes above 4xx or 5xx".
- Can be used to define a `allow/deny` list using a data structure array or a file to store predefined IP addresses.
- Makes it possible to call an external resource such as an API or database to enhance our data.
- Allows all methods to be thoroughly tested and shared as a binary bundle or library.
These examples can be applied in our demo and can serve as an ideal starting point to create more complex logic, depending on our requirements.

### Optimize execution of WASM programs
This filter approach offers several advantages inherent to programming languages. For example:
- It can be extended by adding type conversion to fields, such as `sc_bytes`, `cs_bytes`, and `time_taken`. You can use this to validate data results.
- It allows for the use of conditions to apply more descriptive filters. For example, you can get only all logs that contain status codes higher than `4xx` or `5xx`.
- It can be used to define allow lists and deny lists using a data structure array or a file to store predefined IP addresses.
- It makes it possible to call an external resource such as an API or database to enhance your data.
- It allows all methods to be thoroughly tested and shared as a binary bundle or library.

To optimize WASM program execution, there is the option of using `flb-wamrc`.
`flb-wamrc` will reduce runtime footprint and to be best performance for filtering operations.
This tool will be built when `-DFLB_WAMRC=On` cmake option is specified and llvm infrastructure is installed on the building box.
These examples can be applied in a demo and can serve as an ideal starting point to create more complex logic, depending on your requirements.

### Optimize execution of Wasm programs

To optimize Wasm program execution, there is the option of using `flb-wamrc`. This option reduces your runtime footprint and to be best performance for filtering operations.

This tool will be built when the `-DFLB_WAMRC=On` CMake option is specified and LLVM infrastructure is installed on the building box.

```shell
$ flb-wamrc -o /path/to/built_wasm.aot /path/to/built_wasm.wasm
```

For further optimizations to the specific CPU such as Cortex-A57 series, e.g:
For further optimizations to the specific CPU, such as Cortex-A57 series:

```text
$ flb-wamrc --size-level=3 --target=aarch64v8 --cpu=cortex-a57 -o /path/to/built_wasm.aot /path/to/built_wasm.wasm
```

Then, when AOT (Ahead Of Time) compiling is succeeded:
Then, when AOT (Ahead Of Time) compiling has succeeded:

```text
Create AoT compiler with:
Expand All @@ -196,13 +193,10 @@ Create AoT compiler with:
Compile success, file /path/to/built_wasm.aot was generated.
```

Note that AOT compiling should generate CPU architecture-dependent objects. If users want to use AOT compiled object on the different archtecture, it must align the **target** and **target cpu** for actual environments.
AOT compiling should generate CPU architecture-dependent objects. If you want to use an AOT compiled object on different architecture, it must align the `target` and `target cpu` for actual environments.

### Further Concrete Examples
### Further examples

* C Filter
* https://github.com/fluent/fluent-bit/tree/master/examples/filter_wasm_c
* Rust Filter
* https://github.com/fluent/fluent-bit/tree/master/examples/filter_rust
* TinyGo Filter
* https://github.com/fluent/fluent-bit/tree/master/examples/filter_wasm_go
* [C filter](https://github.com/fluent/fluent-bit/tree/master/examples/filter_wasm_c)
* [Rust Filter](https://github.com/fluent/fluent-bit/tree/master/examples/filter_rust)
* [TinyGo filter](https://github.com/fluent/fluent-bit/tree/master/examples/filter_wasm_go)
2 changes: 2 additions & 0 deletions vale-styles/FluentBit/Acronyms.yml
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ exceptions:
- JSX
- LESS
- LLDB
- LLVM
- LTS
- NET
- NOTE
Expand Down Expand Up @@ -89,6 +90,7 @@ exceptions:
- UTC
- UTF
- UUID
- WASI
- XML
- XSS
- YAML
Expand Down
1 change: 1 addition & 0 deletions vale-styles/FluentBit/Spelling-exceptions.txt
Original file line number Diff line number Diff line change
Expand Up @@ -191,6 +191,7 @@ VictoriaMetrics
VictorOps
Vivo
VMs
Wasm
Wavefront
Worldmap
Yocto
Expand Down