Skip to content
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 16 additions & 19 deletions pipeline/processors/sql.md
Original file line number Diff line number Diff line change
@@ -1,27 +1,24 @@
# Structured Query Language (SQL)
# SQL

The **sql** processor provides a simple interface to select content from Logs by also supporting conditional expressions.
The _SQL_ processor lets you use conditional expressions to select content from logs. This processor doesn't depend on a database or table. Instead, your queries run on the stream.

Our SQL processor does not depend on a database or indexing; it runs everything on the fly (this is good). We don't have the concept of tables but you run the query on the STREAM.

Note that this processor differs from the "stream processor interface" that runs after the filters; this one can only be used in the processor's section of the input plugins when using YAML configuration mode.
This processor differs from the stream processor interface that runs after filters.

{% hint style="info" %}

**Note:** Both processors and this specific component can be enabled only by using
the YAML configuration format. Classic mode configuration format doesn't support processors.
Only [YAML configuration files](../administration/configuring-fluent-bit/yaml/README.md) support processors.

{% endhint %}

## Configuration Parameters
## Configuration parameters

| Key | Description |
| :---------- | :--- |
| query | Define the SQL statement to run on top of the Logs stream; it must end with `;` . |
| Key | Description |
| --- | ----------- |
| `query` | The SQL statement to query your logs stream. This statement must end with `;`. |

### Simple selection example
## Basic selection example

The following example generates a sample message with two keys called `key` and `http.url`. By using a simple SQL statement we will select only the key `http.url`.
The following example generates a sample message with two keys: `key` and `http.url`, then uses a SQL statement to select only the key `http.url`.

{% tabs %}
{% tab title="fluent-bit.yaml" %}
Expand All @@ -31,12 +28,12 @@ pipeline:
inputs:
- name: dummy
dummy: '{"key1": "123.4", "http.url": "https://fluentbit.io/search?q=docs"}'

processors:
logs:
- name: sql
query: "SELECT http.url FROM STREAM;"

outputs:
- name : stdout
match: '*'
Expand All @@ -46,9 +43,9 @@ pipeline:
{% endtab %}
{% endtabs %}

### Extract and select example
## Extract and select example

Similar to the example above, now we will extract the parts of `http.url` and only select the domain from the value, for that we will use together content-modifier and sql processors together:
The following example is similar to the previous example, but additionally extracts part of `http.url` to select the domain from the value. To accomplish this, use the `content-modifier` and `sql` processors in tandem:

{% tabs %}
{% tab title="fluent-bit.yaml" %}
Expand Down Expand Up @@ -78,11 +75,11 @@ pipeline:
{% endtab %}
{% endtabs %}

the expected output of this pipeline will be something like this:
The resulting output resembles the following:

```json
{
"date": 1711059261.630668,
"http_domain": "fluentbit.io"
}
```
```
1 change: 1 addition & 0 deletions vale-styles/FluentBit/Headings.yml
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ exceptions:
- ServiceMonitor
- SignalFx
- Slack
- SQL
- SSL
- StatsD
- Studio
Expand Down