Skip to content

Commit 90a40e3

Browse files
Merge pull request #1972 from fluent/alexakreizinger/sc-136248/update-fluent-bit-docs-pipeline-outputs-logdna
2 parents d78b758 + 16ae1a5 commit 90a40e3

File tree

3 files changed

+38
-107
lines changed

3 files changed

+38
-107
lines changed

pipeline/outputs/logdna.md

Lines changed: 36 additions & 107 deletions
Original file line numberDiff line numberDiff line change
@@ -1,112 +1,41 @@
11
# LogDNA
22

3-
[LogDNA](https://logdna.com/) is an intuitive cloud based log management system that provides you an easy interface to query your logs once they are stored.
4-
5-
The Fluent Bit `logdna` output plugin allows you to send your log or events to a [LogDNA](https://logdna.com/) compliant service like:
6-
7-
* [LogDNA](https://logdna.com/)
8-
* [IBM Log Analysis](https://www.ibm.com/cloud/log-analysis)
9-
10-
Before to get started with the plugin configuration, make sure to obtain the proper account to get access to the service. You can start with a free trial in the following link:
11-
12-
* [LogDNA Sign Up ](https://logdna.com/sign-up/)
13-
14-
## Configuration Parameters
15-
16-
<table>
17-
<thead>
18-
<tr>
19-
<th style="text-align:left">Key</th>
20-
<th style="text-align:left">Description</th>
21-
<th style="text-align:left">Default</th>
22-
</tr>
23-
</thead>
24-
<tbody>
25-
<tr>
26-
<td style="text-align:left">logdna_host</td>
27-
<td style="text-align:left">LogDNA API host address</td>
28-
<td style="text-align:left">logs.logdna.com</td>
29-
</tr>
30-
<tr>
31-
<td style="text-align:left">logdna_port</td>
32-
<td style="text-align:left">LogDNA TCP Port</td>
33-
<td style="text-align:left">443</td>
34-
</tr>
35-
<tr>
36-
<td style="text-align:left">logdna_endpoint</td>
37-
<td style="text-align:left">LogDNA ingestion endpoint</td>
38-
<td style="text-align:left">/logs/ingest</td>
39-
</tr>
40-
<tr>
41-
<td style="text-align:left">api_key</td>
42-
<td style="text-align:left">API key to get access to the service. This property is <b>mandatory</b>.</td>
43-
<td
44-
style="text-align:left"></td>
45-
</tr>
46-
<tr>
47-
<td style="text-align:left">hostname</td>
48-
<td style="text-align:left">
49-
<p>Name of the local machine or device where Fluent Bit is running.
50-
<br />
51-
</p>
52-
<p>When this value is not set, Fluent Bit lookup the hostname and auto populate
53-
the value. If it cannot be found, an <code>unknown</code> value will be set
54-
instead.</p>
55-
</td>
56-
<td style="text-align:left"></td>
57-
</tr>
58-
<tr>
59-
<td style="text-align:left">mac</td>
60-
<td style="text-align:left">Mac address. This value is optional.</td>
61-
<td style="text-align:left"></td>
62-
</tr>
63-
<tr>
64-
<td style="text-align:left">ip</td>
65-
<td style="text-align:left">IP address of the local hostname. This value is optional.</td>
66-
<td style="text-align:left"></td>
67-
</tr>
68-
<tr>
69-
<td style="text-align:left">tags</td>
70-
<td style="text-align:left">A list of comma separated strings to group records in LogDNA and simplify
71-
the query with filters.</td>
72-
<td style="text-align:left"></td>
73-
</tr>
74-
<tr>
75-
<td style="text-align:left">file</td>
76-
<td style="text-align:left">Optional name of a file being monitored. Note that this value is only
77-
set if the record do not contain a reference to it.</td>
78-
<td style="text-align:left"></td>
79-
</tr>
80-
<tr>
81-
<td style="text-align:left">app</td>
82-
<td style="text-align:left">Name of the application. This value is auto discovered on each record,
83-
if not found, the default value is used.</td>
84-
<td style="text-align:left">Fluent Bit</td>
85-
</tr>
86-
<tr>
87-
<td style="text-align:left">workers</td>
88-
<td style="text-align:left">The number of <a href="https://docs.fluentbit.io/manual/administration/multithreading#outputs">workers</a> to perform flush operations for this output.</td>
89-
<td style="text-align:left">`0`</td>
90-
</tr>
91-
</tbody>
92-
</table>
93-
94-
## Auto Enrichment & Data Discovery
95-
96-
One of the features of Fluent Bit + LogDNA integration is the ability to auto enrich each record with further context.
97-
98-
When the plugin process each record \(or log\), it tries to lookup for specific key names that might contain specific context for the record in question, the following table describe the keys and the discovery logic:
3+
The _LogDNA_ output plugin sends logs and events to a LogDNA-compliant service.
4+
5+
## Configuration parameters
6+
7+
This plugin uses the following configuration parameters:
8+
9+
| Key | Description | Default |
10+
| --- | ----------- | ------- |
11+
| `logdna_host` | The LogDNA API host address. | `logs.logdna.com` |
12+
| `logdna_port` | The LogDNA TCP Port. | `443` |
13+
| `logdna_endpoint` | The LogDNA ingestion endpoint. | `/logs/ingest` |
14+
| `api_key` | Required. The API key to get access to the service. | _none_ |
15+
| `hostname` | Name of the local machine or device where Fluent Bit is running. If no value is specifies, Fluent Bit will look up the hostname and auto populate its value. If Fluent Bit is unable to find a value, it will set the value `unknown` instead. | _none_ |
16+
| `mac` | The MAC address. This value is optional. | |
17+
| `ip` | The IP address of the local hostname. This value is optional. | |
18+
| `tags` | A list of comma-separated strings to group records in LogDNA and simplify the query with filters. | _none_ |
19+
| `file` | Optional name of a file being monitored. This value is only set if the record doesn't contain a reference to it. | _none_ |
20+
| `app` | Name of the application. This value is automatically discovered on each record. If no value is found, the default value is used. | `Fluent Bit` |
21+
| `workers` | The number of [workers](../administration/multithreading#outputs) to perform flush operations for this output. | `0` |
22+
23+
## Data discovery and enrichment
24+
25+
The LogDNA output plugin can automatically discover and enrich records with additional content.
26+
27+
When the login processes a record or log, it searches for specific key names that might contain context for the record in question. The following table describe these keys and the discovery logic:
9928

10029
| Key | Description |
10130
| :--- | :--- |
102-
| level | If the record contains a key called `level` or `severity`, it will populate the context `level` key with that value. If not found, the context key is not set. |
103-
| file | if the record contains a key called `file`, it will populate the context `file` with the value found, otherwise If the plugin configuration provided a `file` property, that value will be used instead \(see table above\). |
104-
| app | If the record contains a key called `app`, it will populate the context `app` with the value found, otherwise it will use the value set for `app` in the configuration property \(see table above\). |
105-
| meta | if the record contains a key called `meta`, it will populate the context `meta` with the value found. |
31+
| `level` | If the record contains a key called `level` or `severity`, it will populate the context `level` key with that value. If not found, the context key is not set. |
32+
| `file` | If the record contains a key called `file`, it will populate the context `file` with the value found. Otherwise, if the plugin configuration provided a `file` property, that value will be used instead. |
33+
| `app` | If the record contains a key called `app`, it will populate the context `app` with the value found, otherwise it will use the value set for `app` in the configuration property. |
34+
| `meta` | If the record contains a key called `meta`, it will populate the context `meta` with the value found. |
10635

107-
## Getting Started
36+
## Example configuration
10837

109-
The following configuration example, will emit a dummy example record and ingest it on LogDNA. In your main configuration file append the following:
38+
The following example configuration uses a `dummy` input and `logdna` output:
11039

11140
{% tabs %}
11241
{% tab title="fluent-bit.yaml" %}
@@ -115,13 +44,13 @@ The following configuration example, will emit a dummy example record and ingest
11544
service:
11645
flush: 1
11746
log_level: info
118-
47+
11948
pipeline:
12049
inputs:
12150
- name: dummy
12251
dummy: '{"log":"a simple log message", "severity": "INFO", "meta": {"s1": 12345, "s2": true}, "app": "Fluent Bit"}'
12352
samples: 1
124-
53+
12554
outputs:
12655
- name: logdna
12756
match: '*'
@@ -182,8 +111,8 @@ Fluent Bit output:
182111

183112
Your record will be available and visible in your LogDNA dashboard after a few seconds.
184113

185-
### Query your Data in LogDNA
114+
### Query your data in LogDNA
186115

187-
In your LogDNA dashboard, go to the top filters and mark the Tags `aa` and `bb`, then you will be able to see your records as the example below:
116+
In your LogDNA dashboard, go to the top filters and mark the Tags `aa` and `bb`, then you will be able to see your records as shown:
188117

189-
![](../../.gitbook/assets/logdna.png)
118+
![LogDNA dashboard](../../.gitbook/assets/logdna.png)

vale-styles/FluentBit/Acronyms.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,7 @@ exceptions:
6161
- LLVM
6262
- LTS
6363
- LTSV
64+
- MAC
6465
- MQTT
6566
- MSK
6667
- NET

vale-styles/FluentBit/Headings.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,7 @@ exceptions:
6262
- Kubernetes
6363
- LaunchDarkly
6464
- Linux
65+
- LogDNA
6566
- LTSV
6667
- macOS
6768
- Marketplace

0 commit comments

Comments
 (0)