Skip to content

Commit 1617bc3

Browse files
adiforlulsadiforluls
authored
public docs for oci logging analytics (#1186)
* public docs for oci logging analytics Signed-off-by: adiforluls <[email protected]> * address review comments Signed-off-by: adiforluls <[email protected]> * rename plugin Signed-off-by: adiforluls <[email protected]> --------- Signed-off-by: adiforluls <[email protected]>
1 parent cc1644f commit 1617bc3

File tree

1 file changed

+187
-0
lines changed

1 file changed

+187
-0
lines changed

‎pipeline/outputs/oci-logging-analytics.md‎

Lines changed: 187 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,187 @@
1+
---
2+
description: Send logs to Oracle Cloud Infrastructure Logging Analytics Service
3+
---
4+
5+
# Oracle Cloud Infrastructure Logging Analytics
6+
7+
Oracle Cloud Infrastructure Logging Analytics output plugin allows you to ingest your log records into [OCI Logging Analytics](https://www.oracle.com/manageability/logging-analytics) service.
8+
9+
Oracle Cloud Infrastructure Logging Analytics is a machine learning-based cloud service that monitors, aggregates, indexes, and analyzes all log data from on-premises and multicloud environments. Enabling users to search, explore, and correlate this data to troubleshoot and resolve problems faster and derive insights to make better operational decisions.
10+
11+
For details about OCI Logging Analytics refer to https://docs.oracle.com/en-us/iaas/logging-analytics/index.html
12+
13+
## Configuration Parameters
14+
15+
Following are the top level configuration properties of the plugin:
16+
17+
| Key | Description | default |
18+
|:---------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------|
19+
| config_file_location | The location of the configuration file containing OCI authentication details. Reference for generating the configuration file - https://docs.oracle.com/en-us/iaas/Content/API/Concepts/sdkconfig.htm#SDK_and_CLI_Configuration_File | |
20+
| profile_name | OCI Config Profile Name to be used from the configuration file | DEFAULT |
21+
| namespace | OCI Tenancy Namespace in which the collected log data is to be uploaded | |
22+
| proxy | define proxy if required, in http://host:port format, supports only http protocol | |
23+
24+
The following parameters are to set the Logging Analytics resources that must be used to process your logs by OCI Logging Analytics.
25+
26+
| Key | Description | default |
27+
|:------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------|
28+
| oci_config_in_record | If set to true, the following oci_la_* will be read from the record itself instead of the output plugin configuration. | false |
29+
| oci_la_log_group_id | The OCID of the Logging Analytics Log Group where the logs must be stored. This is a mandatory parameter | |
30+
| oci_la_log_source_name | The Logging Analytics Source that must be used to process the log records. This is a mandatory parameter | |
31+
| oci_la_entity_id | The OCID of the Logging Analytics Entity | |
32+
| oci_la_entity_type | The entity type of the Logging Analytics Entity | |
33+
| oci_la_log_path | Specify the original location of the log files | |
34+
| oci_la_global_metadata | Use this parameter to specify additional global metadata along with original log content to Logging Analytics. The format is 'key_name value'. This option can be set multiple times | |
35+
| oci_la_metadata | Use this parameter to specify additional metadata for a log event along with original log content to Logging Analytics. The format is 'key_name value'. This option can be set multiple times | |
36+
37+
## TLS/SSL
38+
39+
OCI Logging Analytics output plugin supports TTL/SSL, for more details about the properties available and general configuration, please refer to the [TLS/SSL](../../administration/transport-security.md) section.
40+
41+
## Getting Started
42+
43+
### Prerequisites
44+
45+
- OCI Logging Analytics service must be onboarded with the minumum required policies, in the OCI region where you want to monitor. Refer [Logging Analytics Quick Start](https://docs.oracle.com/en-us/iaas/logging-analytics/doc/quick-start.html) for details.
46+
- Create OCI Logging Analytics LogGroup(s) if not done already. Refer [Create Log Group](https://docs.oracle.com/en-us/iaas/logging-analytics/doc/create-logging-analytics-resources.html#GUID-D1758CFB-861F-420D-B12F-34D1CC5E3E0E) for details.
47+
48+
### Running the output plugin
49+
50+
In order to insert records into the OCI Logging Analytics service, you can run the plugin from the command line or through the configuration file:
51+
52+
#### Command line
53+
54+
The OCI Logging Analytics plugin can read the parameters from the command line in two ways, through the -p argument (property), e.g:
55+
56+
```text
57+
$ fluent-bit -i dummy -t dummy -o oci_logan -p config_file_location=<location> -p namespace=<namespace> \
58+
-p oci_la_log_group_id=<lg_id> -p oci_la_log_source_name=<ls_name> -p tls=on -p tls.verify=off -m '*'
59+
```
60+
61+
#### Configuration file
62+
63+
In your main configuration file append the following Input & Output sections:
64+
65+
```text
66+
[INPUT]
67+
Name dummy
68+
Tag dummy
69+
[Output]
70+
Name oracle_log_analytics
71+
Match *
72+
Namespace <namespace>
73+
config_file_location <location>
74+
profile_name ADMIN
75+
oci_la_log_source_name <log-source-name>
76+
oci_la_log_group_id <log-group-ocid>
77+
tls On
78+
tls.verify Off
79+
```
80+
81+
### Insert oci_la configs in the record
82+
83+
In case of multiple inputs, where oci_la_* properties can differ, you can add the properties in the record itself and instruct the plugin to read these properties from the record. The option oci_config_in_record, when set to true in the output config, will make the plugin read the mandatory and optional oci_la properties from the incoming record. The user must ensure that the necessary configs have been inserted using relevant filters, otherwise the respective chunk will be dropped. Below is an example to insert oci_la_log_source_name and oci_la_log_group_id in the record:
84+
85+
```text
86+
[INPUT]
87+
Name dummy
88+
Tag dummy
89+
[Filter]
90+
Name modify
91+
Match *
92+
Add oci_la_log_source_name <LOG_SOURCE_NAME>
93+
Add oci_la_log_group_id <LOG_GROUP_OCID>
94+
[Output]
95+
Name oracle_log_analytics
96+
Match *
97+
config_file_location <oci_file_path>
98+
profile_name ADMIN
99+
oci_config_in_record true
100+
tls On
101+
tls.verify Off
102+
```
103+
104+
### Add optional metadata
105+
106+
You can attach certain metadata to the log events collected from various inputs.
107+
108+
```text
109+
[INPUT]
110+
Name dummy
111+
Tag dummy
112+
[Output]
113+
Name oracle_log_analytics
114+
Match *
115+
Namespace example_namespace
116+
config_file_location /Users/example_file_location
117+
profile_name ADMIN
118+
oci_la_log_source_name example_log_source
119+
oci_la_log_group_id ocid.xxxxxx
120+
oci_la_global_metadata glob_key1 value1
121+
oci_la_global_metadata glob_key2 value2
122+
oci_la_metadata key1 value1
123+
oci_la_metadata key2 value2
124+
tls On
125+
tls.verify Off
126+
```
127+
128+
The above configuration will generate a payload that looks like this
129+
130+
```json
131+
{
132+
"metadata": {
133+
"glob_key1": "value1",
134+
"glob_key2": "value2"
135+
},
136+
"logEvents": [
137+
{
138+
"metadata": {
139+
"key1": "value1",
140+
"key2": "value2"
141+
},
142+
"logSourceName": "example_log_source",
143+
"logRecords": [
144+
"dummy"
145+
]
146+
}
147+
]
148+
}
149+
```
150+
151+
The multiple oci_la_global_metadata and oci_la_metadata options are turned into a JSON object of key value pairs, nested under the key metadata.
152+
153+
With oci_config_in_record option set to true, the metadata key-value pairs will need to be injected in the record as an object of key value pair nested under the respective metadata field. Below is an example of one such configuration
154+
155+
```text
156+
[INPUT]
157+
Name dummy
158+
Tag dummy
159+
[FILTER]
160+
Name Modify
161+
Match *
162+
Add olgm.key1 val1
163+
Add olgm.key2 val2
164+
[FILTER]
165+
Name nest
166+
Match *
167+
Operation nest
168+
Wildcard olgm.*
169+
Nest_under oci_la_global_metadata
170+
Remove_prefix olgm.
171+
[Filter]
172+
Name modify
173+
Match *
174+
Add oci_la_log_source_name <LOG_SOURCE_NAME>
175+
Add oci_la_log_group_id <LOG_GROUP_OCID>
176+
[Output]
177+
Name oracle_log_analytics
178+
Match *
179+
config_file_location <oci_file_path>
180+
namespace <oci_tenancy_namespace>
181+
profile_name ADMIN
182+
oci_config_in_record true
183+
tls On
184+
tls.verify Off
185+
```
186+
187+
The above configuration first injects the necessary metadata keys and values in the record directly, with a prefix olgm. attached to the keys in order to segregate the metadata keys from rest of the record keys. Then, using a nest filter only the metadata keys are selected by the filter and nested under oci_la_global_metadata key in the record, and the prefix olgm. is removed from the metadata keys.

0 commit comments

Comments
 (0)