Skip to content

pipeline: outputs: oci: general cleanup #1986

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jul 23, 2025
Merged

pipeline: outputs: oci: general cleanup #1986

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
0a298c2
pipeline: outputs: oci: general cleanup
alexakreizinger Jul 23, 2025
fd46112
Update pipeline/outputs/oci-logging-analytics.md
alexakreizinger Jul 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
89 changes: 40 additions & 49 deletions pipeline/outputs/oci-logging-analytics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,56 +4,47 @@ description: Send logs to Oracle Cloud Infrastructure Logging Analytics Service

# Oracle Cloud Infrastructure Logging Analytics

Oracle Cloud Infrastructure Logging Analytics output plugin allows you to ingest your log records into [OCI Logging Analytics](https://www.oracle.com/manageability/logging-analytics/) service.
The _Oracle Cloud Infrastructure Logging Analytics_ output plugin lets you ingest your log records into the [Oracle Cloud Infrastructure (OCI) Logging Analytics](https://docs.oracle.com/en-us/iaas/logging-analytics/home.htm) service.

Oracle Cloud Infrastructure Logging Analytics is a machine learning-based cloud service that monitors, aggregates, indexes, and analyzes all log data from on-premises and multicloud environments. Enabling users to search, explore, and correlate this data to troubleshoot and resolve problems faster and derive insights to make better operational decisions.
## Configuration parameters

For details about OCI Logging Analytics refer to [the documentation](https://docs.oracle.com/en-us/iaas/logging-analytics/index.html).
This plugin uses the following configuration parameters:

## Configuration Parameters
| Key | Description | Default |
| --- | ----------- | ------- |
| `config_file_location` | The location of the [configuration file](https://docs.oracle.com/en-us/iaas/Content/API/Concepts/sdkconfig.htm#SDK_and_CLI_Configuration_File) that contains OCI authentication details. | _none_ |
| `profile_name` | The OCI configuration profile name to be used from the configuration file. | `DEFAULT` |
| `namespace` | The OCI tenancy namespace to upload log data to. | _none_ |
| `proxy` | The proxy name, in `http://host:port` format. Only supports HTTP protocol. | _none_ |
| `workers` | The number of [workers](../administration/multithreading.md#outputs) to perform flush operations for this output. | `1` |
| `oci_config_in_record` | If set to `true`, the following `oci_la_*` will be read from the record itself instead of the output plugin configuration. | `false` |
| `oci_la_log_group_id` | Required. The Oracle Cloud Identifier (OCID) of the Logging Analytics where you want to store logs. | _none_ |
| `oci_la_log_source_name` | Required. The Logging Analytics Source to use for processing log records. | _none_ |
| `oci_la_entity_id` | The OCID of the Logging Analytics entity. | _none_ |
| `oci_la_entity_type` | The entity type of the Logging Analytics entity. | _none_ |
| `oci_la_log_path` | Specifies the original location of the log files. | _none_ |
| `oci_la_global_metadata` | Specifies additional global metadata along with original log content to Logging Analytics. The format is `key_name value`. This option can be set multiple times. | _none_ |
| `oci_la_metadata` | Specifies additional metadata for a log event along with original log content to Logging Analytics. The format is `key_name value`. This option can be set multiple times. | _none_ |

Following are the top level configuration properties of the plugin:
### TLS/SSL

| Key | Description | default |
|:---------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------|
| config_file_location | The location of the configuration file containing OCI authentication details. Reference for generating the configuration file - https://docs.oracle.com/en-us/iaas/Content/API/Concepts/sdkconfig.htm#SDK_and_CLI_Configuration_File | |
| profile_name | OCI Config Profile Name to be used from the configuration file | DEFAULT |
| namespace | OCI Tenancy Namespace in which the collected log data is to be uploaded | |
| proxy | define proxy if required, in http://host:port format, supports only http protocol | |
| workers | The number of [workers](../../administration/multithreading.md#outputs) to perform flush operations for this output. | `1` |
The OCI Logging Analytics output plugin supports TLS/SSL. For more details about the properties available and general configuration, see [TLS/SSL](../../administration/transport-security.md).

The following parameters are to set the Logging Analytics resources that must be used to process your logs by OCI Logging Analytics.

| Key | Description | default |
|:------------------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------|
| oci_config_in_record | If set to true, the following oci_la_* will be read from the record itself instead of the output plugin configuration. | false |
| oci_la_log_group_id | The OCID of the Logging Analytics Log Group where the logs must be stored. This is a mandatory parameter | |
| oci_la_log_source_name | The Logging Analytics Source that must be used to process the log records. This is a mandatory parameter | |
| oci_la_entity_id | The OCID of the Logging Analytics Entity | |
| oci_la_entity_type | The entity type of the Logging Analytics Entity | |
| oci_la_log_path | Specify the original location of the log files | |
| oci_la_global_metadata | Use this parameter to specify additional global metadata along with original log content to Logging Analytics. The format is 'key_name value'. This option can be set multiple times | |
| oci_la_metadata | Use this parameter to specify additional metadata for a log event along with original log content to Logging Analytics. The format is 'key_name value'. This option can be set multiple times | |

## TLS/SSL

The OCI Logging Analytics output plugin supports TLS/SSL.
For more details about the properties available and general configuration, see [TLS/SSL](../../administration/transport-security.md).

## Getting Started
## Get started

### Prerequisites

- OCI Logging Analytics service must be onboarded with the minimum required policies, in the OCI region where you want to monitor. Refer [Logging Analytics Quick Start](https://docs.oracle.com/en-us/iaas/logging-analytics/doc/quick-start.html) for details.
- Create OCI Logging Analytics LogGroup(s) if not done already. Refer [Create Log Group](https://docs.oracle.com/en-us/iaas/logging-analytics/doc/create-logging-analytics-resources.html#GUID-D1758CFB-861F-420D-B12F-34D1CC5E3E0E) for details.
- You must onboard with the OCI Logging Analytics service for the minimum required policies in the OCI region where you want to monitor. Refer to [Logging Analytics Quick Start](https://docs.oracle.com/en-us/iaas/logging-analytics/doc/quick-start.html) for details.

- You must create one or more OCI Logging Analytics log groups. Refer to [Create Log Group](https://docs.oracle.com/en-us/iaas/logging-analytics/doc/create-logging-analytics-resources.html#GUID-D1758CFB-861F-420D-B12F-34D1CC5E3E0E) for details.

### Running the output plugin
### Run the output plugin

In order to insert records into the OCI Logging Analytics service, you can run the plugin from the command line or through the configuration file:
To insert records into the OCI Logging Analytics service, you can run the plugin from the command line or through a configuration file.

#### Command line

The OCI Logging Analytics plugin can read the parameters from the command line in two ways, through the -p argument (property), e.g:
The OCI Logging Analytics plugin can read the parameters from the command line in two ways, through the `-p` (property) argument. For example:

```shell
fluent-bit -i dummy -t dummy -o oci_logan -p config_file_location=<location> -p namespace=<namespace> \
Expand Down Expand Up @@ -92,7 +83,7 @@ pipeline:
[INPUT]
Name dummy
Tag dummy

[Output]
Name oracle_log_analytics
Match *
Expand All @@ -108,9 +99,9 @@ pipeline:
{% endtab %}
{% endtabs %}

### Insert oci_la configs in the record
### Insert `oci_la` configurations in the record

In case of multiple inputs, where `oci_la_*` properties can differ, you can add the properties in the record itself and instruct the plugin to read these properties from the record. The option `oci_config_in_record`, when set to true in the output config, will make the plugin read the mandatory and optional `oci_la` properties from the incoming record. The user must ensure that the necessary configs have been inserted using relevant filters, otherwise the respective chunk will be dropped. Below is an example to insert `oci_la_log_source_name` and `oci_la_log_group_id` in the record:
In case of multiple inputs, where `oci_la_*` properties can differ, you can add the properties in the record itself and instruct the plugin to read these properties from the record. The option `oci_config_in_record`, when set to `true` in the output config, will make the plugin read the mandatory and optional `oci_la` properties from the incoming record. The user must ensure that the necessary configurations have been inserted using relevant filters, otherwise the respective chunk will be dropped. The following example inserts `oci_la_log_source_name` and `oci_la_log_group_id` in the record:

{% tabs %}
{% tab title="fluent-bit.yaml" %}
Expand All @@ -127,7 +118,7 @@ pipeline:
add:
- oci_la_log_source_name <LOG_SOURCE_NAME>
- oci_la_log_group_id <LOG_GROUP_OCID>

outputs:
- name: oracle_log_analytics
match: '*'
Expand Down Expand Up @@ -186,10 +177,10 @@ pipeline:
profile_name: ADMIN
oci_la_log_source_name: example_log_source
oci_la_log_group_id: ocid.xxxxxx
oci_la_global_metadata:
oci_la_global_metadata:
- glob_key1 value1
- glob_key2 value2
oci_la_metadata:
oci_la_metadata:
- key1 value1
- key2 value2
tls: on
Expand All @@ -203,7 +194,7 @@ pipeline:
[INPUT]
Name dummy
Tag dummy

[Output]
Name oracle_log_analytics
Match *
Expand All @@ -223,7 +214,7 @@ pipeline:
{% endtab %}
{% endtabs %}

The above configuration will generate a payload that looks like this
The previous configuration will generate a payload that resembles the following:

```json
{
Expand All @@ -248,7 +239,7 @@ The above configuration will generate a payload that looks like this

The multiple `oci_la_global_metadata` and `oci_la_metadata` options are turned into a JSON object of key value pairs, nested under the key metadata.

With `oci_config_in_record` option set to true, the metadata key-value pairs will need to be injected in the record as an object of key value pair nested under the respective metadata field. Below is an example of one such configuration
With `oci_config_in_record` option set to `true`, the metadata key/value pairs will need to be injected in the record as an object of key/value pairs nested under the respective metadata field. The following example shows one such configuration:

{% tabs %}
{% tab title="fluent-bit.yaml" %}
Expand All @@ -265,7 +256,7 @@ pipeline:
add:
- olgm.key1 val1
- olgm.key2 val2

- name: nest
match: '*'
operation: olgm.*
Expand All @@ -278,7 +269,7 @@ pipeline:
add:
- oci_la_log_source_name <LOG_SOURCE_NAME>
- oci_la_log_group_id <LOG_GROUP_OCID>

outputs:
- name: oracle_log_analytics
match: '*'
Expand Down Expand Up @@ -326,10 +317,10 @@ pipeline:
profile_name ADMIN
oci_config_in_record true
tls On
tls.verify Off
tls.verify Off
```

{% endtab %}
{% endtabs %}

The above configuration first injects the necessary metadata keys and values in the record directly, with a prefix olgm. attached to the keys in order to segregate the metadata keys from rest of the record keys. Then, using a nest filter only the metadata keys are selected by the filter and nested under `oci_la_global_metadata` key in the record, and the prefix `olgm`. is removed from the metadata keys.
The previous configuration first injects the necessary metadata keys and values in the record directly, with a prefix `olgm.` attached to the keys, which segregates the metadata keys from rest of the record keys. Then, using a `nest` filter only the metadata keys are selected by the filter and nested under `oci_la_global_metadata` key in the record, and the prefix `olgm.` is removed from the metadata keys.
1 change: 1 addition & 0 deletions vale-styles/FluentBit/Headings.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,6 +84,7 @@ exceptions:
- OpenObserve
- OpenTelemetry
- Opsgenie
- Oracle Cloud Infrastructure Logging Analytics
- PagerDuty
- PostgreSQL
- Prometheus
Expand Down