|
| 1 | +:_mod-docs-content-type: ASSEMBLY |
| 2 | +[id="log6x-configuring-lokistack-otlp-6-2"] |
| 3 | += OTLP data ingestion in Loki |
| 4 | +include::_attributes/common-attributes.adoc[] |
| 5 | +:context: log6x-configuring-lokistack-otlp-6-2 |
| 6 | + |
| 7 | +toc::[] |
| 8 | + |
| 9 | +You can use an API endpoint by using the OpenTelemetry Protocol (OTLP) with Logging. As OTLP is a standardized format not specifically designed for Loki, OTLP requires an additional Loki configuration to map data format of OpenTelemetry to data model of Loki. OTLP lacks concepts such as _stream labels_ or _structured metadata_. Instead, OTLP provides metadata about log entries as *attributes*, grouped into the following three categories: |
| 10 | + |
| 11 | +* Resource |
| 12 | +* Scope |
| 13 | +* Log |
| 14 | +
|
| 15 | +You can set metadata for multiple entries simultaneously or individually as needed. |
| 16 | + |
| 17 | +include::modules/log6x-configuring-lokistack-otlp-data-ingestion.adoc[leveloffset=+1] |
| 18 | + |
| 19 | +[id="attribute-mapping_{context}"] |
| 20 | +== Attribute mapping |
| 21 | + |
| 22 | +When you set the {loki-op} to the `openshift-logging` mode, {loki-op} automatically applies a default set of attribute mappings. These mappings align specific OTLP attributes with stream labels and structured metadata of Loki. |
| 23 | + |
| 24 | +For typical setups, these default mappings are sufficient. However, you might need to customize attribute mapping in the following cases: |
| 25 | + |
| 26 | +* Using a custom collector: If your setup includes a custom collector that generates additional attributes that you do not want to store, consider customizing the mapping to ensure these attributes are dropped by Loki. |
| 27 | +* Adjusting attribute detail levels: If the default attribute set is more detailed than necessary, you can reduce it to essential attributes only. This can avoid excessive data storage and streamline the {logging} process. |
| 28 | + |
| 29 | +[id="custom-attribute-mapping-for-openshift_{context}"] |
| 30 | +=== Custom attribute mapping for OpenShift |
| 31 | + |
| 32 | +When using the {loki-op} in `openshift-logging` mode, attribute mapping follow OpenShift default values, but you can configure custom mappings to adjust default values. |
| 33 | +In the `openshift-logging` mode, you can configure custom attribute mappings globally for all tenants or for individual tenants as needed. When you define custom mappings, they are appended to the OpenShift default values. If you do not need default labels, you can disable them in the tenant configuration. |
| 34 | + |
| 35 | +[NOTE] |
| 36 | +==== |
| 37 | +A major difference between the {loki-op} and Loki lies in inheritance handling. Loki copies only `default_resource_attributes_as_index_labels` to tenants by default, while the {loki-op} applies the entire global configuration to each tenant in the `openshift-logging` mode. |
| 38 | +==== |
| 39 | + |
| 40 | +Within `LokiStack`, attribute mapping configuration is managed through the `limits` setting. See the following example `LokiStack` configuration: |
| 41 | + |
| 42 | +[source,yaml] |
| 43 | +---- |
| 44 | +# ... |
| 45 | +spec: |
| 46 | + limits: |
| 47 | + global: |
| 48 | + otlp: {} # <1> |
| 49 | + tenants: |
| 50 | + application: # <2> |
| 51 | + otlp: {} |
| 52 | +---- |
| 53 | +<1> Defines global OTLP attribute configuration. |
| 54 | +<2> Defines the OTLP attribute configuration for the `application` tenant within the `openshift-logging` mode. You can also configure `infrastructure` and `audit` tenants in addition to `application` tenants. |
| 55 | + |
| 56 | +[NOTE] |
| 57 | +==== |
| 58 | +You can use both global and per-tenant OTLP configurations for mapping attributes to stream labels. |
| 59 | +==== |
| 60 | + |
| 61 | +Stream labels derive only from resource-level attributes, which the `LokiStack` resource structure reflects. See the following `LokiStack` example configuration: |
| 62 | + |
| 63 | +[source,yaml] |
| 64 | +---- |
| 65 | +spec: |
| 66 | + limits: |
| 67 | + global: |
| 68 | + otlp: |
| 69 | + streamLabels: |
| 70 | + resourceAttributes: |
| 71 | + - name: "k8s.namespace.name" |
| 72 | + - name: "k8s.pod.name" |
| 73 | + - name: "k8s.container.name" |
| 74 | +---- |
| 75 | + |
| 76 | +You can drop attributes of type resource, scope, or log from the log entry. |
| 77 | + |
| 78 | +[source,yaml] |
| 79 | +---- |
| 80 | +# ... |
| 81 | +spec: |
| 82 | + limits: |
| 83 | + global: |
| 84 | + otlp: |
| 85 | + streamLabels: |
| 86 | +# ... |
| 87 | + drop: |
| 88 | + resourceAttributes: |
| 89 | + - name: "process.command_line" |
| 90 | + - name: "k8s\\.pod\\.labels\\..+" |
| 91 | + regex: true |
| 92 | + scopeAttributes: |
| 93 | + - name: "service.name" |
| 94 | + logAttributes: |
| 95 | + - name: "http.route" |
| 96 | +---- |
| 97 | + |
| 98 | +You can use regular expressions by setting `regex: true` to apply a configuration for attributes with similar names. |
| 99 | + |
| 100 | +[IMPORTANT] |
| 101 | +==== |
| 102 | +Avoid using regular expressions for stream labels, as this can increase data volume. |
| 103 | +==== |
| 104 | + |
| 105 | +Attributes that are not explicitly set as stream labels or dropped from the entry are saved as structured metadata by default. |
| 106 | + |
| 107 | +[id="customizing-openshift-defaults_{context}"] |
| 108 | +=== Customizing OpenShift defaults |
| 109 | + |
| 110 | +In the `openshift-logging` mode, certain attributes are required and cannot be removed from the configuration due to their role in OpenShift functions. Other attributes, labeled *recommended*, might be dropped if performance is impacted. For information about the attributes, see link:https://docs.openshift.com/container-platform/4.17/observability/logging/logging-6.1/log6x-opentelemetry-data-model-6.1.html#attributes_log6x-opentelemetry-data-model-6-1[OpenTelemetry data model attributes]. |
| 111 | + |
| 112 | +When using the `openshift-logging` mode without custom attributes, you can achieve immediate compatibility with OpenShift tools. If additional attributes are needed as stream labels or some attributes need to be droped, use custom configuration. Custom configurations can merge with default configurations. |
| 113 | + |
| 114 | +[id="removing-recommended-attributes_{context}"] |
| 115 | +=== Removing recommended attributes |
| 116 | + |
| 117 | +To reduce default attributes in the `openshift-logging` mode, disable recommended attributes: |
| 118 | + |
| 119 | +[source,yaml] |
| 120 | +---- |
| 121 | +# ... |
| 122 | +spec: |
| 123 | + tenants: |
| 124 | + mode: openshift-logging |
| 125 | + openshift: |
| 126 | + otlp: |
| 127 | + disableRecommendedAttributes: true # <1> |
| 128 | +---- |
| 129 | +<1> Set `disableRecommendedAttributes: true` to remove recommended attributes, which limits default attributes to the required attributes or stream labels. |
| 130 | ++ |
| 131 | +[NOTE] |
| 132 | +==== |
| 133 | +This setting might negatively impact query performance, as it removes default stream labels. You must pair this option with a custom attribute configuration to retain attributes essential for queries. |
| 134 | +==== |
| 135 | + |
| 136 | +[role="_additional-resources"] |
| 137 | +[id="additional-resources_{context}"] |
| 138 | +== Additional resources |
| 139 | +* link:https://grafana.com/docs/loki/latest/get-started/labels/[Loki labels] (Grafana documentation) |
| 140 | +* link:https://grafana.com/docs/loki/latest/get-started/labels/structured-metadata/[Structured metadata] (Grafana documentation) |
| 141 | +* link:https://docs.openshift.com/container-platform/4.17/observability/logging/logging-6.1/log6x-opentelemetry-data-model-6.1.html[OpenTelemetry data model] |
| 142 | +* link:https://opentelemetry.io/docs/specs/otel/common/#attribute[OpenTelemetry attribute] (OpenTelemetry documentation) |
0 commit comments