|
| 1 | +--- |
| 2 | +navigation_title: "Normalize for Stream" |
| 3 | +mapped_pages: |
| 4 | + - https://www.elastic.co/guide/en/elasticsearch/reference/current/normalize-for-stream-processor.html |
| 5 | +--- |
| 6 | + |
| 7 | +# Normalize-for-Stream processor [normalize-for-stream-processor] |
| 8 | + |
| 9 | + |
| 10 | +Detects whether a document is OpenTelemetry-compliant and if not - |
| 11 | +normalizes it as described below. If used in combination with the OTel-related |
| 12 | +mappings such as the ones defined in `logs-otel@template`, the resulting |
| 13 | +document can be queried seamlessly by clients that expect either [ECS](https://www.elastic.co/guide/en/ecs/current/index.html) or OpenTelemetry-[Semantic-Conventions](https://github.com/open-telemetry/semantic-conventions) formats. |
| 14 | + |
| 15 | +::::{note} |
| 16 | +This processor is in tech preview and is not available in our serverless offering. |
| 17 | +:::: |
| 18 | + |
| 19 | +## Detecting OpenTelemetry compliance |
| 20 | + |
| 21 | +The processor detects OpenTelemetry compliance by checking the following fields: |
| 22 | +* `resource` exists as a key and the value is a map |
| 23 | +* `resource` either doesn't contain an `attributes` field, or contains an `attributes` field of type map |
| 24 | +* `scope` is either missing or a map |
| 25 | +* `attributes` is either missing or a map |
| 26 | +* `body` is either missing or a map |
| 27 | +* `body` either doesn't contain a `text` field, or contains a `text` field of type `String` |
| 28 | +* `body` either doesn't contain a `structured` field, or contains a `structured` field that is not of type `String` |
| 29 | + |
| 30 | +If all of these conditions are met, the document is considered OpenTelemetry-compliant and is not modified by the processor. |
| 31 | + |
| 32 | +## Normalization |
| 33 | + |
| 34 | +If the document is not OpenTelemetry-compliant, the processor normalizes it as follows: |
| 35 | +* Specific ECS fields are renamed to have their corresponding OpenTelemetry Semantic Conventions attribute names. These include the following: |
| 36 | + |
| 37 | + | ECS Field | Semantic Conventions Attribute | |
| 38 | + |-------------|--------------------------------| |
| 39 | + | `span.id` | `span_id` | |
| 40 | + | `trace.id` | `trace_id` | |
| 41 | + | `message` | `body.text` | |
| 42 | + | `log.level` | `severity_text` | |
| 43 | + The processor first looks for the nested form of the ECS field and if such does not exist, it looks for a top-level field with the dotted field name. |
| 44 | +* Other specific ECS fields that describe resources and have corresponding counterparts in the OpenTelemetry Semantic Conventions are moved to the `resource.attribtues` map. Fields that are considered resource attributes are such that conform to the following conditions: |
| 45 | + * They are ECS fields that have corresponding counterparts (either with |
| 46 | + the same name or with a different name) in OpenTelemetry Semantic Conventions. |
| 47 | + * The corresponding OpenTelemetry attribute is defined in |
| 48 | + [Semantic Conventions](https://github.com/open-telemetry/semantic-conventions/tree/main/model) |
| 49 | + within a group that is defined as `type: enitity`. |
| 50 | +* All other fields, except for `@timestamp`, are moved to the `attributes` map. |
| 51 | +* All non-array entries of the `attributes` and `resource.attributes` maps are flattened. Flattening means that nested objects are merged into their parent object, and the keys are concatenated with a dot. See examples below. |
| 52 | + |
| 53 | +## Examples |
| 54 | + |
| 55 | +If an OpenTelemetry-compliant document is detected, the processor does nothing. For example, the following document will stay unchanged: |
| 56 | + |
| 57 | +```json |
| 58 | +{ |
| 59 | + "resource": { |
| 60 | + "attributes": { |
| 61 | + "service.name": "my-service" |
| 62 | + } |
| 63 | + }, |
| 64 | + "scope": { |
| 65 | + "name": "my-library", |
| 66 | + "version": "1.0.0" |
| 67 | + }, |
| 68 | + "attributes": { |
| 69 | + "http.method": "GET" |
| 70 | + }, |
| 71 | + "body": { |
| 72 | + "text": "Hello, world!" |
| 73 | + } |
| 74 | +} |
| 75 | +``` |
| 76 | + |
| 77 | +If a non-OpenTelemetry-compliant document is detected, the processor normalizes it. For example, the following document: |
| 78 | + |
| 79 | +```json |
| 80 | +{ |
| 81 | + "@timestamp": "2023-10-01T12:00:00Z", |
| 82 | + "service": { |
| 83 | + "name": "my-service", |
| 84 | + "version": "1.0.0", |
| 85 | + "environment": "production", |
| 86 | + "language": { |
| 87 | + "name": "python", |
| 88 | + "version": "3.8" |
| 89 | + } |
| 90 | + }, |
| 91 | + "log": { |
| 92 | + "level": "INFO" |
| 93 | + }, |
| 94 | + "message": "Hello, world!", |
| 95 | + "http": { |
| 96 | + "method": "GET", |
| 97 | + "url": { |
| 98 | + "path": "/api/v1/resource" |
| 99 | + }, |
| 100 | + "headers": [ |
| 101 | + { |
| 102 | + "name": "Authorization", |
| 103 | + "value": "Bearer token" |
| 104 | + }, |
| 105 | + { |
| 106 | + "name": "User-Agent", |
| 107 | + "value": "my-client/1.0" |
| 108 | + } |
| 109 | + ] |
| 110 | + }, |
| 111 | + "span" : { |
| 112 | + "id": "1234567890abcdef" |
| 113 | + }, |
| 114 | + "span.id": "abcdef1234567890", |
| 115 | + "trace.id": "abcdef1234567890abcdef1234567890" |
| 116 | +} |
| 117 | +``` |
| 118 | +will be normalized into the following form: |
| 119 | + |
| 120 | +```json |
| 121 | +{ |
| 122 | + "@timestamp": "2023-10-01T12:00:00Z", |
| 123 | + "resource": { |
| 124 | + "attributes": { |
| 125 | + "service.name": "my-service", |
| 126 | + "service.version": "1.0.0", |
| 127 | + "service.environment": "production" |
| 128 | + } |
| 129 | + }, |
| 130 | + "attributes": { |
| 131 | + "service.language.name": "python", |
| 132 | + "service.language.version": "3.8", |
| 133 | + "http.method": "GET", |
| 134 | + "http.url.path": "/api/v1/resource", |
| 135 | + "http.headers": [ |
| 136 | + { |
| 137 | + "name": "Authorization", |
| 138 | + "value": "Bearer token" |
| 139 | + }, |
| 140 | + { |
| 141 | + "name": "User-Agent", |
| 142 | + "value": "my-client/1.0" |
| 143 | + } |
| 144 | + ] |
| 145 | + }, |
| 146 | + "body": { |
| 147 | + "text": "Hello, world!" |
| 148 | + }, |
| 149 | + "span_id": "1234567890abcdef", |
| 150 | + "trace_id": "abcdef1234567890abcdef1234567890" |
| 151 | +} |
| 152 | +``` |
0 commit comments