OpenTelemetry Envelope - Not being applied correctly when you have a structured log

[rafael-gumiero]

Bug Report

We are trying to apply the Envelope to a JSON structured log and the output does not conform to the data model expected by the OTEL standard.

To Reproduce

• Example log message:

{"message":"The entertaining kiwi reveals itself as a plucky giraffe to those who look. Before raspberries, monkeys were only apricots. A pro-active zebra without rabbits is truly a apple of calm chimpanzees. A grape is an orange's lemon!","timestamp_app":"2024-09-06T19:35:25.767494843Z","log_type":"APPLICATION","log_level":"ALERT","event":{"data":"98In7cO5Fc"},"http_request":"GET /shouting-with-happiness-one-cannot-separate-deers-from-inventive-tigers","http_method":"GET","X-Request-ID":"bc5a0dec-df87-48ae-a049-a55d928ee0f0","custom_fields":{"custom.field_1":"vw","custom.field_2":"g","custom.field_3":"c0LwCH","custom.field_4":"Sg6f0W","service.name":"poc-opensearch-logs"},"trace_id":"9e44b233-9b2f-4656-a4e5-994d1da6ea18","span_id":"8a9a4217-0808-4ef6-91e4-a9f2eb4435ee","trace_flags":"0102040800000000"}
{"message":"The entertaining kiwi reveals itself as a plucky giraffe to those who look. Before raspberries, monkeys were only apricots. A pro-active zebra without rabbits is truly a apple of calm chimpanzees. A grape is an orange's lemon!","timestamp_app":"2024-09-06T19:35:25.767494843Z","log_type":"APPLICATION","log_level":"ALERT","event":{"data":"98In7cO5Fc"},"http_request":"GET /shouting-with-happiness-one-cannot-separate-deers-from-inventive-tigers","http_method":"GET","X-Request-ID":"bc5a0dec-df87-48ae-a049-a55d928ee0f0","custom_fields":{"custom.field_1":"vw","custom.field_2":"g","custom.field_3":"c0LwCH","custom.field_4":"Sg6f0W","service.name":"poc-opensearch-logs"},"trace_id":"9e44b233-9b2f-4656-a4e5-994d1da6ea18","span_id":"8a9a4217-0808-4ef6-91e4-a9f2eb4435ee","trace_flags":"0102040800000000"}
{"message":"The entertaining kiwi reveals itself as a plucky giraffe to those who look. Before raspberries, monkeys were only apricots. A pro-active zebra without rabbits is truly a apple of calm chimpanzees. A grape is an orange's lemon!","timestamp_app":"2024-09-06T19:35:25.767494843Z","log_type":"APPLICATION","log_level":"ALERT","event":{"data":"98In7cO5Fc"},"http_request":"GET /shouting-with-happiness-one-cannot-separate-deers-from-inventive-tigers","http_method":"GET","X-Request-ID":"bc5a0dec-df87-48ae-a049-a55d928ee0f0","custom_fields":{"custom.field_1":"vw","custom.field_2":"g","custom.field_3":"c0LwCH","custom.field_4":"Sg6f0W","service.name":"poc-opensearch-logs"},"trace_id":"9e44b233-9b2f-4656-a4e5-994d1da6ea18","span_id":"8a9a4217-0808-4ef6-91e4-a9f2eb4435ee","trace_flags":"0102040800000000"}
{"message":"The entertaining kiwi reveals itself as a plucky giraffe to those who look. Before raspberries, monkeys were only apricots. A pro-active zebra without rabbits is truly a apple of calm chimpanzees. A grape is an orange's lemon!","timestamp_app":"2024-09-06T19:35:25.767494843Z","log_type":"APPLICATION","log_level":"ALERT","event":{"data":"98In7cO5Fc"},"http_request":"GET /shouting-with-happiness-one-cannot-separate-deers-from-inventive-tigers","http_method":"GET","X-Request-ID":"bc5a0dec-df87-48ae-a049-a55d928ee0f0","custom_fields":{"custom.field_1":"vw","custom.field_2":"g","custom.field_3":"c0LwCH","custom.field_4":"Sg6f0W","service.name":"poc-opensearch-logs"},"trace_id":"9e44b233-9b2f-4656-a4e5-994d1da6ea18","span_id":"8a9a4217-0808-4ef6-91e4-a9f2eb4435ee","trace_flags":"0102040800000000"}
{"message":"The entertaining kiwi reveals itself as a plucky giraffe to those who look. Before raspberries, monkeys were only apricots. A pro-active zebra without rabbits is truly a apple of calm chimpanzees. A grape is an orange's lemon!","timestamp_app":"2024-09-06T19:35:25.767494843Z","log_type":"APPLICATION","log_level":"ALERT","event":{"data":"98In7cO5Fc"},"http_request":"GET /shouting-with-happiness-one-cannot-separate-deers-from-inventive-tigers","http_method":"GET","X-Request-ID":"bc5a0dec-df87-48ae-a049-a55d928ee0f0","custom_fields":{"custom.field_1":"vw","custom.field_2":"g","custom.field_3":"c0LwCH","custom.field_4":"Sg6f0W","service.name":"poc-opensearch-logs"},"trace_id":"9e44b233-9b2f-4656-a4e5-994d1da6ea18","span_id":"8a9a4217-0808-4ef6-91e4-a9f2eb4435ee","trace_flags":"0102040800000000"}
{"message":"The entertaining kiwi reveals itself as a plucky giraffe to those who look. Before raspberries, monkeys were only apricots. A pro-active zebra without rabbits is truly a apple of calm chimpanzees. A grape is an orange's lemon!","timestamp_app":"2024-09-06T19:35:25.767494843Z","log_type":"APPLICATION","log_level":"ALERT","event":{"data":"98In7cO5Fc"},"http_request":"GET /shouting-with-happiness-one-cannot-separate-deers-from-inventive-tigers","http_method":"GET","X-Request-ID":"bc5a0dec-df87-48ae-a049-a55d928ee0f0","custom_fields":{"custom.field_1":"vw","custom.field_2":"g","custom.field_3":"c0LwCH","custom.field_4":"Sg6f0W","service.name":"poc-opensearch-logs"},"trace_id":"9e44b233-9b2f-4656-a4e5-994d1da6ea18","span_id":"8a9a4217-0808-4ef6-91e4-a9f2eb4435ee","trace_flags":"0102040800000000"}
{"message":"The entertaining kiwi reveals itself as a plucky giraffe to those who look. Before raspberries, monkeys were only apricots. A pro-active zebra without rabbits is truly a apple of calm chimpanzees. A grape is an orange's lemon!","timestamp_app":"2024-09-06T19:35:25.767494843Z","log_type":"APPLICATION","log_level":"ALERT","event":{"data":"98In7cO5Fc"},"http_request":"GET /shouting-with-happiness-one-cannot-separate-deers-from-inventive-tigers","http_method":"GET","X-Request-ID":"bc5a0dec-df87-48ae-a049-a55d928ee0f0","custom_fields":{"custom.field_1":"vw","custom.field_2":"g","custom.field_3":"c0LwCH","custom.field_4":"Sg6f0W","service.name":"poc-opensearch-logs"},"trace_id":"9e44b233-9b2f-4656-a4e5-994d1da6ea18","span_id":"8a9a4217-0808-4ef6-91e4-a9f2eb4435ee","trace_flags":"0102040800000000"}

• Using the following configuration file (fluentbit):

service:
  flush: 1
  log_level: info

pipeline:
  inputs:
    - name: tail
      path: /var/logs/otelflat-exporter/fluentbit/*.log
      multiline.parser: docker, cri
      tag: kube.*
      Mem_Buf_Limit: 5MB
      Skip_Long_Lines: Off

      processors:
        logs:
          - name: opentelemetry_envelope

  outputs:
    - name: stdout
      match: "*"

    - name: opentelemetry
      match: "*"
      host: 127.0.0.1
      port: 4318

• Using the following configuration file (OTel Collector):

    extensions:
      basicauth/client:
        client_auth:
          username: "xxxxx"
          password: "xxxxx"
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    processors:
      k8sattributes:
        auth_type: "serviceAccount"
        passthrough: false
        extract:
          metadata:
            - k8s.pod.name
            - k8s.pod.uid
            - k8s.deployment.name
            - k8s.namespace.name
            - k8s.node.name
            - k8s.pod.start_time
            - k8s.cluster.uid
        # Pod association using resource attributes and connection
        pod_association:
          - sources:
              - from: resource_attribute
                name: k8s.pod.ip
          - sources:
              - from: resource_attribute
                name: k8s.pod.uid
          - sources:
              - from: connection
                name: ip
      memory_limiter:
        check_interval: 1s
        limit_percentage: 75
        spike_limit_percentage: 15
      batch:
        send_batch_size: 10000
        timeout: 10s
    exporters:
      opensearch/log:
        http:
          endpoint: https://xxxxxxxxxx
          auth:
            authenticator: basicauth/client
      debug:
        verbosity: detailed
    service:
      extensions: [basicauth/client]
      pipelines:
        logs:
          receivers: [otlp]
          processors: [memory_limiter, batch, k8sattributes]
          exporters:
            - debug
            - opensearch/log

• Final Output:

{
    "resourceLogs": [
        {
            "resource": {
                "attributes": [
                    {
                        "key": "cloud.availability_zone",
                        "value": {
                            "stringValue": "zone-1"
                        }
                    }
                ]
            },
            "scopeLogs": [
                {
                    "scope": {},
                    "logRecords": [
                        {
                            "timeUnixNano": "1725651721220525000",
                            "body": {
                                "stringValue": "{\"log\"=>\"{\"message\":\"The entertaining kiwi reveals itself as a plucky giraffe to those who look. Before raspberries, monkeys were only apricots. A pro-active zebra without rabbits is truly a apple of calm chimpanzees. A grape is an orange's lemon!\",\"timestamp_app\":\"2024-09-06T19:35:25.767494843Z\",\"log_type\":\"APPLICATION\",\"log_level\":\"ALERT\",\"event\":{\"data\":\"98In7cO5Fc\"},\"http_request\":\"GET /shouting-with-happiness-one-cannot-separate-deers-from-inventive-tigers\",\"http_method\":\"GET\",\"X-Request-ID\":\"bc5a0dec-df87-48ae-a049-a55d928ee0f0\",\"custom_fields\":{\"custom.field_1\":\"vw\",\"custom.field_2\":\"g\",\"custom.field_3\":\"c0LwCH\",\"custom.field_4\":\"Sg6f0W\",\"service.name\":\"poc-opensearch-logs\"},\"trace_id\":\"9e44b233-9b2f-4656-a4e5-994d1da6e"
                            },
                            "traceId": "",
                            "spanId": ""
                        }
                    ]
                }
            ]
        }
    ]
}

Your Environment

• Version used: 3.1.7
• Configuration: attached
• Environment name and version (e.g. Kubernetes? What version?): k8s

[SkyCTing]

I have the same question like this, i have a structured log,but fluent bit can not verify it corrently.

[rafael-gumiero]

I tested it with unstructured log and it worked perfectly but unfortunately not with the structured example I posted. I even tried to change the lib of one of my applications to generate in OTEL format but then I had another challenge with the OpenTelemetry Output #9372.

[SkyCTing]

I tested it with unstructured log and it worked perfectly but unfortunately not with the structured example I posted. I even tried to change the lib of one of my applications to generate in OTEL format but then I had another challenge with the OpenTelemetry Output #9372.

I have fixed it by a very strange way, use filebeat to collect my logs,use logstash to receive them and output to the specific files,then insert into mysql with fluentd.

[edsiper]

examples provided do not match what multiline.parser: docker, cri expects. It seems tail is not parsing as JSON.

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.