[Fluentbit OpenTelemetry Output] It is not performing the mapping correctly using 'logs_body_key'

[rafael-gumiero]

Describe the bug
We are using Fluent-Bit '3.1.7' to transport and parse our micro-services logs that are in JSON format following the OTel data specification. Our output will be an OTel Collector but we are having some challenges in this parse process. Basically we are not able to apply the output: opentelemetry "logs_body_key, logs_span_id_message_key, etc" to achieve the correct mapping.

To Reproduce
-fluentbit/fluent-bit.yaml

service:
  flush: 1
  log_level: info
  parsers_file: parsers.conf

pipeline:
  inputs:
    - name: dummy
      dummy: '{"resource": {"attributes": {"host.name": "PPM-JAMF-S-38577","host.arch": "arm64","host.id": "6BB99699-7C41-5541-9779-0CB0C096BBB6","os.type": "darwin","os.description": "23.6.0","os.name": "Darwin","os.version": "Darwin Kernel Version 23.6.0: Mon Jul 29 21:14:21 PDT 2024; root:xnu-10063.141.2~1\/RELEASE_ARM64_T8103","process.pid": 68504,"process.executable.path": "\/opt\/homebrew\/Cellar\/php@8.1\/8.1.29\/bin\/php","process.command": "otlp_http.php","process.command_args": ["otlp_http.php"],"process.owner": "elton.minetto","process.runtime.name": "cli","process.runtime.version": "8.1.29","telemetry.sdk.name": "opentelemetry","telemetry.sdk.language": "php","telemetry.sdk.version": "dev-main","service.name": "open-telemetry\/opentelemetry","service.version": "dev-main"},"dropped_attributes_count": 0},"scopes": [{"name": "demo","version": "1.0","attributes": {"foo": "bar"},"dropped_attributes_count": 0,"schema_url": "https:\/\/opentelemetry.io\/schemas\/1.7.1","logs": [{"timestamp": 1725887216000000000,"observed_timestamp": 1725887216473381888,"severity_number": 9,"severity_text": "error","body": {"foo": "bar","baz": "bat","xx": "hello world"},"trace_id": "00000000000000000000000000000000","span_id": "0000000000000000","trace_flags": 0,"attributes": {"event.name": "foo"},"dropped_attributes_count": 0}]}]}'
  outputs:
    - name: stdout
      match: "*"
      format: json

    - name: opentelemetry
      match: "*"
      host: 127.0.0.1
      port: 4318
      logs_body_key: $scopes[0].logs[0].body
      logs_span_id_message_key: $scopes[0].logs[0].span_id
      logs_trace_id_message_key: $scopes[0].logs[0].trace_id
      logs_severity_text_message_key: $scopes[0].logs[0].severity_text
      logs_severity_number_message_key: $scopes[0].logs[0].severity_number

-otel-collector

receivers:
  otlp:
    protocols:
      http:
        endpoint: 127.0.0.1:4318
      grpc:
processors:
  resource:
    attributes:
      - key: cloud.availability_zone
        value: "zone-1"
        action: upsert
      - key: k8s.cluster.name
        from_attribute: k8s-cluster
        action: insert
      - key: redundant-attribute
        action: delete
  k8sattributes:
    auth_type: "serviceAccount"
    passthrough: false
    extract:
      metadata:
        - k8s.pod.name
        - k8s.pod.uid
        - k8s.deployment.name
        - k8s.namespace.name
        - k8s.node.name
        - k8s.pod.start_time
        - k8s.cluster.uid
    # Pod association using resource attributes and connection
    pod_association:
      - sources:
          - from: resource_attribute
            name: k8s.pod.ip
      - sources:
          - from: resource_attribute
            name: k8s.pod.uid
      - sources:
          - from: connection
            name: ip
exporters:
  debug:
    verbosity: detailed
  file/no_rotation:
    path: otel.log
  opensearch:
    http:
      endpoint: http://127.0.0.1:9200/

service:
  telemetry:
    metrics:
      address: 0.0.0.0:8889
  pipelines:
    logs:
      receivers:
        - otlp
      processors: []
      # - resource
      # - k8sattributes
      exporters:
        - opensearch
        - debug
        - file/no_rotation

-output

[2024/09/09 15:39:54] [ info] [fluent bit] version=3.1.7, commit=, pid=38106
[2024/09/09 15:39:54] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/09/09 15:39:54] [ info] [cmetrics] version=0.9.5
[2024/09/09 15:39:54] [ info] [ctraces ] version=0.5.5
[2024/09/09 15:39:54] [ info] [input:dummy:dummy.0] initializing
[2024/09/09 15:39:54] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2024/09/09 15:39:54] [ info] [output:stdout:stdout.0] worker #0 started
[2024/09/09 15:39:54] [error] [record accessor] syntax error, unexpected ']', expecting STRING or INTEGER at '$scopes[0]'
[2024/09/09 15:39:54] [error] [output:opentelemetry:opentelemetry.1] could not process event_field with pattern '$scopes[0].logs[0].body'
[2024/09/09 15:39:54] [error] [output] failed to initialize 'opentelemetry' plugin
[2024/09/09 15:39:54] [error] [engine] output initialization failed
[2024/09/09 15:39:54] [ info] [input] pausing dummy.0
[2024/09/09 15:39:54] [ info] [output:stdout:stdout.0] thread worker #0 stopping...
[2024/09/09 15:39:54] [ info] [output:stdout:stdout.0] thread worker #0 stopped

We also tested other configurations on the keys and without success.

      logs_body_key: $scopes[].logs[].body
      logs_span_id_message_key: $scopes[].logs[].span_id
      logs_trace_id_message_key: $scopes[].logs[].trace_id
      logs_severity_text_message_key: $scopes[].logs[].severity_text
      logs_severity_number_message_key: $scopes[].logs[].severity_number

and

      logs_body_key: $scopes.logs.body
      logs_span_id_message_key: $scopes.logs.span_id
      logs_trace_id_message_key: $scopes.logs.trace_id
      logs_severity_text_message_key: $scopes.logs.severity_text
      logs_severity_number_message_key: $scopes.logs.severity_number

Your Environment

• Version used: 3.1.7
• Configuration: Attached
• Environment name and version (e.g. Kubernetes? What version?): Kubernetes
• Filters and plugins: opentelemetry output

[github-actions]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions]

This issue was closed because it has been stalled for 5 days with no activity.