feat(opentelemetry sink): add automatic native log to OTLP conversion

[szibis]

Summary

Add automatic conversion from Vector's native flat log format to OTLP (OpenTelemetry Protocol) format in the opentelemetry sink's otlp codec.

Problem: Users currently need 50+ lines of complex VRL to manually build the nested OTLP structure (resourceLogs → scopeLogs → logRecords with KeyValue arrays).

Solution: The OTLP encoder now automatically detects native log events and converts them to valid OTLP protobuf. Pre-formatted OTLP events continue to use passthrough encoding (backward compatible).

Performance Impact

Scenario	OLD (VRL + encode)	NEW (auto-convert)	Improvement
Single event	378 µs	352 µs	7.4% faster
Batch 100	2,718 µs	575 µs	4.7x faster
Batch throughput	61 MiB/s	288 MiB/s	4.7x higher

Architecture Comparison

%%{init: {'theme': 'base', 'themeVariables': { 'lineColor': '#000000', 'primaryTextColor': '#000000'}}}%%
flowchart TB
    subgraph OLD["OLD: 50 lines VRL"]
        direction LR
        O1[Source] ==> O2[VRL Transform]
        O2 ==> O3[OTLP Encoder]
        O3 ==> O4[Collector]
    end

    subgraph NEW["NEW: Zero VRL"]
        direction LR
        N1[Source] ==> N2[OTLP Encoder]
        N2 ==> N3[Collector]
    end

    OLD ==> NEW

    style O1 fill:#ffffff,stroke:#000000,stroke-width:2px,color:#000000
    style O2 fill:#cccccc,stroke:#000000,stroke-width:3px,color:#000000
    style O3 fill:#ffffff,stroke:#000000,stroke-width:2px,color:#000000
    style O4 fill:#ffffff,stroke:#000000,stroke-width:2px,color:#000000
    style N1 fill:#ffffff,stroke:#000000,stroke-width:2px,color:#000000
    style N2 fill:#999999,stroke:#000000,stroke-width:3px,color:#000000
    style N3 fill:#ffffff,stroke:#000000,stroke-width:2px,color:#000000

    linkStyle default stroke:#000000,stroke-width:2px

Loading

Vector configuration

Before (Complex VRL Required)

50+ lines of VRL transformation

sources:
  otel_source:
    type: opentelemetry
    grpc:
      address: 0.0.0.0:4317

transforms:
  build_otlp_structure:
    type: remap
    inputs: ["otel_source.logs"]
    source: |
      resource_attrs = []
      if exists(.resources) {
        for_each(object!(.resources)) -> |k, v| {
          resource_attrs = push(resource_attrs, {
            "key": k,
            "value": { "stringValue": to_string(v) ?? "" }
          })
        }
      }

      log_attrs = []
      if exists(.attributes) {
        for_each(object!(.attributes)) -> |k, v| {
          attr_value = if is_boolean(v) {
            { "boolValue": v }
          } else if is_integer(v) {
            { "intValue": to_string!(v) }
          } else if is_float(v) {
            { "doubleValue": v }
          } else {
            { "stringValue": to_string(v) ?? "" }
          }
          log_attrs = push(log_attrs, { "key": k, "value": attr_value })
        }
      }

      .resource_logs = [{
        "resource": { "attributes": resource_attrs },
        "scopeLogs": [{
          "scope": {
            "name": .scope.name ?? "",
            "version": .scope.version ?? ""
          },
          "logRecords": [{
            "timeUnixNano": to_string(to_unix_timestamp(.timestamp, unit: "nanoseconds")),
            "severityText": .severity_text ?? "INFO",
            "severityNumber": .severity_number ?? 9,
            "body": { "stringValue": .message ?? "" },
            "attributes": log_attrs,
            "traceId": .trace_id ?? "",
            "spanId": .span_id ?? ""
          }]
        }]
      }]

sinks:
  otel_out:
    type: opentelemetry
    inputs: ["build_otlp_structure"]
    endpoint: http://collector:4317
    encoding:
      codec: otlp

After (Zero VRL)

sources:
  otel_source:
    type: opentelemetry
    grpc:
      address: 0.0.0.0:4317
    # use_otlp_decoding: false (default)

sinks:
  otel_out:
    type: opentelemetry
    inputs: ["otel_source.logs"]
    endpoint: http://collector:4317
    encoding:
      codec: otlp  # Automatic conversion!

With Optional Enrichment

sources:
  otel_source:
    type: opentelemetry
    grpc:
      address: 0.0.0.0:4317

transforms:
  enrich:
    type: remap
    inputs: ["otel_source.logs"]
    source: |
      # Simple flat field access - no nested structure needed
      .attributes.processed_by = "vector"
      .resources."deployment.region" = "us-west-2"

sinks:
  otel_out:
    type: opentelemetry
    inputs: ["enrich"]
    endpoint: http://collector:4317
    encoding:
      codec: otlp

Supported Native Log Format

{
  "message": "User login successful",
  "timestamp": "2024-01-15T10:30:00Z",
  "severity_text": "INFO",
  "severity_number": 9,
  "trace_id": "0123456789abcdef0123456789abcdef",
  "span_id": "fedcba9876543210",
  "attributes": {
    "user_id": "user-12345",
    "duration_ms": 42.5
  },
  "resources": {
    "service.name": "auth-service"
  },
  "scope": {
    "name": "auth-module",
    "version": "1.0.0"
  }
}

Field Mapping

Native Field	OTLP Field	Notes
.message / .body / .msg	body.stringValue	Auto-detected
.timestamp	timeUnixNano	chrono, epoch, RFC3339
.severity_text	severityText
.severity_number	severityNumber	Auto-inferred from text if missing
.trace_id	traceId	Hex string → 16 bytes
.span_id	spanId	Hex string → 8 bytes
.attributes.*	attributes[]	Object → KeyValue array
.resources.*	resource.attributes[]	Object → KeyValue array
.scope.name	scope.name	Instrumentation scope
.scope.version	scope.version	Instrumentation scope

How did you test this PR?

Unit Tests

cargo test -p opentelemetry-proto  # 44 tests
cargo test -p codecs --test otlp   # 28 tests

Tests cover:

• Basic encoding, error handling, type conversion
• Timestamp formats (chrono, epoch seconds/nanos, RFC3339)
• Severity inference from text
• Message field fallbacks
• Hex validation for trace_id/span_id
• Roundtrip encode/decode

E2E Tests

cargo vdev test e2e opentelemetry-native

Docker Compose with telemetrygen validates:

• Native logs convert to valid OTLP
• Field preservation through pipeline
• Custom attributes via VRL transforms
• Correct event counting metrics

Benchmarks

cargo bench --bench codecs --features codecs-benches -- otlp_encoding

Change Type

• Bug fix
• New feature
• Non-functional (chore, refactoring, docs)
• Performance

Is this a breaking change?

• Yes
• No

Pre-formatted OTLP events (with resourceLogs field) continue using existing passthrough path.

Does this PR include user facing changes?

• Yes. Please add a changelog fragment based on our guidelines.
• No. A maintainer will apply the no-changelog label to this PR.

References

Issue/PR	Relationship
#22789	Directly related - Our codec-level approach provides the same outcome with simpler implementation
#24515	Solves - User requesting OTLP transform helper; this eliminates the need
#23971	Solves root cause - OTLP encoding failing; our encoder handles all formats
#22550	Complementary - Adds metrics support; our patterns can extend to metrics

• Related: feat!(opentelemetry sink): Add support for encoders #22789
• Closes: Support OTLP transform for opentelemetry sink #24515
• Related: OTLP encoding not working with OpenTelemetry sink #23971
• Related: feat(sink): add opentelemetry metrics support #22550

[github-actions]

All contributors have signed the CLA ✍️ ✅
_{Posted by the CLA Assistant Lite bot.}

[szibis]

I have read the CLA Document and I hereby sign the CLA