Support for native otlp_proto (OTLP protobuf) decoding codec in Kafka source

[tank-500m]

Question

Currently, the opentelemetry source in Vector can receive and decode OTLP (protobuf) payloads directly, parsing them into structured Vector events that can be used in transforms and sinks.
However, when using Kafka as a transport or buffer for OTLP data, there is no official otlp_proto (protobuf) decoding codec available for the Kafka source. This means users must rely on custom transforms, Lua/VRL scripts, or build their own plugins to parse OTLP protobuf data from Kafka topics.
In our case, we use Kafka to buffer and transport OTLP logs, metrics, and traces in protobuf format (as sent from OpenTelemetry SDKs and agents). It would be extremely helpful if Vector could natively decode OTLP protobuf payloads directly in the Kafka source, so downstream transforms and sinks could process them seamlessly.

Are there any plans to support OTLP (protobuf) as a native codec in the Kafka source (like codec = "otlp_proto" or similar)?

Vector Config

No response

Vector Logs

No response

[pront]

Hi @tank-500m,

Thank you for your question.
We do want to improve Vector/OTEL interoperability.

Currently, the opentelemetry source in Vector can receive and decode OTLP (protobuf) payloads directly, parsing them into structured Vector events that can be used in transforms and sinks.

Yes, but then Vector uses an internal representation for the events for example:

{
  "message" => "[FOO] WARN Something happened",
  "severity_text" => "WARN",
  "timestamp" => "2025-06-06T06:06:06Z",
  "observed_timestamp" => "2025-06-06T07:06:06Z",
  "attributes" => { ... },
  "resources" => { ... },
  "scope" => { ... },
  "source_type" => "opentelemetry",
}

And as you mentioned above then you will have to manipulate the above manually.

Batching before shipping them somewhere outside of Vector is also challenging. See here:

https://github.com/vectordotdev/vector/blob/master/lib/opentelemetry-proto/src/proto/opentelemetry-proto/opentelemetry/proto/logs/v1/logs.proto#L56

Are there any plans to support OTLP (protobuf) as a native codec in the Kafka source (like codec = "otlp_proto" or similar)?

I would like to experiment a bit with the existing protobuf decoder before committing to a dedicated otlp_proto codec. Note that we will be implementing seamless conversion from a Vector-style OTEL event to ResourceLogs while also doing the batching potentially based on the InstrumentationScope.

[tank-500m]

hi @pront
Thanks again for taking the time to explain the current state—really appreciate it

I would like to experiment a bit with the existing protobuf decoder before committing to a dedicated otlp_proto codec.

I’m particularly intrigued by the experiments you mentioned with the existing protobuf decoder. Can you share any rough examples?

[pront]

Hi @tank-500m, I created a local setup (added decoding options to the OTEL source) and I did the following:

api:
  enabled: true

sources:
  otel:
    type: opentelemetry
    grpc:
      address: 0.0.0.0:5317
    http:
      address: 0.0.0.0:5318
      keepalive:
        max_connection_age_jitter_factor: 0.1
        max_connection_age_secs: 300
    decoding:
      codec: protobuf
      protobuf:
        desc_file: /Users/pavlos.rontidis/CLionProjects/vector/pront/otel/proto/vector-proto-related/otlp.desc
        message_type: opentelemetry.proto.collector.logs.v1.ExportLogsServiceRequest

sinks:
  logs_sink:
    type: console
    inputs:
      - otel.logs
    encoding:
      codec: json

  otel_sink:
    inputs:
      - otel.logs
    type: opentelemetry
    protocol:
      type: http
      uri: http://localhost:5318/v1/logs
      method: post
      encoding:
        codec: protobuf
        protobuf:
          desc_file: /Users/pavlos.rontidis/CLionProjects/vector/pront/otel/proto/vector-proto-related/otlp.desc
          message_type: opentelemetry.proto.collector.logs.v1.ExportLogsServiceRequest
      framing:
        method: "bytes"
      request:
        headers:
          content-type: "application/x-protobuf"

Nit - A lot of the above fields should be the default setting and the config would look even simpler:

sources:
  otel:
    type: opentelemetry
    grpc:
      address: 0.0.0.0:5317
    http:
      address: 0.0.0.0:5318
      
sinks:
  otel_sink:
    inputs:
      - otel.logs
    type: opentelemetry
    protocol:
      type: http
      uri: http://localhost:5318/v1/logs

This obviously breaking the existing custom mapping from OTEL logs to internal Vector event. But I honestly cannot think of any reason not to go ahead with this. I assume when the opentelemetry source was originally developed, we didn't have the protobuf codecs.

The final piece of the puzzle is adding a batch/partioning policy, probably using the opentelemetry.proto.common.v1.InstrumentationScope value as key.

[tank-500m]

@pront
Sorry for the late reply.
I think we should support two options:

1. Vector-native event mode. Treat the payload like regular Vector events. In this mode, OTLP logs in Kafka are fanned out so each LogRecord becomes its own event/row (e.g., for inserting individually into ClickHouse), and can be used with the usual transforms and sinks for a variety of use cases.
2. Envelope-preserving mode. Keep the OTLP envelope as-is (ResourceLogs → ScopeLogs → LogRecord) and pass it through intact for scenarios that want to operate on or forward the original structure/batching.

Both modes are valuable: the first for systems that expect flattened events, and the second for workflows that need the OTLP envelope untouched. It would be ideal if Vector let us choose between them at configuration time.

[tank-500m]

Additionally, I brought over the OTLP codec approach used by the opentelemetry source to test the Vector-native event mode, and it worked well in my environment!

[pront]

Thank you for sharing @tank-500m. Also, there is ongoing discussion here: #17310 (comment)