Trace support for datadog-agent => vector => kafka => vector => datadog

[hvtuananh]

I am setting up a Vector pipeline that receives Traces from datadog-agent and puts them into a Kafka queue, then another Vector pipeline picks up Traces and send them to Datadog. However the Kafka sink only supports Logs and Metrics (https://github.com/vectordotdev/vector/blob/master/src/sinks/kafka/config.rs#L279) and the Kafka source only supports Logs.

Is there any plan to add Trace support for Kafka source/sink?

[jszwedko]

Hi @hvtuananh ,

I think it would just be a matter of loosening the restriction you linked there to:

Input::new(self.encoding.config().input_type())

and adding a test. We'd be happy to review a PR doing so if you feel inclined.

We'd originally restricted the data types beyond what the codec supports because there were sinks that made assumptions about the data. I don't think the kafka sink is one of those.

[hvtuananh]

@jszwedko Thank you for the response! I'll make a PR for the Kafka sink per your suggestion. I assume the appropriate place for loosening the restriction in the Kafka source would be https://github.com/vectordotdev/vector/blob/master/src/sources/kafka.rs#L365-L367

[jszwedko]

Great! I think the kafka source actually already allows any data type the codec emits. The function naming is a bit confusing there since it says new_logs() but the first argument, in this case self.decoding.output_type(), is actually what controls the types of the output.

[hvtuananh]

You're right! I tested Kafka source with native decoding codec and it works great! I've just got my approval from my company to contribute to this repository, so I'll prepare a patch and submit it soon