[receiver/kafka] Add support for topic and exclude_topic as an array #44575

khushijain21 · 2025-11-27T09:51:28Z

Description

This PR adds support for topics and exclude_topics. (accepts a list instead of a single string).
It also deprecates topic and exclude_topic in favor of two new config options introduced above.

Link to tracking issue

Fixes #44477

Testing

Added UT
Performed manual testing

Documentation

Added

axw

I'm a bit torn on the deprecation.

On the one hand, "topics" seems like the best config name, OTOH it's a bit annoying to deprecate config again. A couple of other options:

Keep both "topic" and "topics" indefinitely, allow only one or the other. Might be a little bit confusing as to why there are two config options.
Keep "topic" (don't add "topics"), and allow it to be either an array or a single string. Maybe that's a bit dirty.

@paulojmdias @MovieStoreGuy @pavolloffay any opinions?

paulojmdias · 2025-11-27T19:52:24Z

I’m also a bit torn on the deprecation.

I agree with @axw that there won’t be a perfect solution for this change. However, even though it seems a bit dirty, I’d go with option 2, keeping topic and supporting either an array or a string. As long as we update the documentation well, I will say the impact is short.

If we want to avoid mixed types but also avoid the topic/topics confusion, we could keep topic (string) and support something like additional_topics (array). For this option topic field should kept mandatory and additional_topics optional.

marclop · 2025-11-28T03:22:49Z

I personally would prefer if we deprecate topic and exclude_topic right away since the plurals imply you can specify multiple. That's aligned with what we're doing and the support we're adding.

I agree, It'll be annoying for that initial version but I think it captures the semantic meaning better than the singular.

axw · 2025-11-28T05:29:32Z

After sleeping on it, I'm now inclined to deprecate the existing config and replace with plural. It's annoying, but better now than later when increasingly more users start using the receiver. @paulojmdias are you OK with this?

As an aside, we should really look into marking the receiver stable in the not too distant future. It's pretty solid. Once we do that, we'll have less flexibility to make breaking changes though.

paulojmdias · 2025-11-28T07:42:31Z

After sleeping on it, I'm now inclined to deprecate the existing config and replace with plural. It's annoying, but better now than later when increasingly more users start using the receiver. @paulojmdias are you OK with this?

Yes, I'm ok with that. 👍

axw

Thanks @khushijain21, looks good overall.

Can you please add a warning level log message (probably to the Start method) if topic``exclude_topic is configured, recommending that topics/exclude_topics be used instead?

.chloggen/support-arrayof-string.yaml

receiver/kafkareceiver/config.go

receiver/kafkareceiver/factory.go

receiver/kafkareceiver/config.go

Co-authored-by: Andrew Wilkins <[email protected]>

axw · 2025-11-28T10:21:29Z

receiver/kafkareceiver/config.go

+		if len(zeroConfig.Logs.Topics) == 0 {
+			c.Logs.Topics = []string{zeroConfig.Logs.Topic}
+		} else {
+			return fmt.Errorf("both logs.topic and logs.topics cannot be set")


This sort of things should be done in Validate

I agree, because we were mutating an empty logs.topics in Unmarshall function here https://github.com/khushijain21/opentelemetry-collector-contrib/blob/kafkarec/receiver/kafkareceiver/config.go#L66 - doing this check in Validate would result in a false error. For ex: it would fail even when just "logs.topic" is set. (We are appending logs.topics above so).

We would have to clear logs.Topic in case we really want this to go in Validate.

So I guess we can keep it here if this is okay. Either way, your call

anyway, I moved it to validate in this commit 0fdab34

We would have to clear logs.Topic in case we really want this to go in Validate

I cleared the logs.Topic but when we unmarshal it here again https://github.com/khushijain21/opentelemetry-collector-contrib/blob/kafkarec/receiver/kafkareceiver/config.go#L135 - it sets log.topic back to what user configured.

Hence validate() method returns a false error. I think it is okay to do this check in Unmarshal

axw · 2025-11-28T10:23:00Z

receiver/kafkareceiver/kafka_receiver.go

ICYMI (#44575 (review)):

Can you please add a warning level log message (probably to the Start method) if topic/exclude_topic is configured, recommending that topics/exclude_topics be used instead?

yeah I missed it, fixed it in 0fdab34

paulojmdias

The version here should be the next one.

paulojmdias · 2025-11-28T11:18:47Z