Request feature to load balance across multiple Kafka topics

[brian-grabau]

We have a very large deployment and we hit limits on what a single Kafka topic can support. Limitation is due to the number of nodes partitions, number of disks on nodes number threads etc... We find keeping our topics to a manageable size we have less issues. Can you add a feature to Kafka output to load balance across multiple topics?

I know we can set the policy to point to different paths with different backends in the endpoints but that defeats the autoscaling load balancing that this service can support.

Maybe a number_of_topics where _# could be appended to topic names like topic_1, topic_2 etc…
base_topic_name = "topic_base_name"
number_of_topics = 3
Results in incoming messages being published to topics topic_base_name_1, topic_base_name_2, topic_base_name_3

Or topics be made a list...
topics = "topic_base_name_1, topic_base_name_2, topic_base_name_3"

[vruello]

Hi!

I'm quite surprised that you reached the limits of Kafka by collecting events with OpenWEC. Out of curiosity, could you share an estimate of how many machines you are collecting events from and what your traffic is like (e.g. number of events, byte rate)?

I don't understand how increasing the number of topics would improve performance (except if you have only one partition per topic). Could you share a documentation or an article stating about that?

I would rather suggest to:

• increase the number of partitions in your topic
• increase the number of nodes in the Kafka cluster (at least one partition per node)
• improve the storage technology used to store Kafka partitions (SSD)
• tweak producer(s) configuration (batch size for example)
• tweak consumer(s) configuration
• tweak topic configuration (e.g. partition size, partition retention time)

[KrishnanandSingh]

Hi! I am from Brian's team. We have 100k+ workstations and ~5500 servers that we going to collect from.

If there is a substantially huge volume on one single topic, it fills up the disk unevenly on a Kafka cluster. This makes the team go into maintenance mode, rebalancing partitions which is tedious, sometimes error prone and time consuming all while hoping we don't miss realtime logs. So we have retention time and retention bytes applied to the partition. This keeps the busy work from us. But this means we need to monitor the volume and split the topic into more as they reach a certain level otherwise retention time criteria is not met.
Our consumers scaling logic is shared across all log sources including wef. They all(300+) scale for up to 24 partitions so we don't have to handle scaling on case by case basis.

So we keep it fixed to 24 and make new topics if we reach more volume(message/sec) or more size(bytes/sec) than a number.

Long story short, one could use 100s of partitions in Kafka on single topic if we are building a new system just to process windows events but after having tuned all other bits and pieces of the system, an ability to load balance across topics would make our life easier.

Forwarders like winlogbeats, logstash, etc. provide some sort of scripting and variable name for topic etc.. that we use to split volume to multiple topics. Does openwec has similar capability? Hoping we could work around.

[vruello]

Thanks for the explanations! I'm very interested in knowing how you deployed OpenWEC and how it behaves with that type of charge.

Does openwec has similar capability?

We don't have that feature yet.

As you mentioned, if the Kafka driver configuration contains multiple topics, the expected behavior is for the received events to be distributed fairly among the configured topics. This behavior does not seem to be implemented in the Kafka producer library itself, so we need to implement it ourselves. However, the Kafka producer library can send each message (e.g., a Windows event) to a different topic.

Currently, the Kafka driver is stateless. This means that we cannot use a "global" round-robin index to fairly rotate between topics without making significant changes to the Kafka driver logic and possibly impacting performance. The Kafka driver, however, receives a batch of events to send in a single "write" call (https://github.com/cea-sec/openwec/blob/main/server/src/drivers/kafka.rs#L84). Therefore, I think we can use a "local" round-robin index to achieve relative fairness. The idea is to start at a random position in the topics array and iterate through the array for each event.

I prefer your second proposal of adding a "topics" parameter to the Kafka driver configuration that contains a pre-made topics list. This approach seems much easier to understand and less likely to be used incorrectly. Additionally, if both "topics" and "topic" are present, an error should be raised.

Does this approach feel reasonable to you? It does not seem difficult to implement. Would you be interested in implementing it?

[brian-grabau]

Yes that would be perfect

[brian-grabau]

"Would you be interested in implementing it?" If you mean, would we be able to deploy and test in our environment; then yes, we can validate feature runs in a large-scale environment.

[vruello]

"Would you be interested in implementing it?" If you mean, would we be able to deploy and test in our environment; then yes, we can validate feature runs in a large-scale environment.

Sorry, I meant writing the code ;)

[lyradc]

Hi @vruello, I'm also on Brian's team. Thank you for the recommendation, that does sound reasonable. I can write the code, though it'll be a few weeks before I can get to it.

[lyradc]

@vruello I've created PR#299 to allow round-robin LB where topic contains commas. I diverged from our conversation on implementation and kept to a single parameter for topics: topic. Hopefully this won't impact performance too much.

[vruello]

#299 has been merged