[FEATURE] Add Apache Kafka as an Event Interface alongside MQTT for Enhanced Message Processing and Reliability #522
Description
Description:
Currently, our system provides an MQTT-based event interface. To improve flexibility and compatibility, especially in larger deployments, adding Apache Kafka as an alternative event interface would be highly beneficial.
Motivation
Integrating Kafka as an additional message broker would allow us to leverage Kafka’s capabilities across both repositories and registries. Kafka offers several key advantages that would enhance our event processing:
- Commit-based Acknowledgment: Kafka allows consumers to confirm message processing through a commit, ensuring reliable delivery and processing confirmation. This feature would provide a robust mechanism to ensure that each event is fully processed, adding a layer of reliability over MQTT.
- Scalability and Performance: Kafka’s architecture is designed to handle high-throughput, low-latency data feeds, making it ideal for applications that may require scaling over time.
- Standardization: Kafka is widely used in the industry and would allow users to standardize their message processing infrastructure without needing to switch brokers for different parts of the system.
Proposed Solution
- Develop an event interface for Kafka that can utilize Kafka’s advanced features, such as commit-based acknowledgments, to enhance the reliability and robustness of message processing.
- Ensure Kafka can be applied across both registries and repositories for seamless integration.
I am interested in taking on the implementation of this feature.
Thank you for considering this enhancement!