Connectors to build a modular, plugin system

[spetz]

This proposal is a natural extension of #1641, focusing on the creation of a flexible plugin system for the Iggy server. The goal is to support a variety of use cases, such as data transformation, aggregation, external storage (e.g., databases), and acting as middleware between third-party components.

Connector Types

We envision two types of connectors, based on how they interact with the Iggy server:

• Local (IPC-based): Utilizing ultra-fast iceoryx2 for inter-process communication.
• Remote (Network-based): Communicating via TCP, UDP, QUIC, or HTTP, depending on the plugin’s requirements and characteristics.

Regardless of the connector type, there would be a unified interface that standardizes the data structure and transmission method.

Event Types

The server could dispatch two kinds of events to connectors:

• Diagnostic Events: e.g., topic_created, user_deleted, etc.
• Data Events: actual messages appended to a partition (filtered by stream and topic), based on the connector's configuration and purpose.

Why Use Standalone Connector Applications?

There are several advantages to designing connectors as separate applications:

• Language Flexibility: Any language supported by iceoryx can be used. Remote connectors don’t have this constraint, making them accessible to virtually any mature language.
• Minimal Overhead: IPC with iceoryx introduces negligible latency, making it suitable for performance-critical environments.
• Clean Server Configuration: Iggy would maintain a minimal and consistent config schema per connector (e.g., event subscriptions, destinations).
• Stable Server Codebase: Once the plugin interface is defined, Iggy’s core code remains unaffected by future plugin changes.
• Isolated Complexity: Complex plugin-specific configurations are contained within their own codebases.
• Better Language Interoperability: UDFs invoked directly within the server would require Rust or a compatible language, or possibly a scripting solution like Lua—limiting flexibility. WASM is an option but may introduce too much overhead in high-performance setups.
• Avoid Configuration Bloat: Embedding plugins directly in Iggy could lead to scattered and unrelated configuration files.

Extended Design: Pipes and Filters

Connectors can be composed in a “pipes and filters” architecture. For example, in a low-latency trading platform using SBE serialization:

• One connector could handle real-time transformation of SBE-encoded records.
• Transformed data could then be forwarded to downstream connectors or external APIs/microservices expecting data in a different format.

---

Curious to hear your thoughts. Does this direction align with your expectations or spark any ideas for improvement?

[spetz]

After some initial research, discussions and the quick prototyping, we came up with the following conclusions:

• the plugin needs to be able to deal with the potential back pressure (e.g. lots of received data that needs to be transformed and then stored in some external DB/Service)
• instead of pushing the raw message data to the plugin (even with IPC), the Iggy server would simply notify about the updated offset in the particular stream->topic->partition (depending on the subscribed topics)
• there will be other types of notifications as well, such as deleted segments, created users, etc.
• connector will pull the data as if it was the regular stream consumer (once again, pull model instead of the push one)
• to read the data as efficiently as possible without the additional overhead, the connector will keep the opened file descriptor in read-only mode to the subscribed resources (the segment is the actual file storing an append-only log of the messages)
• there will be no interference with the server having another FD in append-mode, as the plugin will only read the committed records
• thanks to the zero-copy deserialization, and the upcoming support for DirectIO and io_uring, there will be no issues with page cache, and the reads will be extremely fast
• we will start with the local connectors only for now, written purely in Rust, as the separate applications
• having the separate processes, instead of UDF directly embedded/invoked by the server, will make it easier to deal with the potential errors (invalid plugin won't crash the streaming server)

As always, looking forward to your feedback :)

[spetz]

The very first version of the connector runtime has been recently implemented, more details can be found in the repository https://github.com/apache/iggy/tree/master/core/connectors and on the blog https://iggy.apache.org/blogs/2025/06/06/connectors-runtime