Kafka Connect volume mount for camel-file-kafka-connector

[spasieczny]

Hi,

we use Strimzi to setup kafka and kafka connect on openshift. We have use case that some kafka messages should be routed to file. File hierarchy and file names a dynamic (e.g. use correlation id from message as part of file name). We found that camel-file-kafka-connector meets all requirements and we wanted to use this connector with Kafka Connect but we noticed that we are not able to mount pvc as storage for files created by this connector. We aware of discussions about such funcitonality in #3693 or in #2571.
We don't want to use such pvc as connectors jars store or something like this but only as storage for camel-file-kafka-connector output files. I wonder why such funtionality is not supported and how can we use camel-file-kafka-connector with Kafka Connect Strimzi? As i think about this, it seems that without possibllity to mount pvc with Kafka Connect Strimzi, camel-file-kafka-connector is useless. Is there any other way to meet such use case with Kafka Connet Strimzi and camel-file-kafka-connector ?
Thank in advance for any response and clarification.

RGDS,

[scholzj]

You are right, currently, connectors such as camel-file-kafka-connector have no use with Strimzi.

The possibility to add a custom volume for something like this is covered in #3693. But keep in mind that there are many challenges. There is no advanced scheduling of connectors on top of the Connect pods. If you store the data in file, you would also need to see the file in some other application to process it. So you would be required to use shared storage and that might bring a lot of limitations and runtime issues.

I think that as a good alternative, you might use some S3 connector (for example the one from Camel) and use object storage to store the data. You could also run Connect on your own for example as a sidecar to your application using the file, which makes (in my opinion) more sense than having a dedicated Connect cluster and shared storage.