Shared volume/directory in kafkaconnect

[audirline]

Hello,

After looking at the documentation, I didn't find anything about share volume in KafkaConnect.
Here is my use case: I'm using Kakfaconnect and Kafkaconnectors in order to stream data from files into topics.
How can Kafkaconnect acces my files? For example, i would like to drop files in /data/directory1, /data/directory2 etc... which are directories shared by NFS.
How can I configure the yaml file for that?

apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnect
metadata:
name: my-connect-cluster
annotations:
strimzi.io/use-connector-resources: 'true'
spec:
version: 3.5.1
replicas: 2
bootstrapServers: 'kafka-cluster-kafka-bootstrap:9092'
config:
group.id: connect-cluster
offset.storage.topic: connect-cluster-offsets
config.storage.topic: connect-cluster-configs
status.storage.topic: connect-cluster-status
config.storage.replication.factor: -1
offset.storage.replication.factor: -1
status.storage.replication.factor: -1
build:
output:
type: docker
image: 'ttl.sh/strimzi-connect-example-3.5.1:24h'
plugins:
- name: kafka-connect-file
artifacts:
- type: maven
group: org.apache.kafka
artifact: connect-file
version: 3.5.1
- name: debezium-postgres-connector
artifacts:
- type: zip
url: >-
https://repo1.maven.org/maven2/io/debezium/debezium-connector-postgres/1.4.2.Final/debezium-connector-postgres-1.4.2.Final-plugin.zip
sha512sum: >-
ef1620547e6ddf5be010271849b6a87a19f6e6beee93b379c80883815b8f37ec5137095b2c99975d7704cbf957e6a33d76c61109582cad57c7cbbfae43adc86c
- name: filepulse-connector
artifacts:
- type: zip
url: ftp://fake:yoursecret@ftp_server/streamthoughts-kafka-connect-file-pulse-2.10.0.zip
- name: postgresql-connector
artifacts:
- type: jar
url: >-
https://repo1.maven.org/maven2/io/debezium/debezium-connector-postgres/2.3.3.Final/debezium-connector-postgres-2.3.3.Final.jar
- name: ebcdic-connector
artifacts:
- type: zip
url: ftp://fake:yoursecret@ftp_server/mmolimar-kafka-connect-fs-1.3.0.zip

I though I could use volumes but it seems not supported for the kind Kakfaconnect...
Thanks for your help.

[scholzj]

No, Strimzi does not support mounting any volumes like that.

[audirline]

Oh no :( Will Strimzi support volume in a near future?
Does it mean I can't use Strimzi to stream data from a file source?
Is there a workaround?

[scholzj]

I guess it depends how you define near future. But I'm not aware of anyone working on it. So I guess not.

I don't think streaming data from / to file is the best pattern for distributed Kafka Connect. If you want to interact with some application through files, it is often better to run Connect in the standalone mode as a sidecar for example.

[audirline]

hmm I see. In that case, would it be possible to use the Confluent helm to install Kafka and Kafkaconnect and then use strimzi to configure the connectors?
Would this work?
Or everything must be done via Strimzi?

[scholzj]

You mean the KafkaConnector resources? No, they only work together with the KafkaConnect resource.

[audirline]

Thanks for your quick answer @scholzj
That means i can't use (for now) Strimzi for my use cases :(

Note: I actually found another subject about that:
#5755

How can I ask for this feature to be added?

[scholzj]

It is tracked under this issue: #3693

[audirline]

Thanks @scholzj :)
I just took a look but it seems it's not prioritized for now? Do you know when Strimzi team will start looking at it?
Wait and see then.

[scholzj]

I'm not sure what you mean by not prioritized. If you want to contribute this feature, you can do so. You should start by writing a proposal about how will it work, how will the code and APIs look like etc.

[audirline]

I meant if it was under the radar of Strimzi team to implement this feature as there is a demand from other people to allow mounting volume.