delay kafka connect task startup

[cstmgl]

Hi all, I have a bit of a weird problem, that was not happening before but now that is happening and looking at the kafka code I don't know how to get around.

I got here from a discussion similar to this: #1572

Mainly I get an error in all my KafkaConnectors because they are failing at the DLQ creation point.
This because I want to do the creation of the DLQ over a KafkaTopic CRD managed by the cluster operator but seems somehow the KafkaConnector worker task is starting before my cluster operator as time to do anything.
We have the automatic topic creation disabled in kafka but it still seems that here this has no effect and the worker just tries to create the topic.
I end up getting this error:

com.blablabla.jdbc.dlq
	at org.apache.kafka.connect.runtime.errors.DeadLetterQueueReporter.createAndSetup(DeadLetterQueueReporter.java:93)
	at org.apache.kafka.connect.runtime.Worker.sinkTaskReporters(Worker.java:797)
	at org.apache.kafka.connect.runtime.Worker.buildWorkerTask(Worker.java:634)
	at org.apache.kafka.connect.runtime.Worker.startTask(Worker.java:555)
	at org.apache.kafka.connect.runtime.distributed.DistributedHerder.startTask(DistributedHerder.java:1251)
	at org.apache.kafka.connect.runtime.distributed.DistributedHerder.access$1700(DistributedHerder.java:127)
	at org.apache.kafka.connect.runtime.distributed.DistributedHerder$10.call(DistributedHerder.java:1266)
	at org.apache.kafka.connect.runtime.distributed.DistributedHerder$10.call(DistributedHerder.java:1262)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 1.
	at org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
	at org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
	at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
	at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
	at org.apache.kafka.connect.runtime.errors.DeadLetterQueueReporter.createAndSetup(DeadLetterQueueReporter.java:87)
	... 11 more
Caused by: org.apache.kafka.common.errors.InvalidReplicationFactorException: Replication factor: 3 larger than available brokers: 1.

My question is not why I get the error, this seems obvious my question is if there is any way to prevent the automatic DQL topic creation and if not is there any way I could delay the task start-up so that I could give sometime for the cluster operator to do what it needs to do with the topic creation?
Any idea?

[scholzj]

I guess it looks like the Connect or the connector create the topic using the Admin API automatically. So it is not auto-created and does not get affected by the automatic topic creation being disabled. But I have no clue how to block it I'm afraid.

[cstmgl]

as a work-around for now I just made sure I also set the replication factor to it, but I guess this one will always be whatever gets there first will do the creation (either the operator or the worker task)
thank you

[cstmgl]

going back to this topic I actually just stopped the execution of the connectors for a couple of seconds to give the entity operator enough time to create everything. I then also increased a bit the readiness probe on the resource. seems ugly but does the job