Correlation between topicmap and tasks.max CosmosDBSourceConnector properties

[shivaprathipati]

Looks like it's required to have tasks.max property to be set to a number that is at least equal to the number of containers in topicmap property.

For example, if the topic map is set to something like this "topic1#container1,topic2#container2,...,topic100#container100", tasks.max property needs to be set to 100. If tasks.max is set to anything less than 100, not all 100 containers will have a lease container created.

Is my understanding correct? If yes, this will mean a large Kafka Connect cluster is needed to support all the tasks. Is there a better/efficient way to handle change feeds of all containers in cosmos database with one or fewer number of tasks?

[kushagraThapar]

@shivaprathipati we are investigating it and will get back to you soon.

[kushagraThapar]

@shivaprathipati - we investigated and yes, it is a limitation right now. It is required to have as many tasks as containers, because change feed processor (which only listens to one container per processor) is already a long running process, and we execute it on 1:1 task mapping. That way, it ensures resiliency, scalability and separation of concerns of different containers within the same connector.

It would become convoluted if we create multiple instances of change feed processor per task (specially with offset handling, multi-threading, etc.).

[shivaprathipati]

@kushagraThapar, thanks for looking into my question. It would be good if the documentation is updated to state this requirement/limitation.

[kushagraThapar]

@shivaprathipati - here you go, please review - #476