Multi-thread for connector

[dangokuson]

Hi @erdemcer,

As my understood, connector is running as single thread. Therefore, it will be a problem if a connector will have to read log for a lot of tables. Is there anyway to increase number of thread for connector?

Thanks!

[erdemcer]

Hi,
Single thread opens single logminer session and reads redo logs sequentially . Multiple threads cause multiple logminer session and overhead on database. Single thread is enough for reading. Did you test any case ?
Thanks

[dangokuson]

Hi @erdemcer, It is okay to create only one connector that will be responsible for reading the logs of 500 tables? Best regards, Dang Son

…

On Thu, Oct 21, 2021 at 5:40 PM Erdem Çer ***@***.***> wrote: Hi, Single thread opens single logminer session and reads redo logs sequentially . Multiple threads cause multiple logminer session and overhead on database. Single thread is enough for reading. Did you test any case ? Thanks — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#108 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUSHT6GYQXBSH5UVDKRER7LUH7U2RANCNFSM5GNDYYZQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

[erdemcer]

Hi,
In my production i am using single connector to read more than 1500 tables . I have not seen any lag or gap because of performance issues. Of course test on your environment will give the best answer.
Regards

[dangokuson]

@erdemcer I used a single connector to read 1000 tables, those tables are located on a different schema. However, the performance was not good then I separated it into several connectors, each connector will responsible for the different schema. After changing, the latency was still big, it was around 10 minutes for that message available on Kafka. I guess it was because of in my case there are a lot of event messages that have been generated for a while. For example, for some tables, there are around 1 million messages have been generated with 5~10 minutes.

Is there any way to improve its performance, and do you think single connector is good enough in this case?