[source-mysql] Concurrency configuration #49849
Unanswered
alonso-cobre
asked this question in
Platform General Questions
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Im facing some issues for having multiple sync jobs, Im expecting to have 500 sync/jobs to be running every 5 minutes, right now I have 100 ish sync/jobs running fine, but once I started adding more connections (Every connection is scheduled as a cron to run every 5 minutes), jobs runs for longer (normally they take about 1-2 minute, I have Incremental Append from AuroraMySQL to S3), sometimes they remain in a running status for hours without finishing properly, I need to cancel the sync and the next run normally finishes normally, any configuration parameters that I could look into? I am already using MAX_*_WORKERS and WORKLOAD_LAUNCHER_PARALLELISM with values ranging from 100-200, I have 3 workers (3 Kubernetes pods), I also read that TEMPORAL_WORKER_PORTS somewhat affect concurrency, I want to know how you can configure Airbyte to support this type of concurrency?
Thanks.
Beta Was this translation helpful? Give feedback.
All reactions