Scaling Airbyte #35813
Unanswered
Roman-Mekicki
asked this question in
Connector Questions
Scaling Airbyte
#35813
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Hello,
I am trying to understand how Airbyte can be scaled but this is still unclear to me after reading documentation.
I get that scaling is for worker jobs. But where are the workers/jobs processes actually run?
With Docker deployment:
Application starts as set of 9 main containers (microservices?), one of them being 'worker' (https://docs.airbyte.com/understanding-airbyte/high-level-view)
Q1: What happens when I run, say 3 synchronizations? how many containers will I see when I run 'docker ps' command?
Q2: Here (https://docs.airbyte.com/understanding-airbyte/jobs#worker-responsibilities) it shows that worker create 2 docker containers. Are those containers created within main 'worker' container or alongside 9 main containers?
Q3: Does source spec, check,discover, read processes all run in newly created source container?
Q4: Does destination spec, check, write processes all run in newly created destination container?
In general where are all new processes (containers) created?
With Kubernetes deployment:
I understand all basic components will be created initially on one pod.
Q5: When I run 3 synchronizations will only pods with sync jobs be created? Or full Airbyte setup will be created on additional pods? In second scenario how does different instances communicate with each other?
In general, what is structure of additional pods created during data transfers?
Beta Was this translation helpful? Give feedback.
All reactions