Airbyte slowness issue in Open source Environment #29788
Unanswered
ChidambaramCG
asked this question in
Connector Questions
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Hi Team,
We are using Airbyte OSS running in our EC2 Instance of 2x Large type. We started experiencing slowness of data sync frequently and while analysing we could see the instance size is getting filled up rapidly.
We allotted around 100 GB for the EBS Volume and since it filled up, We bumped the size to 150 GB and now 175 GB. So when checked, the folder /var/lib/docker/volumes/airbyte_workspace/_data is consuming a large space of around 131 GB.
We have so many connections running in our airbyte instance but the data available in the sources are all sample KB Sized data for most cases. I also tried removing connections from many workspaces but that does not help me in reducing the size of the folder mentioned. It kept growing in size only.
Our product is heavily dependent on Airbyte and it is the driving force for our data pipeline. Since it is having issues, we could not do anything.
My Question is,
Can the above mentioned folder be cleaned up? If so, How it can be cleaned ?
what is the impact and what is there in the folder and what should be done to prevent it from growing in size again?
Kindly provide some suggestions so that it will be very helpful.
Thanks & Regards,
Chidambara Ganapathy
Beta Was this translation helpful? Give feedback.
All reactions