Minimal or Recommended Disk Space Usage To Start Using Airbyte

[tomt00001]

I have an ec2 instance with 64gb of storage with Airbyte 0.50.30 and a connection setup on a 10 minute time schedule. This job takes about 2 or 3 minutes to run. I have noticed that my airbyte_workspace volume in Docker fills very fast. To fix I deleted some of the older folders in the _data folder but I am looking for a more long-term solution.

I set TEMPORAL_HISTORY_RETENTION_IN_DAYS=1 in my .env file and is there any other way to control disk space usage?

Related is there a recommended minimum disk space size for Airbyte?

Thank you

[marcosmarxm]

Hello @tomt00001 in general the log generation won't be too alarming if you have few connections.
The only variable you have is TEMPORAL_HISTORY_RETENTION_IN_DAYS where it will trigger in the airbyte-cron to clean up old logs in the workspace folder.
Other log generation is the container themselves which is configured here the maximum retention is 100 Mb per service.
What can use more disk can be the container images, take this example:

(pipelines-py3.10) ➜  airbyte git:(master) docker images | grep 0.50.33
airbyte/airbyte-api-server                                0.50.33     1.01GB
airbyte/worker                                            0.50.33     1.85GB
airbyte/webapp                                            0.50.33     52.4MB
airbyte/temporal                                          0.50.33     344MB
airbyte/server                                            0.50.33     1.06GB
airbyte/proxy                                             0.50.33     188MB
airbyte/db                                                0.50.33     232MB
airbyte/connector-builder-server                          0.50.33     2.24GB
airbyte/cron                                              0.50.33     1.06GB
airbyte/init                                              0.50.33     5.61MB
airbyte/bootloader                                        0.50.33     1.04GB

The basic images to run Airbyte have ~10Gb, this without using any connectors which can add sum up more.
Pay attention whatever you upgrade Airbyte you will download new images and can end up with disk quickly.
Over time if you're not using external database it will also consume some local disk to keep the airbyte-db

My recommendation to start using Airbyte is around 50~100Gb of disk.
Highly recommend to enable external logs and database for production envs.