How to optimize performance when running in Docker Desktop (on Windows)

[LarsHanegraaf]

Hi all,

I'm currently using Airbyte in Docker Desktop on Windows (I used the abctl command to install it). As I want to create a local development environment, I would like to run Airbyte alongside Dagster and a Postgres database.

Currently I'm incidentally able to run a single sync from an API (custom connector) to a database (max. 30 MB of data). If it succeeds it often takes 3 to 4 minutes to run. If it doesn't succeed, it hangs forever and it looks like it is short on resources.

I think I have a decent sized machine (11th generation i7 and 16GB of RAM), but the Windows Resource Manager shows that the memory is always on 100% (probably because abctl requires Kubernetes, which doesn't play that nice on Windows due to WSL2).

Happy to hear some ideas on how to make Airbyte performant on my local machine. I like Airbyte becasue of the ecosystem and community, but this performance kills the developer productivity unfortunately.

Thanks!
Lars