Compose airbyte-server high memory usage

[mickutz]

Hi,

We've got airbyte running on an ec2 instance with docker compose.
Had some instance failures because OOM issues and it looks like the airbyte-server is using a ridiculous amount of memory. 29 gigs out of 64 available on the instance. We're running 32 connection on that instance.

Looking through old posts on the old discourse page, wasn't able to find much information aside from the fact that a restart of the container might temporarily fix the issue.

What I haven't managed to figure out is what's causing such high mem usage?
Is there an actual fix to this problem or hacking in a container restart would be the only solution?

[sajarin]

The high memory usage issue you're experiencing with Airbyte is a known issue that has been reported by several users. It seems that the Airbyte server and worker gradually increase their memory usage over time, which can lead to out-of-memory errors and cause the server to crash or restart 0.

Some users have found a temporary workaround by manually triggering the Java Garbage Collector (GC) in the worker container. Here are the steps to do this:

SSH into the worker container using docker exec -it airbyte-worker /bin/bash
Trigger the GC using jcmd 1 GC.run
However, please note that this is not a permanent solution and it may cause issues with the sync jobs 1.

Another user suggested updating the default JAVA_OPTS for both the worker and server to manage non-heap memory consumption in Java. The suggested settings are DEFAULT_JVM_OPTS='"-XX:+ExitOnOutOfMemoryError" "-XX:MaxRAMPercentage=75.0"' 2.

Lastly, you might want to consider adjusting the memory limits for your jobs in the .env file. For example, you can set JOB_MAIN_CONTAINER_MEMORY_LIMIT to 2GB and NORMALIZATION_JOB_MAIN_CONTAINER_MEMORY_LIMIT to 1GB 3.

Please note that these are all workarounds and the root cause of the issue is still being investigated. We recommend following the Github issues linked above for updates on this issue.