Airbyte Sync Issues (Self-hosted via abctl)

[malinosqui]

Hi everyone!

I'm running into an issue with my self-hosted Airbyte installation using the abctl installation method (version 1.4.1). The problem occurs when I try to start a sync. The source (PostgreSQL) and destination (BigQuery) connections successfully pass the connection tests in the UI, but the syncs themselves don’t execute. Upon checking the logs, I found the following:

Logs (Partial):

1. Workloads are being created, but no pods are running:

No pod found running for workload 30db08bb-bab9-4f16-8b0b-3136ed79e6a3_5_0_sync

2. The system attempts to clean up and recreate pods but doesn’t proceed:

Mutex key: 30db08bb-bab9-4f16-8b0b-3136ed79e6a3 specified for workload. Attempting to delete existing pods...
Existing pods for mutex key: 30db08bb-bab9-4f16-8b0b-3136ed79e6a3 deleted.

3. Containers are set up but fail to execute the replication:

[source] image: airbyte/source-postgres:3.6.28 resources: ResourceRequirements(claims=[], limits={memory=4Gi, cpu=3}, requests={memory=1Gi, cpu=1})
[destination] image: airbyte/destination-bigquery:2.9.2 resources: ResourceRequirements(claims=[], limits={memory=4Gi, cpu=3}, requests={memory=1Gi, cpu=1})

4. System resources appear sufficient, with enough memory and disk space:

• Memory available: ~9.8 GiB
• Disk space available: ~43 GiB

What I’ve Tried So Far:

• Tested connections via the Airbyte UI – both the source and destination are working fine.
• Restarted services using abctl stop/start, but the issue persists.
• Checked abctl logs, but no critical errors appear other than what’s mentioned above.

My Setup:

• Airbyte Version: 1.4.1
• Installation Method: abctl (Kubernetes using kind)
• System Resources (AWS EC2):
• Memory: 15 GiB
• Disk: 97 GiB (43 GiB free)

[Charlesnorris509]

Hey @malinosqui. I personally believe that the error comes from your kubernetes cluster, i would probably still update Airbytes as a possible fix but i would try the following steps to fix the cluster:

Use kubectl describe node to check resource allocation and usage.Ensure that your Kubernetes cluster has enough resources to run the pods.

Use kubectl logs to get detailed logs for the failing pods.Verify if there are any issues with pod creation by checking the status of the pods using kubectl get pods -A.Check for any misconfigurations in the abctl setup. Ensure that the Airbyte configuration is correct and that the necessary permissions are set for the source and destination connectors.