Sync jobs failing, not able access UI and VM instance.

[sakent]

Deployment: Docker
Airbyte Version: 0.42.0

• I'm new to airbyte, any help is much appreciated.
• Our sync jobs are failing and not able to access Airbyte UI, at the same time I'm unable to ssh into our vm instance.
• Restarting the VM instance is unblocking us for sometime and then again we see the same issue.
• observed following warn message from multiple components like(airbyte-cron, airbyte-connector-builder-server, airbyte-worker)
• I have already read github discussions where they have faced this warn log related to Unable to resolve host airbyte-temporal, but I don't think they apply to my situation because we don't have trouble deploying airbyte(like here).

io.grpc.internal.ManagedChannelImpl$NameResolverListener handleErrorInSyncContext
WARNING: [Channel<1>: (airbyte-temporal:7233)] Failed to resolve name. status=Status{code=UNAVAILABLE, description=
**Unable to resolve host airbyte-temporal**, cause=java.lang.RuntimeException: java.net.UnknownHostException: airbyte-temporal

        at io.grpc.internal.DnsNameResolver.resolveAddresses(DnsNameResolver.java:223)
        at io.grpc.internal.DnsNameResolver.doResolve(DnsNameResolver.java:282)
        at io.grpc.grpclb.GrpclbNameResolver.doResolve(GrpclbNameResolver.java:63)
        at io.grpc.internal.DnsNameResolver$Resolve.run(DnsNameResolver.java:318)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1589)
Caused by: java.net.UnknownHostException: airbyte-temporal
        at java.base/java.net.InetAddress$CachedAddresses.get(InetAddress.java:952)
        at java.base/java.net.InetAddress.getAllByName0(InetAddress.java:1663)
        at java.base/java.net.InetAddress.getAllByName(InetAddress.java:1528)
        at io.grpc.internal.DnsNameResolver$JdkAddressResolver.resolveAddress(DnsNameResolver.java:631)
        at io.grpc.internal.DnsNameResolver.resolveAddresses(DnsNameResolver.java:219)
        ... 6 more

• These are some other errors I downloaded for a connection/sync job

 WARN c.z.h.p.HikariPool$HouseKeeper(run):804 **HikariPool-1 - Thread starvation or clock leap detected** (housekeeper delta=50s35ms503µs359ns).
 WARN c.z.h.p.HikariPool$HouseKeeper(run):804 HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=1m15s223ms225µs414ns).
 WARN c.z.h.p.HikariPool$HouseKeeper(run):804 HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=49s314ms985µs723ns).
INFO�[m i.a.w.g.DefaultReplicationWorker(lambda$readFromSrcAndWriteToDstRunnable$7):430 - Records read: 931000 (1 GB)
WARN c.z.h.p.HikariPool$HouseKeeper(run):804 HikariPool-1 - Thread starvation or clock leap detected (housekeeper delta=48s517ms884µs615ns).
 �[43mdestination�[0m > ERROR i.a.i.d.b.u.AbstractBigQueryUploader(upload):68 Got an error while writing message: **Remote host terminated the handshake** com.google.cloud.bigquery.BigQueryException: Remote host terminated the handshake

• one more

2024-01-24 00:22:27 �[33mWARN�[m i.t.i.w.ActivityWorker$TaskHandlerImpl(logExceptionDuringResultReporting):365 - **Failure during reporting of activity result to the server**. ActivityId = a213a199-4e5f-3edb-ad72-706374318266, ActivityType = Replicate, WorkflowId=sync_18501, WorkflowType=SyncWorkflow, RunId=3b1f1604-dfc8-4fe0-a7e2-03b7e67975c9
io.grpc.StatusRuntimeException: NOT_FOUND: invalid activityID or activity already timed out or invoking workflow is completed
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271) ~[grpc-stub-1.52.1.jar:1.52.1]
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252) ~[grpc-stub-1.52.1.jar:1.52.1]
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165) ~[grpc-stub-1.52.1.jar:1.52.1]
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.respondActivityTaskFailed(WorkflowServiceGrpc.java:3866) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.lambda$sendReply$1(ActivityWorker.java:320) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.retryer.GrpcRetryer.lambda$retry$0(GrpcRetryer.java:52) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.retryer.GrpcSyncRetryer.retry(GrpcSyncRetryer.java:67) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.retryer.GrpcRetryer.retryWithResult(GrpcRetryer.java:60) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.retryer.GrpcRetryer.retry(GrpcRetryer.java:50) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.sendReply(ActivityWorker.java:315) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handleActivity(ActivityWorker.java:252) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:206) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.ActivityWorker$TaskHandlerImpl.handle(ActivityWorker.java:179) ~[temporal-sdk-1.17.0.jar:?]
	at io.temporal.internal.worker.PollTaskExecutor.lambda$process$0(PollTaskExecutor.java:93) ~[temporal-sdk-1.17.0.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]

[sajarin]

The primary issue seems related to the inability of various components to resolve the hostname airbyte-temporal, which suggests a problem with service discovery or DNS configuration inside your network or container orchestration setup. The HikariPool warnings could be a symptom of an overloaded system or resource constraints that affect the application's performance.

Can you double check the resource configuration for the instance and if possible can you upgrade to a new version?

[sakent]

@sajarin Thank you for your reply, I'm new to docker, I'm not the one who set up our airbyte, can you provide me further assistance on following:
can you provide me any references on how to sort DNS configuration problem?
when you say double check resource configuration do you mean cpu, memory? or no of workers(we are currently using default config of 5)?

[sajarin]

One thing you might want to try is to see if you can ping the temporal container via the command line tool, (in other words, airbyte is a microservice architecture and depends on airbyte-temporal to orcheastrate everything, so if some airbyte can't access/resolve temporal, can anything resolve temporal? we can answer that question using something like ping or curl

and yes by resource configuration, i mean the cpu/memory for the temporal issue and # of workers for the sync issue.

and again i'd highly recommend upgrading to the latest version, some of these issues might already be fixed and 0.42 is a super old version by this point (don't upgrade Prod on a Friday! but perhaps launch a dev instance of the latest version of Airbyte and see if you can migrate the connections)

[sakent]

@sajarin Thank you, I will consider upgrading to new version.