Airbyte-Cron Error o.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: namespace rate limit exceeded

[jrmbrgs]

Hello all,

ENV :
Hosting : GCE w/ docker
Version : 0.50.29

Issue :

Since my upgrade from 0.50.0 to the latest 0.50.29 I see an error popping every 10 seconds from the airbyte-cron container :

ERROR i.m.s.DefaultTaskExceptionHandler(handle):47 - Error invoking scheduled task for bean [io.airbyte.cron.jobs.SelfHealTemporalWorkflows@7ff2b454] RESOURCE_EXHAUSTED: namespace rate limit exceeded
io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: namespace rate limit exceeded

Did not found any thread related to this error on Airbyte' Slack, discord or github issue...
found a thread on temporal.io https://community.temporal.io/t/resource-exhausted-namespace-rate-limit-exceeded-for-cron-job/7583

[TimothyZhang7]

Experienced same issue

[anaselmhamdi]

Same here with version 0.50.29 on GKE on the airbyte-cron pod

[marcosmarxm]

Thanks for reporting this @jrmbrgs. At the moment the airbyte-cron service is responsible to clean up old logs and shouldn't impact in the platform itself performance. In any case I escalated this to our engineering team to take a look.

I'm also experiencing this on Airbyte 0.50.33 using Docker deployment.

2023-11-08 11:06:26 ERROR i.m.s.DefaultTaskExceptionHandler(handle):47 - Error invoking scheduled task for bean [io.airbyte.cron.jobs.SelfHealTemporalWorkflows@153e793d] RESOURCE_EXHAUSTED: namespace rate limit exceeded
io.grpc.StatusRuntimeException: RESOURCE_EXHAUSTED: namespace rate limit exceeded
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:271) ~[grpc-stub-1.55.1.jar:1.55.1]
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:252) ~[grpc-stub-1.55.1.jar:1.55.1]
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:165) ~[grpc-stub-1.55.1.jar:1.55.1]
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.listClosedWorkflowExecutions(WorkflowServiceGrpc.java:4011) ~[temporal-serviceclient-1.17.0.jar:?]
	at io.airbyte.commons.temporal.WorkflowServiceStubsWrapped.lambda$blockingStubListClosedWorkflowExecutions$0(WorkflowServiceStubsWrapped.java:47) ~[io.airbyte-airbyte-commons-temporal-0.50.33.jar:?]
	at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243) ~[failsafe-3.3.2.jar:3.3.2]
	at dev.failsafe.Functions.lambda$get$0(Functions.java:46) ~[failsafe-3.3.2.jar:3.3.2]
	at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74) ~[failsafe-3.3.2.jar:3.3.2]
	at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187) ~[failsafe-3.3.2.jar:3.3.2]
	at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376) ~[failsafe-3.3.2.jar:3.3.2]
	at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112) ~[failsafe-3.3.2.jar:3.3.2]
	at io.airbyte.commons.temporal.RetryHelper.withRetries(RetryHelper.java:60) ~[io.airbyte-airbyte-commons-temporal-0.50.33.jar:?]
	at io.airbyte.commons.temporal.WorkflowServiceStubsWrapped.withRetries(WorkflowServiceStubsWrapped.java:67) ~[io.airbyte-airbyte-commons-temporal-0.50.33.jar:?]
	at io.airbyte.commons.temporal.WorkflowServiceStubsWrapped.blockingStubListClosedWorkflowExecutions(WorkflowServiceStubsWrapped.java:47) ~[io.airbyte-airbyte-commons-temporal-0.50.33.jar:?]
	at io.airbyte.commons.temporal.TemporalClient.fetchClosedWorkflowsByStatus(TemporalClient.java:132) ~[io.airbyte-airbyte-commons-temporal-0.50.33.jar:?]
	at io.airbyte.commons.temporal.TemporalClient.restartClosedWorkflowByStatus(TemporalClient.java:110) ~[io.airbyte-airbyte-commons-temporal-0.50.33.jar:?]
	at io.airbyte.cron.jobs.SelfHealTempo^C	at io.airbyte.cron.jobs.$SelfHealTemporalWorkflows$Definition$Exec.dispatch(Unknown Source) ~[io.airbyte-airbyte-cron-0.50.33.jar:?]
	at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invoke(AbstractExecutableMethodsDefinition.java:371) ~[micronaut-inject-3.10.1.jar:3.10.1]
	at io.micronaut.inject.DelegatingExecutableMethod.invoke(DelegatingExecutableMethod.java:76) ~[micronaut-inject-3.10.1.jar:3.10.1]
	at io.micronaut.scheduling.processor.ScheduledMethodProcessor.lambda$process$5(ScheduledMethodProcessor.java:127) ~[micronaut-context-3.10.1.jar:3.10.1]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:577) ~[?:?]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]

[marcosmarxm]

I'm closing the discussion in favor of the issue #30691 which engineering team will work on.

[aballiet]

We upgraded temporal version with @anaselmhamdi to 1.22.2 and seems to solve our issue @marcosmarxm 👍

image:
    repository: temporalio/auto-setup
    pullPolicy: IfNotPresent
    tag: "1.22.2"