Stripe source to Snowflake destination connection failure

[leheriss]

Hey,
I am getting a Failed to load resource: the server responded with a status of 502 (Bad Gateway) when trying to set up a connection between a stripe account and snowflake. I am using airbyte OpenSource on an AWS EC2 instance.
Airbyte version: 0.63.1
Stripe connector version: 5.4.5
snowflake connector version: 3.11.0

Here are some logs on my docker containers:
airbyte/worker

2024-08-07 12:16:30 platform > Docker volume job log path: /tmp/workspace/948bf1ea-96e9-4dcb-9263-20e17b2202dc/0/logs.log
2024-08-07 12:16:30 platform > Executing worker wrapper. Airbyte version: 0.63.1
2024-08-07 12:16:30 platform > Using default value for environment variable SIDECAR_KUBE_CPU_LIMIT: '2.0'
2024-08-07 12:16:30 platform > Using default value for environment variable SOCAT_KUBE_CPU_LIMIT: '2.0'
2024-08-07 12:16:30 platform > Using default value for environment variable SIDECAR_KUBE_CPU_REQUEST: '0.1'
2024-08-07 12:16:30 platform > Using default value for environment variable SOCAT_KUBE_CPU_REQUEST: '0.1'
2024-08-07 12:16:30 platform > Checking if airbyte/source-stripe:5.4.5 exists...
2024-08-07 12:16:30 platform > airbyte/source-stripe:5.4.5 was found locally.
2024-08-07 12:16:30 platform > Creating docker container = source-stripe-discover-948bf1ea-96e9-4dcb-9263-20e17b2202dc-0-xhult with resources io.airbyte.config.ResourceRequirements@7074a64b[cpuRequest=,cpuLimit=,memoryRequest=,memoryLimit=,additionalProperties={}] and allowedHosts null
2024-08-07 12:16:30 platform > Preparing command: docker run --rm --init -i -w /data/948bf1ea-96e9-4dcb-9263-20e17b2202dc/0 --log-driver none --name source-stripe-discover-948bf1ea-96e9-4dcb-9263-20e17b2202dc-0-xhult --network host -v airbyte_workspace:/data -v oss_local_root:/local -e DEPLOYMENT_MODE=OSS -e WORKER_CONNECTOR_IMAGE=airbyte/source-stripe:5.4.5 -e AUTO_DETECT_SCHEMA=true -e LAUNCHDARKLY_KEY= -e SOCAT_KUBE_CPU_REQUEST=0.1 -e SOCAT_KUBE_CPU_LIMIT=2.0 -e FIELD_SELECTION_WORKSPACES= -e USE_STREAM_CAPABLE_STATE=true -e AIRBYTE_ROLE=dev -e WORKER_ENVIRONMENT=DOCKER -e APPLY_FIELD_SELECTION=false -e WORKER_JOB_ATTEMPT=0 -e OTEL_COLLECTOR_ENDPOINT=http://host.docker.internal:4317 -e FEATURE_FLAG_CLIENT=config -e AIRBYTE_VERSION=0.63.1 -e WORKER_JOB_ID=948bf1ea-96e9-4dcb-9263-20e17b2202dc airbyte/source-stripe:5.4.5 discover --config source_config.json
2024-08-07 12:16:31 platform > Reading messages from protocol version 0.2.0
2024-08-07 12:16:32 platform > Using concurrent cdk with concurrency level 20
2024-08-07 12:18:28 WARN i.t.i.s.WorkflowExecutionHandler(throwAndFailWorkflowExecution):171 - Workflow execution failure WorkflowId='cc118478-5685-4a5f-8125-8ca9bc8e62a6', RunId=1f0e0847-d493-460a-ad63-6682747fd099, WorkflowType='DiscoverCatalogWorkflow'
io.temporal.failure.ActivityFailure: Activity with activityType='Run' failed: 'Activity task timed out'. scheduledEventId=12, startedEventId=13, activityId=e11ce862-844d-3a84-9722-d100ebd0a31e, identity='', retryState=RETRY_STATE_NON_RETRYABLE_FAILURE
	at java.base/java.lang.Thread.getStackTrace(Thread.java:2450) [?:?]
	at io.temporal.internal.sync.ActivityStubBase.execute(ActivityStubBase.java:49) ~[temporal-sdk-1.22.3.jar:?]
	at io.temporal.internal.sync.ActivityInvocationHandler.lambda$getActivityFunc$0(ActivityInvocationHandler.java:83) ~[temporal-sdk-1.22.3.jar:?]
	at io.temporal.internal.sync.ActivityInvocationHandlerBase.invoke(ActivityInvocationHandlerBase.java:60) ~[temporal-sdk-1.22.3.jar:?]
	at jdk.proxy2/jdk.proxy2.$Proxy87.run(Unknown Source) ~[?:?]
	at io.airbyte.workers.temporal.discover.catalog.DiscoverCatalogWorkflowImpl.run(DiscoverCatalogWorkflowImpl.java:55) ~[io.airbyte-airbyte-workers-0.63.1.jar:?]
	at DiscoverCatalogWorkflowImplProxy.run$accessor$6iiQaGVG(Unknown Source) ~[?:?]
	at DiscoverCatalogWorkflowImplProxy$auxiliary$jeE0RTjL.call(Unknown Source) ~[?:?]
	at io.airbyte.micronaut.temporal.TemporalActivityStubInterceptor.execute(TemporalActivityStubInterceptor.java:79) ~[io.airbyte-airbyte-micronaut-temporal-0.63.1.jar:?]
	at DiscoverCatalogWorkflowImplProxy.run(Unknown Source) ~[?:?]
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[?:?]
	at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[?:?]
	at io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation$RootWorkflowInboundCallsInterceptor.execute(POJOWorkflowImplementationFactory.java:339) ~[temporal-sdk-1.22.3.jar:?]
	at io.temporal.internal.sync.POJOWorkflowImplementationFactory$POJOWorkflowImplementation.execute(POJOWorkflowImplementationFactory.java:314) ~[temporal-sdk-1.22.3.jar:?]
	at io.temporal.internal.sync.WorkflowExecutionHandler.runWorkflowMethod(WorkflowExecutionHandler.java:70) ~[temporal-sdk-1.22.3.jar:?]
	at io.temporal.internal.sync.SyncWorkflow.lambda$start$0(SyncWorkflow.java:135) ~[temporal-sdk-1.22.3.jar:?]
	at io.temporal.internal.sync.CancellationScopeImpl.run(CancellationScopeImpl.java:102) ~[temporal-sdk-1.22.3.jar:?]
	at io.temporal.internal.sync.WorkflowThreadImpl$RunnableWrapper.run(WorkflowThreadImpl.java:107) ~[temporal-sdk-1.22.3.jar:?]
	at io.temporal.worker.ActiveThreadReportingExecutor.lambda$submit$0(ActiveThreadReportingExecutor.java:53) ~[temporal-sdk-1.22.3.jar:?]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: io.temporal.failure.TimeoutFailure: message='activity ScheduleToClose timeout', timeoutType=TIMEOUT_TYPE_SCHEDULE_TO_CLOSE
2024-08-07 12:18:48 WARN i.a.c.t.CancellationHandler$TemporalCancellationHandler(checkAndHandleCancellation):79 - The job timeout and was not a sync, we will destroy the pods related to it
io.temporal.client.ActivityNotExistsException: WorkflowId=cc118478-5685-4a5f-8125-8ca9bc8e62a6, RunId=1f0e0847-d493-460a-ad63-6682747fd099, ActivityType=Run, ActivityId=e11ce862-844d-3a84-9722-d100ebd0a31e
	at io.temporal.internal.activity.HeartbeatContextImpl.sendHeartbeatRequest(HeartbeatContextImpl.java:211) ~[temporal-sdk-1.22.3.jar:?]
	at io.temporal.internal.activity.HeartbeatContextImpl.doHeartBeatLocked(HeartbeatContextImpl.java:156) ~[temporal-sdk-1.22.3.jar:?]
	at io.temporal.internal.activity.HeartbeatContextImpl.lambda$scheduleNextHeartbeatLocked$0(HeartbeatContextImpl.java:179) ~[temporal-sdk-1.22.3.jar:?]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.base/java.lang.Thread.run(Thread.java:1583) [?:?]
Caused by: io.grpc.StatusRuntimeException: NOT_FOUND: workflow execution already completed
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:268) ~[grpc-stub-1.62.2.jar:1.62.2]
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:249) ~[grpc-stub-1.62.2.jar:1.62.2]
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:167) ~[grpc-stub-1.62.2.jar:1.62.2]
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.recordActivityTaskHeartbeat(WorkflowServiceGrpc.java:3973) ~[temporal-serviceclient-1.22.3.jar:?]
	at io.temporal.internal.client.ActivityClientHelper.sendHeartbeatRequest(ActivityClientHelper.java:63) ~[temporal-sdk-1.22.3.jar:?]
	at io.temporal.internal.activity.HeartbeatContextImpl.sendHeartbeatRequest(HeartbeatContextImpl.java:197) ~[temporal-sdk-1.22.3.jar:?]
	... 8 more

I also got some warning from the temporal container :

{"level":"warn","ts":"2024-08-07T12:22:27.958Z","msg":"Unspecified task queue kind","service":"frontend","wf-task-queue-name":"DISCOVER_SCHEMA","wf-namespace":"default","logging-call-at":"workflow_handler.go:3772"}

I tried increasing the MAX_DISCOVER_WORKERS env var to 15 and and the BASIC_AUTH_PROXY_TIMEOUT variable to 1 hour but it did not change a anything.
On Stripe side, I cannot see any calls made from Airbyte. The instance on which Airbyte is deployed does not have a CPU usage increase either. The source and destination tests are green though...

I am kind of stuck there, if anyone ever faced that kind of issue, I'd be glad to get some hints :)

[rsotogar]

hi @leheriss , did you find a fix? I am getting these errors when I attempt to launch concurrent syncs (I presume it's due to resource starvation but not getting enough information from the logs)