[latest] Sync is stuck on creating pod for replication job

[nikita-zavadsky-idf]

After migration to abctl we have random connections being stuck after source and destination check. It just doesnt create a proper pod for further replication.

Attached log file below. If there are any questions regarding number of simultaneous jobs running or resources available - We have about 10 more GB RAM available, CPU is good too. For some connections it works fine, for some - doesn't.

2025-02-11 09:29:40 platform INFO 
2025-02-11 09:29:40 platform INFO APPLY Stage: CLAIM — (workloadId = ce0d828e-1dc4-496c-b122-2da42e637e48_61953_0_check) — (dataplaneId = local)
2025-02-11 09:29:40 platform INFO Claimed: true for ce0d828e-1dc4-496c-b122-2da42e637e48_61953_0_check via API for local
2025-02-11 09:29:40 platform INFO APPLY Stage: CHECK_STATUS — (workloadId = ce0d828e-1dc4-496c-b122-2da42e637e48_61953_0_check) — (dataplaneId = local)
2025-02-11 09:29:40 platform INFO No pod found running for workload ce0d828e-1dc4-496c-b122-2da42e637e48_61953_0_check
2025-02-11 09:29:40 platform INFO APPLY Stage: BUILD — (workloadId = ce0d828e-1dc4-496c-b122-2da42e637e48_61953_0_check) — (dataplaneId = local)
2025-02-11 09:29:40 platform INFO APPLY Stage: MUTEX — (workloadId = ce0d828e-1dc4-496c-b122-2da42e637e48_61953_0_check) — (dataplaneId = local)
2025-02-11 09:29:40 platform INFO No mutex key specified for workload: ce0d828e-1dc4-496c-b122-2da42e637e48_61953_0_check. Continuing...
2025-02-11 09:29:40 platform INFO APPLY Stage: LAUNCH — (workloadId = ce0d828e-1dc4-496c-b122-2da42e637e48_61953_0_check) — (dataplaneId = local)
2025-02-11 09:29:40 platform INFO [initContainer] image: airbyte/workload-init-container:1.4.1 resources: ResourceRequirements(claims=[], limits={}, requests={}, additionalProperties={})
2025-02-11 09:29:46 platform INFO Attempting to update workload: ce0d828e-1dc4-496c-b122-2da42e637e48_61953_0_check to LAUNCHED.
2025-02-11 09:29:46 platform INFO Pipeline completed for workload: ce0d828e-1dc4-496c-b122-2da42e637e48_61953_0_check.
2025-02-11 09:29:48 platform INFO 
2025-02-11 09:29:48 platform INFO ----- START CHECK -----
2025-02-11 09:29:48 platform INFO 
2025-02-11 09:29:49 platform INFO Connector exited, processing output
2025-02-11 09:29:49 platform INFO Output file jobOutput.json found
2025-02-11 09:29:49 platform INFO Connector exited with exit code 0
2025-02-11 09:29:49 platform INFO Reading messages from protocol version 0.2.0
2025-02-11 09:29:49 platform WARN MetricClient has not been initialized. Must call MetricClientFactory.CreateMetricClient before using MetricClient. Using a dummy client for now. Ignore this if Airbyte is configured to not publish any metrics.
2025-02-11 09:29:49 platform INFO i.a.i.d.c.ClickhouseDestination(main):113 starting destination: class io.airbyte.integrations.destination.clickhouse.ClickhouseDestination
2025-02-11 09:29:49 platform INFO i.a.i.b.IntegrationCliParser(parseOptions):126 integration args: {check=null, config=/config/connectionConfiguration.json}
2025-02-11 09:29:49 platform INFO i.a.i.b.IntegrationRunner(runInternal):106 Running integration: io.airbyte.integrations.base.ssh.SshWrappedDestination
2025-02-11 09:29:49 platform INFO i.a.i.b.IntegrationRunner(runInternal):107 Command: CHECK
2025-02-11 09:29:49 platform INFO i.a.i.b.IntegrationRunner(runInternal):108 Integration config: IntegrationConfig{command=CHECK, configPath='/config/connectionConfiguration.json', catalogPath='null', statePath='null'}
2025-02-11 09:29:49 platform WARN c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword order - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2025-02-11 09:29:49 platform WARN c.n.s.JsonMetaSchema(newValidator):278 Unknown keyword airbyte_secret - you should define your own Meta Schema. If the keyword is irrelevant for validation, just use a NonValidationKeyword
2025-02-11 09:29:49 platform INFO i.a.i.b.s.SshTunnel(getInstance):204 Starting connection with method: NO_TUNNEL
2025-02-11 09:29:49 platform INFO c.z.h.HikariDataSource(<init>):80 HikariPool-1 - Starting...
2025-02-11 09:29:49 platform INFO c.z.h.HikariDataSource(<init>):82 HikariPool-1 - Start completed.
2025-02-11 09:29:49 platform INFO c.z.h.HikariDataSource(close):350 HikariPool-1 - Shutdown initiated...
2025-02-11 09:29:49 platform INFO c.z.h.HikariDataSource(close):352 HikariPool-1 - Shutdown completed.
2025-02-11 09:29:49 platform INFO i.a.i.b.IntegrationRunner(runInternal):195 Completed integration: io.airbyte.integrations.base.ssh.SshWrappedDestination
2025-02-11 09:29:49 platform INFO i.a.i.d.c.ClickhouseDestination(main):115 completed destination: class io.airbyte.integrations.destination.clickhouse.ClickhouseDestination
2025-02-11 09:29:49 platform INFO Checking for optional control message...
2025-02-11 09:29:49 platform INFO Writing output of ce0d828e-1dc4-496c-b122-2da42e637e48_61953_0_check to the doc store
2025-02-11 09:29:49 platform INFO Marking workload ce0d828e-1dc4-496c-b122-2da42e637e48_61953_0_check as successful
2025-02-11 09:29:49 platform INFO 
2025-02-11 09:29:49 platform INFO Deliberately exiting process with code 0.
2025-02-11 09:29:49 platform INFO ----- END CHECK -----
2025-02-11 09:29:49 platform INFO 
2025-02-11 09:29:50 platform INFO Starting async replication
2025-02-11 09:29:50 platform INFO Creating workload 3f0c5c49-67cc-45c5-9088-b9b317ebbb71_61953_0_sync
2025-02-11 09:29:50 platform INFO APPLY Stage: CLAIM — (workloadId = 3f0c5c49-67cc-45c5-9088-b9b317ebbb71_61953_0_sync) — (dataplaneId = local)
2025-02-11 09:29:50 platform INFO Claimed: true for 3f0c5c49-67cc-45c5-9088-b9b317ebbb71_61953_0_sync via API for local
2025-02-11 09:29:50 platform INFO APPLY Stage: CHECK_STATUS — (workloadId = 3f0c5c49-67cc-45c5-9088-b9b317ebbb71_61953_0_sync) — (dataplaneId = local)
2025-02-11 09:29:50 platform INFO No pod found running for workload 3f0c5c49-67cc-45c5-9088-b9b317ebbb71_61953_0_sync
2025-02-11 09:29:50 platform INFO APPLY Stage: BUILD — (workloadId = 3f0c5c49-67cc-45c5-9088-b9b317ebbb71_61953_0_sync) — (dataplaneId = local)
2025-02-11 09:29:50 platform INFO APPLY Stage: MUTEX — (workloadId = 3f0c5c49-67cc-45c5-9088-b9b317ebbb71_61953_0_sync) — (dataplaneId = local)
2025-02-11 09:29:50 platform INFO Mutex key: 3f0c5c49-67cc-45c5-9088-b9b317ebbb71 specified for workload: 3f0c5c49-67cc-45c5-9088-b9b317ebbb71_61953_0_sync. Attempting to delete existing pods...
2025-02-11 09:29:50 platform INFO Mutex key: 3f0c5c49-67cc-45c5-9088-b9b317ebbb71 specified for workload: 3f0c5c49-67cc-45c5-9088-b9b317ebbb71_61953_0_sync found no existing pods. Continuing...
2025-02-11 09:29:50 platform INFO APPLY Stage: LAUNCH — (workloadId = 3f0c5c49-67cc-45c5-9088-b9b317ebbb71_61953_0_sync) — (dataplaneId = local)
2025-02-11 09:29:50 platform INFO [initContainer] image: airbyte/workload-init-container:1.4.1 resources: ResourceRequirements(claims=[], limits={memory=4Gi, cpu=3}, requests={memory=2Gi, cpu=2}, additionalProperties={})
2025-02-11 09:29:50 platform INFO Launching replication pod: replication-job-61953-attempt-0 with containers:
2025-02-11 09:29:50 platform INFO [source] image: airbyte/source-mongodb-v2:dev resources: ResourceRequirements(claims=[], limits={memory=4Gi, cpu=3}, requests={memory=1Gi, cpu=1}, additionalProperties={})
2025-02-11 09:29:50 platform INFO [destination] image: airbyte/destination-clickhouse:0.2.5 resources: ResourceRequirements(claims=[], limits={memory=4Gi, cpu=3}, requests={memory=1Gi, cpu=1}, additionalProperties={})
2025-02-11 09:29:50 platform INFO [orchestrator] image: airbyte/container-orchestrator:1.4.1 resources: ResourceRequirements(claims=[], limits={memory=4Gi, cpu=3}, requests={memory=2Gi, cpu=2}, additionalProperties={})