Airbyte worker fails to start in Kubernetes (0.50.33)

[adamstrawson]

I'm new to Airbyte, and getting familiar with its setup ahead of our data team using it more widely.

We've setup Airbyte within Kubernetes (GKE) via the Airbyte helm charts. Helm values are pretty much the default with the exception of externalDatabase and logs.

All pods are running successfully, with the exception of the worker pods, which are failing to start-up.

2023-10-30 13:51:46 INFO i.m.c.e.DefaultEnvironment(<init>):167 - Established active environments: [k8s, cloud, gcp, control-plane]
2023-10-30 13:51:47 INFO i.m.l.PropertiesLoggingLevelsConfigurer(configureLogLevelForPrefix):111 - Setting log level 'DEBUG' for logger: 'io.airbyte.bootloader'
2023-10-30 13:51:48 INFO i.a.f.ConfigFileClient(<init>):105 - path /flags does not exist, will return default flag values
2023-10-30 13:51:48 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable METRIC_CLIENT: ''
2023-10-30 13:51:48 INFO i.a.c.EnvConfigs(getEnvOrDefault):1158 - Using default value for environment variable METRIC_CLIENT: ''
2023-10-30 13:51:49 WARN i.a.m.l.MetricClientFactory(initialize):74 - MetricClient was not recognized or not provided. Accepted values are `datadog` or `otel`. 
2023-10-30 13:51:49 ERROR i.m.r.Micronaut(handleStartupException):338 - Error starting Micronaut server: Error instantiating bean of type  [io.airbyte.workers.orchestrator.KubeOrchestratorHandleFactory]

Path Taken: new ApplicationInitializer() --> ApplicationInitializer.syncActivities --> List.syncActivities([ReplicationActivity replicationActivity],NormalizationActivity normalizationActivity,DbtTransformationActivity dbtTransformationActivity,NormalizationSummaryCheckActivity normalizationSummaryCheckActivity,WebhookOperationActivity webhookOperationActivity,ConfigFetchActivity configFetchActivity,RefreshSchemaActivity refreshSchemaActivity) --> new ReplicationActivityImpl(SecretsHydrator secretsHydrator,Path workspaceRoot,WorkerEnvironment workerEnvironment,LogConfigs logConfigs,String airbyteVersion,AirbyteConfigValidator airbyteConfigValidator,TemporalUtils temporalUtils,AirbyteApiClient airbyteApiClient,[OrchestratorHandleFactory orchestratorHandleFactory],MetricClient metricClient,FeatureFlagClient featureFlagClient) --> new KubeOrchestratorHandleFactory([ContainerOrchestratorConfig containerOrchestratorConfig],WorkerConfigsProvider workerConfigsProvider,FeatureFlagClient featureFlagClient,Integer serverPort,MetricClient metricClient)
io.micronaut.context.exceptions.BeanInstantiationException: Error instantiating bean of type  [io.airbyte.workers.orchestrator.KubeOrchestratorHandleFactory]

Path Taken: new ApplicationInitializer() --> ApplicationInitializer.syncActivities --> List.syncActivities([ReplicationActivity replicationActivity],NormalizationActivity normalizationActivity,DbtTransformationActivity dbtTransformationActivity,NormalizationSummaryCheckActivity normalizationSummaryCheckActivity,WebhookOperationActivity webhookOperationActivity,ConfigFetchActivity configFetchActivity,RefreshSchemaActivity refreshSchemaActivity) --> new ReplicationActivityImpl(SecretsHydrator secretsHydrator,Path workspaceRoot,WorkerEnvironment workerEnvironment,LogConfigs logConfigs,String airbyteVersion,AirbyteConfigValidator airbyteConfigValidator,TemporalUtils temporalUtils,AirbyteApiClient airbyteApiClient,[OrchestratorHandleFactory orchestratorHandleFactory],MetricClient metricClient,FeatureFlagClient featureFlagClient) --> new KubeOrchestratorHandleFactory([ContainerOrchestratorConfig containerOrchestratorConfig],WorkerConfigsProvider workerConfigsProvider,FeatureFlagClient featureFlagClient,Integer serverPort,MetricClient metricClient)
	at io.micronaut.context.DefaultBeanContext.resolveByBeanFactory(DefaultBeanContext.java:2367) ~[micronaut-inject-3.10.1.jar:3.10.1]
	at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2305) ~[micronaut-inject-3.10.1.jar:3.10.1]
	at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2251) ~[micronaut-inject-3.10.1.jar:3.10.1]
	at io.micronaut.context.DefaultBeanContext.createRegistration(DefaultBeanContext.java:3016) ~[micronaut-inject-3.10.1.jar:3.10.1]
	at io.micronaut.context.SingletonScope.getOrCreate(SingletonScope.java:80) ~[micronaut-inject-3.10.1.jar:3.10.1]
	at io.micronaut.context.DefaultBeanContext.findOrCreateSingletonBeanRegistration(DefaultBeanContext.java:2918) ~[micronaut-inject-3.10.1.jar:3.10.1]
	at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2879) ~[micronaut-inject-3.10.1.jar:3.10.1]
	at io.micronaut.context.DefaultBeanContext.resolveBeanRegistration(DefaultBeanContext.java:2800) ~[micronaut-inject-3.10.1.jar:3.10.1]
	at io.micronaut.context.DefaultBeanContext.getBean(DefaultBeanContext.java:1617) ~[micronaut-inject-3.10.1.jar:3.10.1]
	at io.micronaut.context.AbstractBeanResolutionContext.getBean(AbstractBeanResolutionContext.java:66) ~[micronaut-inject-3.10.1.jar:3.10.1]
	at io.micronaut.context.AbstractInitializableBeanDefinition.resolveBean(AbstractInitializableBeanDefinition.java:2065) ~[micronaut-inject-3.10.1.jar:3.10.1]
	at io.micronaut.context.AbstractInitializableBeanDefinition.getBeanForConstructorArgument(AbstractInitializableBeanDefinition.java:1297) ~[micronaut-inject-3.10.1.jar:3.10.1]
	at io.airbyte.workers.orchestrator.$KubeOrchestratorHandleFactory$Definition.build(Unknown Source) ~[io.airbyte-airbyte-commons-worker-0.50.33.jar:?]
	at io.micronaut.context.DefaultBeanContext.resolveByBeanFactory(DefaultBeanContext.java:2354) ~[micronaut-inject-3.10.1.jar:3.10.1]
	at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2305) ~[micronaut-inject-3.10.1.jar:3.10.1]
	at io.micronaut.context.DefaultBeanContext.doCreateBean(DefaultBeanContext.java:2251) ~[micronaut-inject-3.10.1.jar:3.10.1]
	at io.micronaut.context.DefaultBeanContext.createRegistration(DefaultBeanContext.java:3016) ~[micronaut-inject-3.10.1.jar:3.10.1]
	at io.micronaut.context.SingletonScope.getOrCreate(SingletonScope.java:80) ~[micronaut-inject-3.10.1.jar:3.10.1]
<snip>

I can't find much information on this error, or any existing issues etc, so I've hit a bit of a wall with troubleshooting. Does anyone have any pointers?

[joeybenamy]

Did you get this working? What did you do?

[adamstrawson]

Sorry, I thought I posted a comment with the solution, but seems I didn't hit the button!

There's an open bug for this, effecting the latest version: #31988

Workaround is to roll back to 0.49.10, which solved it for us in the short term.

[joeybenamy]

Thanks so much! We are doing exactly the same as you -- I'm getting ramped up on Airbyte so the larger data team can use it.

[jensenity]

Getting affected from this too. works on 0.47 but started failing since 0.50