Did oidc-client extension change properties to build time?

[bulldog98]

I'm using kubernetes-config and in the configmap set
quarkus.oidc-client.auth-server-url=https://validurl

This leads to

Caused by: io.smallrye.mutiny.CompositeException: Multiple exceptions caught:
[Exception 0] java.lang.RuntimeException: OIDC Server is not available
[Exception 1] io.quarkus.oidc.client.OidcClientException: OIDC server is not available at the 'https://validurl' URL. Please make sure it is correct. Note it has to end with a realm value if you work with Keycloak, for example: 'https://localhost:8180/auth/realms/quarkus'

Is it because it got made into a build time config, or is it some wierd interaction between the two extensions?

[sberyozkin]

@bulldog98 Looks like 'https://validurl' is not accessible from within the pod running quarkus-oidc-client

[bulldog98]

I Had another pod curl IT and IT worked

[sberyozkin]

@bulldog98 What happens if you try to run it outside Kubernetes ?

[sberyozkin]

@bulldog98 Note, .well-known/openid-configuration should be resolvable against this URL

[bulldog98]

I use the same url for devmode, and there it works

[sberyozkin]

@bulldog98 Can you please try redeploying with this address set in application properties as opposed to config maps ? I wonder if it might be related, also please enable oidc client logs

[bulldog98]

I could not find the documentation on how to enable the oidc client logs, is that a quarkus thing or do I have to enable them in the oidc provider (in my case keycloak)?

[bulldog98]

This is what is getting logged with the quarkus.oicd-client properties set in application.properties (they are still also in the configmap and secret)

Jul 13, 2023 6:07:53 AM io.quarkus.oidc.common.runtime.OidcCommonUtils
WARN: Discovery has failed, status code: %d
Jul 13, 2023 6:07:53 AM io.quarkus.oidc.common.runtime.OidcCommonUtils
WARN: OIDC Server is not available:
io.quarkus.oidc.common.runtime.OidcEndpointAccessException
 at io.quarkus.oidc.common.runtime.OidcCommonUtils.lambda$discoverMetadata$2(OidcCommonUtils.java:425)
 at io.smallrye.context.impl.wrappers.SlowContextualFunction.apply(SlowContextualFunction.java:21)
 at io.smallrye.mutiny.operators.uni.UniOnItemTransform$UniOnItemTransformProcessor.onItem(UniOnItemTransform.java:36)
 at io.smallrye.mutiny.vertx.AsyncResultUni.lambda$subscribe$1(AsyncResultUni.java:35)
 at io.smallrye.mutiny.vertx.DelegatingHandler.handle(DelegatingHandler.java:25)
 at io.vertx.ext.web.client.impl.HttpContext.handleDispatchResponse(HttpContext.java:397)
 at io.vertx.ext.web.client.impl.HttpContext.execute(HttpContext.java:384)
 at io.vertx.ext.web.client.impl.HttpContext.next(HttpContext.java:362)
 at io.vertx.ext.web.client.impl.HttpContext.fire(HttpContext.java:329)
 at io.vertx.ext.web.client.impl.HttpContext.dispatchResponse(HttpContext.java:291)
 at io.vertx.ext.web.client.impl.HttpContext.lambda$null$7(HttpContext.java:507)
 at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:264)
 at io.vertx.core.impl.ContextInternal.dispatch(ContextInternal.java:246)
 at io.vertx.core.impl.EventLoopContext.lambda$runOnContext$0(EventLoopContext.java:43)
 at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
 at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
 at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566)
 at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
 at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
 at [email protected]/java.lang.Thread.run(Thread.java:833)
 at org.graalvm.nativeimage.builder/com.oracle.svm.core.thread.PlatformThreads.threadStartRoutine(PlatformThreads.java:775)
 at org.graalvm.nativeimage.builder/com.oracle.svm.core.posix.thread.PosixPlatformThreads.pthreadStartRoutine(PosixPlatformThreads.java:203)
Jul 13, 2023 6:07:53 AM io.quarkus.arc.impl.ArcContainerImpl shutdown
WARN: An error occurred during delivery of the @Destroyed(ApplicationScoped.class) event
jakarta.enterprise.inject.CreationException: Error creating synthetic bean [2d6aec61168fd09bfddb12d2d84a7c6aacdd2759]: jakarta.enterprise.inject.CreationException: Synthetic bean instance for io.vertx.core.Vertx not initialized yet: io_vertx_core_Vertx_74affa450965de1cd6a7217ce7ce0dbad2c16185
 - a synthetic bean initialized during RUNTIME_INIT must not be accessed during STATIC_INIT
 - RUNTIME_INIT build steps that require access to synthetic beans initialized during RUNTIME_INIT should consume the SyntheticBeansRuntimeInitBuildItem
 at io.vertx.core.Vertx_2d6aec61168fd09bfddb12d2d84a7c6aacdd2759_Synthetic_Bean.doCreate(Unknown Source)
 at io.vertx.core.Vertx_2d6aec61168fd09bfddb12d2d84a7c6aacdd2759_Synthetic_Bean.create(Unknown Source)
 at io.vertx.core.Vertx_2d6aec61168fd09bfddb12d2d84a7c6aacdd2759_Synthetic_Bean.create(Unknown Source)
 at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:113)
 at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:37)
 at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:34)
 at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:26)
 at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69)
 at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:34)
 at io.vertx.core.Vertx_2d6aec61168fd09bfddb12d2d84a7c6aacdd2759_Synthetic_Bean.get(Unknown Source)
 at io.vertx.core.Vertx_2d6aec61168fd09bfddb12d2d84a7c6aacdd2759_Synthetic_Bean.get(Unknown Source)
 at io.quarkus.vertx.runtime.VertxProducer_ProducerMethod_mutiny_d5befbd244a8a884fd08fff108d174c7e738c2d3_Bean.doCreate(Unknown Source)
 at io.quarkus.vertx.runtime.VertxProducer_ProducerMethod_mutiny_d5befbd244a8a884fd08fff108d174c7e738c2d3_Bean.create(Unknown Source)
 at io.quarkus.vertx.runtime.VertxProducer_ProducerMethod_mutiny_d5befbd244a8a884fd08fff108d174c7e738c2d3_Bean.create(Unknown Source)
 at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:113)
 at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:37)
 at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:34)
 at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:26)
 at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69)
 at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:34)
 at io.quarkus.vertx.runtime.VertxProducer_ProducerMethod_mutiny_d5befbd244a8a884fd08fff108d174c7e738c2d3_Bean.get(Unknown Source)
 at io.quarkus.vertx.runtime.VertxProducer_ProducerMethod_mutiny_d5befbd244a8a884fd08fff108d174c7e738c2d3_Bean.get(Unknown Source)
 at io.quarkus.arc.impl.InstanceImpl.getBeanInstance(InstanceImpl.java:325)
 at io.quarkus.arc.impl.InstanceImpl.getInternal(InstanceImpl.java:309)
 at io.quarkus.arc.impl.InstanceImpl.get(InstanceImpl.java:190)
 at io.smallrye.reactive.messaging.providers.connectors.ExecutionHolder.<init>(ExecutionHolder.java:49)
 at io.smallrye.reactive.messaging.providers.connectors.ExecutionHolder_Bean.doCreate(Unknown Source)
 at io.smallrye.reactive.messaging.providers.connectors.ExecutionHolder_Bean.create(Unknown Source)
 at io.smallrye.reactive.messaging.providers.connectors.ExecutionHolder_Bean.create(Unknown Source)
 at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:113)
 at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:37)
 at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:34)
 at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:26)
 at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69)
 at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:34)
 at io.quarkus.arc.impl.ClientProxies.getApplicationScopedDelegate(ClientProxies.java:21)
 at io.smallrye.reactive.messaging.providers.connectors.ExecutionHolder_ClientProxy.arc$delegate(Unknown Source)
 at io.smallrye.reactive.messaging.providers.connectors.ExecutionHolder_ClientProxy.vertx(Unknown Source)
 at io.smallrye.reactive.messaging.mqtt.MqttConnector.init(MqttConnector.java:83)
 at io.smallrye.reactive.messaging.mqtt.MqttConnector_Bean.doCreate(Unknown Source)
 at io.smallrye.reactive.messaging.mqtt.MqttConnector_Bean.create(Unknown Source)
 at io.smallrye.reactive.messaging.mqtt.MqttConnector_Bean.create(Unknown Source)
 at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:113)
 at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:37)
 at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:34)
 at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:26)
 at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69)
 at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:34)
 at io.smallrye.reactive.messaging.mqtt.MqttConnector_Observer_destroy_cf5503d2c17ad56787516cb124dce9dd5004f013.notify(Unknown Source)
 at io.quarkus.arc.impl.EventImpl$Notifier.notifyObservers(EventImpl.java:346)
 at io.quarkus.arc.impl.EventImpl$Notifier.notify(EventImpl.java:334)
 at io.quarkus.arc.impl.EventImpl$Notifier.notify(EventImpl.java:263)
 at io.quarkus.arc.impl.ArcContainerImpl.shutdown(ArcContainerImpl.java:464)
 at io.quarkus.arc.Arc.shutdown(Arc.java:66)
 at io.quarkus.arc.runtime.ArcRecorder$1.run(ArcRecorder.java:53)
 at io.quarkus.runtime.StartupContext.runAllInReverseOrder(StartupContext.java:84)
 at io.quarkus.runtime.StartupContext.close(StartupContext.java:73)
 at io.quarkus.runner.ApplicationImpl.doStart(Unknown Source)
 at io.quarkus.runtime.Application.start(Application.java:101)
 at io.quarkus.runtime.ApplicationLifecycleManager.run(ApplicationLifecycleManager.java:111)
 at io.quarkus.runtime.Quarkus.run(Quarkus.java:71)
 at io.quarkus.runtime.Quarkus.run(Quarkus.java:44)
 at io.quarkus.runtime.Quarkus.run(Quarkus.java:124)
 at io.quarkus.runner.GeneratedMain.main(Unknown Source)
Caused by: jakarta.enterprise.inject.CreationException: Synthetic bean instance for io.vertx.core.Vertx not initialized yet: io_vertx_core_Vertx_74affa450965de1cd6a7217ce7ce0dbad2c16185

[bulldog98]

Not changed at all, when I remove the properties from the k8s resources read

[bulldog98]

Ok now I'm confused. I tried sudo nsenter -t $PID_OF_QUARKUS_CONTAINER -n curl --insecure https://validurl on the worker and got back
curl: (6) Could not resolve host: validurlhost

I did not change anything in the cluster since yesterday, that should work

[sberyozkin]

@bulldog98 I'm not sure what is going on but if curl also fails then it only confirms there is at least intermediate problem related to accessing this host from the cluster.
Apart from that, there are a few interesting exceptions:

jakarta.enterprise.inject.CreationException: Error creating synthetic bean [2d6aec61168fd09bfddb12d2d84a7c6aacdd2759]: jakarta.enterprise.inject.CreationException: Synthetic bean instance for io.vertx.core.Vertx not initialized yet: io_vertx_core_Vertx_74affa450965de1cd6a7217ce7ce0dbad2c16185
 - a synthetic bean initialized during RUNTIME_INIT must not be accessed during STATIC_INIT
 - RUNTIME_INIT build steps that require access to synthetic beans initialized during RUNTIME_INIT should consume the SyntheticBeansRuntimeInitBuildItem
 at io.vertx.core.Vertx_2d6aec61168fd09bfddb12d2d84a7c6aacdd2759_Synthetic_Bean.doCreate(Unknown Source)
 at io.vertx.core.Vertx_2d6aec61168fd09bfddb12d2d84a7c6aacdd2759_Synthetic_Bean.create(Unknown Source)
 at io.vertx.core.Vertx_2d6aec61168fd09bfddb12d2d84a7c6aacdd2759_Synthetic_Bean.create(Unknown Source)
 at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:113)
 at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:37)
 at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:34)
 at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:26)
 at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69)
 at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:34)
 at io.vertx.core.Vertx_2d6aec61168fd09bfddb12d2d84a7c6aacdd2759_Synthetic_Bean.get(Unknown Source)
 at io.vertx.core.Vertx_2d6aec61168fd09bfddb12d2d84a7c6aacdd2759_Synthetic_Bean.get(Unknown Source)
 at io.quarkus.vertx.runtime.VertxProducer_ProducerMethod_mutiny_d5befbd244a8a884fd08fff108d174c7e738c2d3_Bean.doCreate(Unknown Source)
 at io.quarkus.vertx.runtime.VertxProducer_ProducerMethod_mutiny_d5befbd244a8a884fd08fff108d174c7e738c2d3_Bean.create(Unknown Source)
 at io.quarkus.vertx.runtime.VertxProducer_ProducerMethod_mutiny_d5befbd244a8a884fd08fff108d174c7e738c2d3_Bean.create(Unknown Source)
 at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:113)
 at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:37)
 at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:34)
 at io.quarkus.arc.impl.LazyValue.get(LazyValue.java:26)
 at io.quarkus.arc.impl.ComputingCache.computeIfAbsent(ComputingCache.java:69)
 at io.quarkus.arc.impl.AbstractSharedContext.get(AbstractSharedContext.java:34)
 at io.quarkus.vertx.runtime.VertxProducer_ProducerMethod_mutiny_d5befbd244a8a884fd08fff108d174c7e738c2d3_Bean.get(Unknown Source)
 at io.quarkus.vertx.runtime.VertxProducer_ProducerMethod_mutiny_d5befbd244a8a884fd08fff108d174c7e738c2d3_Bean.get(Unknown Source)
 at io.quarkus.arc.impl.InstanceImpl.getBeanInstance(InstanceImpl.java:325)
 at io.quarkus.arc.impl.InstanceImpl.getInternal(InstanceImpl.java:309)
 at io.quarkus.arc.impl.InstanceImpl.get(InstanceImpl.java:190)
 at io.smallrye.reactive.messaging.providers.connectors.ExecutionHolder.<init>(ExecutionHolder.java:49)
 at io.smallrye.reactive.messaging.providers.connectors.ExecutionHolder_Bean.doCreate(Unknown Source)
 at io.smallrye.reactive.messaging.providers.connectors.ExecutionHolder_Bean.create(Unknown Source)
 at io.smallrye.reactive.messaging.providers.connectors.ExecutionHolder_Bean.create(Unknown Source)
 at io.quarkus.arc.impl.AbstractSharedContext.createInstanceHandle(AbstractSharedContext.java:113)
 at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:37)
 at io.quarkus.arc.impl.AbstractSharedContext$1.get(AbstractSharedContext.java:34)

You may want to create an issue with more related details and tag Ozan and Clement, @ozangunalp, @cescoffier, to discuss it.

And in OidcCommonUtils:

WARN: Discovery has failed, status code: %d

which is coming from

final String discoveryUrl = authServerUrl + OidcConstants.WELL_KNOWN_CONFIGURATION;
        return client.getAbs(discoveryUrl).send().onItem().transform(resp -> {
            if (resp.statusCode() == 200) {
                return resp.bodyAsJsonObject();
            } else {
                LOG.warnf("Discovery has failed, status code: %d", resp.statusCode());
                throw new OidcEndpointAccessException(resp.statusCode());
            }
        }

Which is strange since HTTP response is available - and therefore it must have a status code - looks like a logging bug to me, resp.statusCode() is ignored - it is an integer so some kind of value, even 0, should be reported, I'll replace the use of JBoss log with io.quarkus.logging.Log

[bulldog98]

Thanks @sberyozkin I tried a known good container and that also failed, so I'll be off debugging networking in k8s 😿

[sberyozkin]

@bulldog98 Np,

So I'll be off debugging networking in k8s

Best of luck there :-)

By the way, I've confirmed locally that OidcCommonUtils is correctly logging the status code, example,

2023-07-13 11:35:44,754 WARN  [io.qua.oid.com.run.OidcCommonUtils] (vert.x-eventloop-thread-1) Discovery has failed, status code: 404

That fact the status code is not logged, despite the value obviously being available, in your setup, is strange, but I guess it can be hard to isolate why it might be the case, might be a native related problem

[bulldog98]

Found the problem, the nodes had search lan com in resolve.conf which resulted in url.com as search result for url and guess what our url was not .com