How to handle JWKS timeouts

[sebgott]

I have strimzi running with oauth authentication using EntraID. Sometimes Kafka times out when fetching public keys fromt he JWKS endpoint and I have been wondering what is the correct way to handle this. I have a canary application running in my cluster constantly producing and consuming. Over the weekend it failed 233 roundtrips (out of 32632 total). My current thought was to decrease the jwks refresh timer with this authentication config which seemed to have decreased the total failures of my canary:

        authentication:
          type: custom
          sasl: true
          listenerConfig:
            sasl.enabled.mechanisms: OAUTHBEARER
            oauthbearer.sasl.server.callback.handler.class: io.strimzi.kafka.oauth.server.JaasServerOauthValidatorCallbackHandler
            oauthbearer.sasl.jaas.config: >
              org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required
              oauth.valid.issuer.uri="https://login.microsoftonline.com/<tenant-id>/v2.0"
              oauth.jwks.endpoint.uri="https://login.microsoftonline.com/<tenant-id>/discovery/v2.0/keys"
              oauth.jwks.refresh.seconds="100"
              oauth.username.claim="preferred_username"
              oauth.check.access.token.type="false"
              unsecuredLoginStringClaim_sub="unused";
            connections.max.reauth.ms: 3600

The best option would probably be to find out why it sometimes times out, but does anyone have any ideas or suggestion?

This is the error the brokers gets when they time out getting the public keys:

org.apache.kafka.common.KafkaException: io.strimzi.kafka.oauth.services.ServiceException: Failed to fetch public keys needed to validate JWT signatures: https://login.microsoftonline.com/<tenant-id>/discovery/v2.0/keys
    at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:184) ~[kafka-clients-4.1.0.jar:?]
    at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:188) ~[kafka-clients-4.1.0.jar:?]
    at org.apache.kafka.common.network.ChannelBuilders.serverChannelBuilder(ChannelBuilders.java:105) ~[kafka-clients-4.1.0.jar:?]
    at kafka.network.Processor.<init>(SocketServer.scala:876) ~[kafka_2.13-4.1.0.jar:?]
    at kafka.network.Acceptor.newProcessor(SocketServer.scala:784) ~[kafka_2.13-4.1.0.jar:?]
    at kafka.network.Acceptor.$anonfun$addProcessors$1(SocketServer.scala:750) ~[kafka_2.13-4.1.0.jar:?]
    at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:192) ~[scala-library-2.13.16.jar:?]
    at kafka.network.Acceptor.addProcessors(SocketServer.scala:749) ~[kafka_2.13-4.1.0.jar:?]
    at kafka.network.DataPlaneAcceptor.configure(SocketServer.scala:467) ~[kafka_2.13-4.1.0.jar:?]
    at kafka.network.SocketServer.createDataPlaneAcceptorAndProcessors(SocketServer.scala:222) ~[kafka_2.13-4.1.0.jar:?]
    at kafka.network.SocketServer.$anonfun$new$16(SocketServer.scala:148) ~[kafka_2.13-4.1.0.jar:?]
    at kafka.network.SocketServer.$anonfun$new$16$adapted(SocketServer.scala:148) ~[kafka_2.13-4.1.0.jar:?]
    at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:619) ~[scala-library-2.13.16.jar:?]
    at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:617) ~[scala-library-2.13.16.jar:?]
    at scala.collection.AbstractIterable.foreach(Iterable.scala:935) ~[scala-library-2.13.16.jar:?]
    at kafka.network.SocketServer.<init>(SocketServer.scala:148) ~[kafka_2.13-4.1.0.jar:?]
    at kafka.server.BrokerServer.startup(BrokerServer.scala:281) ~[kafka_2.13-4.1.0.jar:?]
    at kafka.server.KafkaRaftServer.$anonfun$startup$2(KafkaRaftServer.scala:95) ~[kafka_2.13-4.1.0.jar:?]
    at kafka.server.KafkaRaftServer.$anonfun$startup$2$adapted(KafkaRaftServer.scala:95) ~[kafka_2.13-4.1.0.jar:?]
    at scala.Option.foreach(Option.scala:437) ~[scala-library-2.13.16.jar:?]
    at kafka.server.KafkaRaftServer.startup(KafkaRaftServer.scala:95) ~[kafka_2.13-4.1.0.jar:?]
    at kafka.Kafka$.main(Kafka.scala:97) [kafka_2.13-4.1.0.jar:?]
    at kafka.Kafka.main(Kafka.scala) [kafka_2.13-4.1.0.jar:?]
Caused by: io.strimzi.kafka.oauth.services.ServiceException: Failed to fetch public keys needed to validate JWT signatures: https://login.microsoftonline.com/<tenant-id>/discovery/v2.0/keys
    at io.strimzi.kafka.oauth.validator.JWTSignatureValidator.fetchKeys(JWTSignatureValidator.java:418) ~[kafka-oauth-common-0.17.1.jar:?]
    at io.strimzi.kafka.oauth.validator.JWTSignatureValidator.setupExecutorAndFetchInitialKeys(JWTSignatureValidator.java:281) ~[kafka-oauth-common-0.17.1.jar:?]
    at io.strimzi.kafka.oauth.validator.JWTSignatureValidator.<init>(JWTSignatureValidator.java:198) ~[kafka-oauth-common-0.17.1.jar:?]
    at io.strimzi.kafka.oauth.server.JaasServerOauthValidatorCallbackHandler.lambda$setupJWKSValidator$1(JaasServerOauthValidatorCallbackHandler.java:498) ~[kafka-oauth-server-0.17.1.jar:?]
    at io.strimzi.kafka.oauth.services.Validators.get(Validators.java:45) ~[kafka-oauth-common-0.17.1.jar:?]
    at io.strimzi.kafka.oauth.server.JaasServerOauthValidatorCallbackHandler.setupJWKSValidator(JaasServerOauthValidatorCallbackHandler.java:524) ~[kafka-oauth-server-0.17.1.jar:?]
    at io.strimzi.kafka.oauth.server.JaasServerOauthValidatorCallbackHandler.delegatedConfigure(JaasServerOauthValidatorCallbackHandler.java:310) ~[kafka-oauth-server-0.17.1.jar:?]
    at io.strimzi.kafka.oauth.server.JaasServerOauthValidatorCallbackHandler.configure(JaasServerOauthValidatorCallbackHandler.java:238) ~[kafka-oauth-server-0.17.1.jar:?]
    at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:151) ~[kafka-clients-4.1.0.jar:?]
    ... 22 more
Caused by: java.net.SocketTimeoutException: Connect timed out
    at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:546) ~[?:?]
    at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:592) ~[?:?]
    at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327) ~[?:?]
    at java.base/java.net.Socket.connect(Socket.java:751) ~[?:?]
    at java.base/sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:304) ~[?:?]
    at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:178) ~[?:?]
    at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:531) ~[?:?]
    at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:636) ~[?:?]
    at java.base/sun.net.www.protocol.https.HttpsClient.<init>(HttpsClient.java:264) ~[?:?]
    at java.base/sun.net.www.protocol.https.HttpsClient.New(HttpsClient.java:377) ~[?:?]
    at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(AbstractDelegateHttpsURLConnection.java:193) ~[?:?]
    at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1257) ~[?:?]
    at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1143) ~[?:?]
    at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:179) ~[?:?]
    at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:141) ~[?:?]
    at io.strimzi.kafka.oauth.common.HttpUtil.request(HttpUtil.java:481) ~[kafka-oauth-common-0.17.1.jar:?]
    at io.strimzi.kafka.oauth.common.HttpUtil.get(HttpUtil.java:196) ~[kafka-oauth-common-0.17.1.jar:?]
    at io.strimzi.kafka.oauth.validator.JWTSignatureValidator.fetchKeys(JWTSignatureValidator.java:385) ~[kafka-oauth-common-0.17.1.jar:?]
    at io.strimzi.kafka.oauth.validator.JWTSignatureValidator.setupExecutorAndFetchInitialKeys(JWTSignatureValidator.java:281) ~[kafka-oauth-common-0.17.1.jar:?]
    at io.strimzi.kafka.oauth.validator.JWTSignatureValidator.<init>(JWTSignatureValidator.java:198) ~[kafka-oauth-common-0.17.1.jar:?]
    at io.strimzi.kafka.oauth.server.JaasServerOauthValidatorCallbackHandler.lambda$setupJWKSValidator$1(JaasServerOauthValidatorCallbackHandler.java:498) ~[kafka-oauth-server-0.17.1.jar:?]
    at io.strimzi.kafka.oauth.services.Validators.get(Validators.java:45) ~[kafka-oauth-common-0.17.1.jar:?]
    at io.strimzi.kafka.oauth.server.JaasServerOauthValidatorCallbackHandler.setupJWKSValidator(JaasServerOauthValidatorCallbackHandler.java:524) ~[kafka-oauth-server-0.17.1.jar:?]
    at io.strimzi.kafka.oauth.server.JaasServerOauthValidatorCallbackHandler.delegatedConfigure(JaasServerOauthValidatorCallbackHandler.java:310) ~[kafka-oauth-server-0.17.1.jar:?]
    at io.strimzi.kafka.oauth.server.JaasServerOauthValidatorCallbackHandler.configure(JaasServerOauthValidatorCallbackHandler.java:238) ~[kafka-oauth-server-0.17.1.jar:?]
    at org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:151) ~[kafka-clients-4.1.0.jar:?]

[see-quick]

Well this seems to me more like network issue within your cluster or cloud provider? Maybe you could potentionally change the Oauth config to make those network issues less disturbing?

@mstruk might know?

[sebgott]

Found out that this is probably due to the firewall not allowing the whole ip range of https://login.microsoftonline.com. I will update when I know more :)

[sebgott]

The issue was the firwall not being able to update the IPs of https://login.microsoftonline.com/ fast enough, so the whole IP-range was allowed. After doing this, the issue was mitigated. From my understanding this is a common issue for networking teams face. Everything works perfectly now.

[see-quick]

Great thanks for the update and I am glad it works now 👍 .