"CrashLoopBackOff" issue after injecting Linkerd.

[nm158b]

I have been working on injecting linkerd for one of the service which has three replica sets and also utilizes AKKA for inter-communication (which may be conflicting with linkerd).

The deployment will create three pods and when it tries to inject linkerd, only one pod keeps on restarting with **CrashLoopBackOff ** status. and the other two have Pendin * status.

>>kubectl get pods
NAME                                   READY   STATUS             RESTARTS   AGE
confidencescore-2-0-578b87dcfd-gjmqp   0/2     Pending            0          22h
confidencescore-2-0-578b87dcfd-kghvd   1/2     CrashLoopBackOff   224        22h
confidencescore-2-0-578b87dcfd-nmzzp   0/2     Pending            0          22h

>>kubectl describe pod
State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    255
Events:
  Type     Reason   Age                     From     Message
  ----     ------   ----                    ----     -------
  Normal   Pulling  29m (x219 over 22h)     kubelet  Pulling image "atthaloeceacr.azurecr.io/com.att.haloece/confidencescore:2.0.100"
  Warning  BackOff  3m50s (x5218 over 22h)  kubelet  Back-off restarting failed container

Logs from one of the failing pod with CrashLoopBackOff error

>> kubectl logs confidencescore-2-0-578b87dcfd-kghvd  -c confidencescore
Error in stage [com.att.cdo.security.halo.components.MessageRouterSource@6204c108]: This consumer has already been closed.
java.lang.IllegalStateException: This consumer has already been closed.
	at org.apache.kafka.clients.consumer.KafkaConsumer.ensureNotClosed(KafkaConsumer.java:1659)
	at org.apache.kafka.clients.consumer.KafkaConsumer.acquire(KafkaConsumer.java:1670)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1031)
	at com.att.cdosec.scalautils.dmaapmr.KafkaMessageRouterClient.$anonfun$fetch$1(KafkaMessageRouterClient.scala:56)
	at scala.util.Try$.apply(Try.scala:209)
	at com.att.cdosec.scalautils.dmaapmr.KafkaMessageRouterClient.fetch(KafkaMessageRouterClient.scala:55)
	at com.att.cdo.security.halo.components.MessageRouterSource$$anon$1$$anon$2.onPull(MessageRouterSource.scala:33)
	at akka.stream.impl.fusing.GraphInterpreter.processPull(GraphInterpreter.scala:532)
	at akka.stream.impl.fusing.GraphInterpreter.processEvent(GraphInterpreter.scala:484)
	at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:376)
	at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:599)
	at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:480)
	at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:574)
	at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:742)
	at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$shortCircuitBatch(ActorGraphInterpreter.scala:732)
	at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:758)
	at akka.actor.Actor.aroundReceive(Actor.scala:539)
	at akka.actor.Actor.aroundReceive$(Actor.scala:537)
	at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:664)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:610)
	at akka.actor.ActorCell.invoke(ActorCell.scala:579)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268)
	at akka.dispatch.Mailbox.run(Mailbox.scala:229)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:241)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2021-09-14 14:01:42 - WARN  a.s.s.RestartWithBackoffSource [default-akka.actor.default-dispatcher-12] 
Restarting graph due to failure. stack_trace: 
java.lang.IllegalStateException: This consumer has already been closed.
	at org.apache.kafka.clients.consumer.KafkaConsumer.ensureNotClosed(KafkaConsumer.java:1659)
	at org.apache.kafka.clients.consumer.KafkaConsumer.acquire(KafkaConsumer.java:1670)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1031)
	at com.att.cdosec.scalautils.dmaapmr.KafkaMessageRouterClient.$anonfun$fetch$1(KafkaMessageRouterClient.scala:56)
	at scala.util.Try$.apply(Try.scala:209)
	at com.att.cdosec.scalautils.dmaapmr.KafkaMessageRouterClient.fetch(KafkaMessageRouterClient.scala:55)
	at com.att.cdo.security.halo.components.MessageRouterSource$$anon$1$$anon$2.onPull(MessageRouterSource.scala:33)
	at akka.stream.impl.fusing.GraphInterpreter.processPull(GraphInterpreter.scala:532)
	at akka.stream.impl.fusing.GraphInterpreter.processEvent(GraphInterpreter.scala:484)
	at akka.stream.impl.fusing.GraphInterpreter.execute(GraphInterpreter.scala:376)
	at akka.stream.impl.fusing.GraphInterpreterShell.runBatch(ActorGraphInterpreter.scala:599)
	at akka.stream.impl.fusing.GraphInterpreterShell$AsyncInput.execute(ActorGraphInterpreter.scala:480)
	at akka.stream.impl.fusing.GraphInterpreterShell.processEvent(ActorGraphInterpreter.scala:574)
	at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$processEvent(ActorGraphInterpreter.scala:742)
	at akka.stream.impl.fusing.ActorGraphInterpreter.akka$stream$impl$fusing$ActorGraphInterpreter$$shortCircuitBatch(ActorGraphInterpreter.scala:732)
	at akka.stream.impl.fusing.ActorGraphInterpreter$$anonfun$receive$1.applyOrElse(ActorGraphInterpreter.scala:758)
	at akka.actor.Actor.aroundReceive(Actor.scala:539)
	at akka.actor.Actor.aroundReceive$(Actor.scala:537)
	at akka.stream.impl.fusing.ActorGraphInterpreter.aroundReceive(ActorGraphInterpreter.scala:664)
	at akka.actor.ActorCell.receiveMessage(ActorCell.scala:610)
	at akka.actor.ActorCell.invoke(ActorCell.scala:579)
	at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:268)
	at akka.dispatch.Mailbox.run(Mailbox.scala:229)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:241)
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
2021-09-14 14:01:42 - WARN  akka.cluster.sharding.ShardRegion [default-akka.actor.default-dispatcher-13] 
User: No coordinator found to register. Probably, no seed-nodes configured and manual cluster join not performed? Total [995] buffered messages.
2021-09-14 14:01:43 - DEBUG c.a.c.s.h.c.UpdateActorState$ [default-akka.actor.default-dispatcher-13] 
Updated state for 1 events in 1ms
2021-09-14 14:01:43 - WARN  akka.stream.Materializer [default-akka.actor.default-dispatcher-13] 
[outbound connection to [akka://[email protected]:2551], message stream] Upstream failed, cause: StreamTcpException: The connection has been aborted
2021-09-14 14:01:43 - WARN  akka.stream.Materializer [default-akka.actor.default-dispatcher-13] 
[outbound connection to [akka://[email protected]:2551], message stream] Upstream failed, cause: StreamTcpException: The connection has been aborted
2021-09-14 14:01:43 - ERROR a.r.a.Association(akka://default) [default-akka.actor.default-dispatcher-13] 
Outbound message stream to [akka://[email protected]:2551] failed. Restarting it. The connection has been aborted
akka.stream.StreamTcpException: The connection has been aborted
2021-09-14 14:01:43 - ERROR a.r.a.Association(akka://default) [default-akka.actor.default-dispatcher-13] 
Outbound message stream to [akka://[email protected]:2551] failed. Restarting it. The connection has been aborted

>> kubectl logs confidencescore-2-0-578b87dcfd-kghvd  -c comfidencescore
time="2021-09-13T15:41:06Z" level=info msg="running version stable-2.10.2"
[     0.002957s]  INFO ThreadId(01) linkerd2_proxy::rt: Using single-threaded proxy runtime
[     0.003871s]  INFO ThreadId(01) linkerd2_proxy: Admin interface on 0.0.0.0:4191
[     0.003893s]  INFO ThreadId(01) linkerd2_proxy: Inbound interface on 0.0.0.0:4143
[     0.003897s]  INFO ThreadId(01) linkerd2_proxy: Outbound interface on 127.0.0.1:4140
[     0.003902s]  INFO ThreadId(01) linkerd2_proxy: Tap interface on 0.0.0.0:4190
[     0.003906s]  INFO ThreadId(01) linkerd2_proxy: Local identity is default.com-att-kepler-haloece-test.serviceaccount.identity.linkerd.cluster.local
[     0.003911s]  INFO ThreadId(01) linkerd2_proxy: Identity verified via linkerd-identity-headless.linkerd.svc.cluster.local:8080 (linkerd-identity.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.003916s]  INFO ThreadId(01) linkerd2_proxy: Destinations resolved via linkerd-dst-headless.linkerd.svc.cluster.local:8086 (linkerd-destination.linkerd.serviceaccount.identity.linkerd.cluster.local)
[     0.041443s]  INFO ThreadId(02) daemon:identity: linkerd_app: Certified identity: default.com-att-kepler-haloece-test.serviceaccount.identity.linkerd.cluster.local
[ 13649.905513s]  INFO ThreadId(01) linkerd_app_core::serve: Connection closed error=request timed out
[ 35768.304384s]  INFO ThreadId(01) linkerd_app_core::serve: Connection closed error=request timed out
[ 48370.292659s]  INFO ThreadId(01) linkerd_app_core::serve: Connection closed error=request timed out
[ 56066.011375s]  INFO ThreadId(01) linkerd_app_core::serve: Connection closed error=request timed out
[ 78192.267334s]  INFO ThreadId(01) linkerd_app_core::serve: Connection closed error=request timed out

Any help would be appreciated!

[adleong]

Hi @nm158b! You may want to try marking the kafka port as opaque to tell Linkerd to not do protocol detection on this traffic: https://linkerd.io/2.10/features/protocol-detection/