Feedback on KYUUBI #6212 & #6233 - We faced a similar issue, Fix is found to be not working

[mahalingamagesthian]

Hi @amanraj2520 , @pan3793

I wish that you are doing really well. We have a scenario to run spark-submit with kyuubi-spark-authz-shaded jar. We have figured out that the Sparkdriver kept running after the completion of the spark programme execution. We are seeing nondaemon threads similar to #6212 & #6233 .

Inspired by this, we are very quick to download kyuubi-spark-authz-shaded-2.12.1.10.1.jar. We don't find it fixing the issue. The sparkdriver logs doesn't have a mention of the Logs in line
https://github.com/amanraj2520/kyuubi/blob/auditShutdown/extensions/spark/kyuubi-spark-authz/src/main/scala/org/apache/kyuubi/plugin/spark/authz/ranger/SparkRangerAdminPlugin.scala#L79

I doubt if the fix is working or Am I missing anything ? I am happy to provide additional information from the test scenario.

Spark-submit is a batch on K8s. Spark submit connects to Apache Ranger download the policy evaluate the policy. kyuubi-spark-authz-shaded.jar is a key module to perform this operation.

JSTACK thread:

"zkConnectionManagerCallback-5-thread-1" #113 prio=5 os_prio=0 cpu=0.90ms elapsed=26.68s tid=0x00007f1de0004000 nid=0xbd waiting on condition  [0x00007f1dfa1d6000]
   java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
        - parking to wait for  <0x00000007111bdff8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:194)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await([email protected]/AbstractQueuedSynchronizer.java:2081)
        at java.util.concurrent.LinkedBlockingQueue.take([email protected]/LinkedBlockingQueue.java:433)
        at java.util.concurrent.ThreadPoolExecutor.getTask([email protected]/ThreadPoolExecutor.java:1054)
        at java.util.concurrent.ThreadPoolExecutor.runWorker([email protected]/ThreadPoolExecutor.java:1114)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run([email protected]/ThreadPoolExecutor.java:628)
        at java.lang.Thread.run([email protected]/Thread.java:829)

[pan3793]

I think this is exactly what is explained in #6212 (comment), to quickly verify it, try disabling ranger solr audit plugin first.

[mahalingamagesthian]

Thanks @pan3793 .
It is true that if we disable ranger solr audit. Spark driver pod is completing. This issue is not happening.

Does #6233 fixes this issue ? without making the solr threads as daemon threads ?

[pan3793]

It should fix the issue. (I don't use Solr, but I get at least 3 positive reports for this case from Kyuubi users)