executors idle移除后，driver资源无法释放 #4014

imperio-wxm · 2022-12-19T11:28:54Z

imperio-wxm
Dec 19, 2022

spark on k8s
cluster模式配置了：
kyuubi.session.engine.idle.timeout=PT2M 或
kyuubi.session.engine.spark.max.lifetime=PT2M

2min后Executors全部被关闭移除了，SparkContext也关闭了，但是driver节点依然存活running状态，无法释放node资源

driver中的日志：

22/12/19 19:23:06.489 [mainhread] INFO KubernetesClusterSchedulerBackend: Shutting down all executors
22/12/19 19:23:06.489 [dispatcher-CoarseGrainedSchedulerhread] INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each executor to shut down
22/12/19 19:23:06.498 [mainhread] WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed.
22/12/19 19:23:06.887 [dispatcher-event-loop-1hread] INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
22/12/19 19:23:06.905 [mainhread] INFO MemoryStore: MemoryStore cleared
22/12/19 19:23:06.906 [mainhread] INFO BlockManager: BlockManager stopped
22/12/19 19:23:06.909 [mainhread] INFO BlockManagerMaster: BlockManagerMaster stopped
22/12/19 19:23:06.915 [dispatcher-event-loop-12hread] INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
22/12/19 19:23:06.924 [mainhread] INFO SparkContext: Successfully stopped SparkContext

Answered by pan3793

Dec 19, 2022

Please enter the Driver Pod and run jstack to check thread stacks.

View full answer

pan3793 · 2022-12-19T11:31:10Z

pan3793
Dec 19, 2022
Collaborator

Please enter the Driver Pod and run jstack to check thread stacks.

3 replies

imperio-wxm Dec 19, 2022
Author

如下为driver pod的jstack信息

Full thread dump OpenJDK 64-Bit Server VM (25.352-b08 mixed mode):

"Attach Listener" #358 daemon prio=9 os_prio=0 tid=0x00007fdab4001000 nid=0x1f6 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"DestroyJavaVM" #355 prio=5 os_prio=0 tid=0x00007fe470055000 nid=0x36 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Timer-1" #241 daemon prio=5 os_prio=0 tid=0x00007fd11c677800 nid=0x148 in Object.wait() [0x00007fbefbfcf000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at java.util.TimerThread.mainLoop(Timer.java:526)
        - locked <0x00007fdc7e23d2a0> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:505)

"pool-36-thread-6" #218 prio=5 os_prio=0 tid=0x00007fd37536d800 nid=0x131 waiting on condition [0x00007fc07bfdb000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007fdbf2825d58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

"pool-36-thread-5" #217 prio=5 os_prio=0 tid=0x00007fd375235800 nid=0x130 waiting on condition [0x00007fc09bfdc000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007fdbf2825d58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

"pool-36-thread-4" #216 prio=5 os_prio=0 tid=0x00007fd3751aa000 nid=0x12f waiting on condition [0x00007fc0bbfdd000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007fdbf2825d58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

"pool-36-thread-3" #182 prio=5 os_prio=0 tid=0x00007fd3759ee000 nid=0x10d waiting on condition [0x00007fc50bfd3000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007fdbf2825d58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

"pool-36-thread-2" #181 prio=5 os_prio=0 tid=0x00007fd375a26800 nid=0x10c waiting on condition [0x00007fc52bfd4000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007fdbf2825d58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

"pool-36-thread-1" #180 prio=5 os_prio=0 tid=0x00007fd375070000 nid=0x10b waiting on condition [0x00007fc54bfd5000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007fdbf2825d58> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

"prometheus-http-1-5" #179 daemon prio=5 os_prio=0 tid=0x00007fda14012000 nid=0x10a waiting on condition [0x00007fc56bfd6000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007fdbc476dfd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

"prometheus-http-1-4" #138 daemon prio=5 os_prio=0 tid=0x00007fda14010000 nid=0xe1 waiting on condition [0x00007fca93fdf000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007fdbc476dfd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

"prometheus-http-1-3" #131 daemon prio=5 os_prio=0 tid=0x00007fda1400e000 nid=0xda waiting on condition [0x00007fcaf3fe2000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007fdbc476dfd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

"BoneCP-pool-watch-thread" #109 daemon prio=5 os_prio=0 tid=0x00007fe473a4f800 nid=0xc4 waiting on condition [0x00007fcd93ff7000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007fdbc9ac2460> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
        at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

"BoneCP-keep-alive-scheduler" #108 daemon prio=5 os_prio=0 tid=0x00007fe473a4e000 nid=0xc3 waiting on condition [0x00007fcdb3ff8000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007fdbc9cfe608> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

"com.google.common.base.internal.Finalizer" #107 daemon prio=5 os_prio=0 tid=0x00007fe473a4c000 nid=0xc2 in Object.wait() [0x00007fcdd3ff9000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
        - locked <0x00007fdbe10031f0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
        at com.google.common.base.internal.Finalizer.run(Finalizer.java:146)
        at java.lang.Thread.run(Thread.java:750)

"BoneCP-pool-watch-thread" #106 daemon prio=5 os_prio=0 tid=0x00007fce75090800 nid=0xc1 waiting on condition [0x00007fcdf3ffa000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007fdbc990f7f8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
        at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:75)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

"BoneCP-keep-alive-scheduler" #105 daemon prio=5 os_prio=0 tid=0x00007fce75097000 nid=0xc0 waiting on condition [0x00007fce13ffb000]
   java.lang.Thread.State: TIMED_WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007fdbc9ac27c0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
        at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

"com.google.common.base.internal.Finalizer" #104 daemon prio=5 os_prio=0 tid=0x00007fce75083800 nid=0xbf in Object.wait() [0x00007fce33ffc000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
        - locked <0x00007fdbd19db610> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
        at com.google.common.base.internal.Finalizer.run(Finalizer.java:146)
        at java.lang.Thread.run(Thread.java:750)

"derby.rawStoreDaemon" #103 daemon prio=5 os_prio=0 tid=0x00007fce74db8800 nid=0xbe in Object.wait() [0x00007fce53ffd000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at org.apache.derby.impl.services.daemon.BasicDaemon.rest(Unknown Source)
        - locked <0x00007fdbd1b7a480> (a org.apache.derby.impl.services.daemon.BasicDaemon)
        at org.apache.derby.impl.services.daemon.BasicDaemon.run(Unknown Source)
        at java.lang.Thread.run(Thread.java:750)

"Timer-0" #102 daemon prio=5 os_prio=0 tid=0x00007fce74d84000 nid=0xbd in Object.wait() [0x00007fce73ffe000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at java.util.TimerThread.mainLoop(Timer.java:526)
        - locked <0x00007fdbd1b7a428> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:505)

"prometheus-http-1-2" #101 daemon prio=5 os_prio=0 tid=0x00007fda1400c800 nid=0xbc waiting on condition [0x00007fce9bffe000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007fdbc476dfd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

"prometheus-http-1-1" #99 daemon prio=5 os_prio=0 tid=0x00007fda14009800 nid=0xba waiting on condition [0x00007fceebffe000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00007fdbc476dfd0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:750)

"idle_connection_reaper" #85 daemon prio=5 os_prio=0 tid=0x00007fe470714800 nid=0xa8 waiting on condition [0x00007fd11bffe000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
        at java.lang.Thread.sleep(Native Method)
        at com.aliyun.oss.common.comm.IdleConnectionReaper.run(IdleConnectionReaper.java:78)

"org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner" #84 daemon prio=5 os_prio=0 tid=0x00007fe471d5b000 nid=0xa7 in Object.wait() [0x00007fd143ffe000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
        - locked <0x00007fdbc7008e60> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
        at org.apache.hadoop.fs.FileSystem$Statistics$StatisticsDataReferenceCleaner.run(FileSystem.java:4021)
        at java.lang.Thread.run(Thread.java:750)

"Service Thread" #23 daemon prio=9 os_prio=0 tid=0x00007fe4714cd800 nid=0x6b runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread11" #22 daemon prio=9 os_prio=0 tid=0x00007fe4714ca800 nid=0x6a waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread10" #21 daemon prio=9 os_prio=0 tid=0x00007fe4714c8800 nid=0x69 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread9" #20 daemon prio=9 os_prio=0 tid=0x00007fe4714c6800 nid=0x68 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread8" #19 daemon prio=9 os_prio=0 tid=0x00007fe4714c4800 nid=0x67 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread7" #18 daemon prio=9 os_prio=0 tid=0x00007fe4714c1800 nid=0x66 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread6" #17 daemon prio=9 os_prio=0 tid=0x00007fe4714bf000 nid=0x65 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread5" #16 daemon prio=9 os_prio=0 tid=0x00007fe4714bd000 nid=0x64 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread4" #15 daemon prio=9 os_prio=0 tid=0x00007fe4714bb000 nid=0x63 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread3" #14 daemon prio=9 os_prio=0 tid=0x00007fe4714b9000 nid=0x62 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread2" #13 daemon prio=9 os_prio=0 tid=0x00007fe4714ac800 nid=0x61 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #12 daemon prio=9 os_prio=0 tid=0x00007fe4714aa800 nid=0x60 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #11 daemon prio=9 os_prio=0 tid=0x00007fe4714a9800 nid=0x5f waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Thread-3" #10 daemon prio=5 os_prio=0 tid=0x00007fda3c001000 nid=0x5e runnable [0x00007fda3bffe000]
   java.lang.Thread.State: RUNNABLE
        at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
        at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
        at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
        at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
        - locked <0x00007fdbc47ffda0> (a sun.nio.ch.Util$3)
        - locked <0x00007fdbc47ffdb8> (a java.util.Collections$UnmodifiableSet)
        - locked <0x00007fdbc47ffd28> (a sun.nio.ch.EPollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
        at sun.net.httpserver.ServerImpl$Dispatcher.run(ServerImpl.java:453)
        at java.lang.Thread.run(Thread.java:750)

"req-rsp-timeout-task" #8 daemon prio=5 os_prio=0 tid=0x00007fe47145b000 nid=0x5c in Object.wait() [0x00007fda8bffe000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.util.TimerThread.mainLoop(Timer.java:552)
        - locked <0x00007fdbc47f4840> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:505)

"idle-timeout-task" #7 daemon prio=5 os_prio=0 tid=0x00007fe471458800 nid=0x5b in Object.wait() [0x00007fdab3ffe000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.util.TimerThread.mainLoop(Timer.java:552)
        - locked <0x00007fdbc48098c8> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:505)

"Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007fe470222800 nid=0x5a runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0 tid=0x00007fe470220800 nid=0x59 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fe4701f4800 nid=0x58 in Object.wait() [0x00007fdb2bffe000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
        - locked <0x00007fdbc47e0140> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:188)

"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fe4701ef800 nid=0x57 in Object.wait() [0x00007fdb53ffe000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:502)
        at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
        - locked <0x00007fdbc47b0f78> (a java.lang.ref.Reference$Lock)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)

"VM Thread" os_prio=0 tid=0x00007fe4701e5800 nid=0x56 runnable 

"Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007fe470068000 nid=0x37 runnable 

"Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007fe470069800 nid=0x38 runnable 

"Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007fe47006b800 nid=0x39 runnable 

"Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007fe47006d800 nid=0x3a runnable 

"Gang worker#4 (Parallel GC Threads)" os_prio=0 tid=0x00007fe47006f800 nid=0x3b runnable 

"Gang worker#5 (Parallel GC Threads)" os_prio=0 tid=0x00007fe470071000 nid=0x3c runnable 

"Gang worker#6 (Parallel GC Threads)" os_prio=0 tid=0x00007fe470073000 nid=0x3d runnable 

"Gang worker#7 (Parallel GC Threads)" os_prio=0 tid=0x00007fe470075000 nid=0x3e runnable 

"Gang worker#8 (Parallel GC Threads)" os_prio=0 tid=0x00007fe470077000 nid=0x3f runnable 

"Gang worker#9 (Parallel GC Threads)" os_prio=0 tid=0x00007fe470078800 nid=0x40 runnable 

"Gang worker#10 (Parallel GC Threads)" os_prio=0 tid=0x00007fe47007a800 nid=0x41 runnable 

"Gang worker#11 (Parallel GC Threads)" os_prio=0 tid=0x00007fe47007c800 nid=0x42 runnable 

"Gang worker#12 (Parallel GC Threads)" os_prio=0 tid=0x00007fe47007e000 nid=0x43 runnable 

"G1 Main Concurrent Mark GC Thread" os_prio=0 tid=0x00007fe4700ae000 nid=0x52 runnable 

"Gang worker#0 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fe4700b0000 nid=0x53 runnable 

"Gang worker#1 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fe4700b2000 nid=0x54 runnable 

"Gang worker#2 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fe4700b4000 nid=0x55 runnable 

"G1 Concurrent Refinement Thread#0" os_prio=0 tid=0x00007fe47009a800 nid=0x51 runnable 

"G1 Concurrent Refinement Thread#1" os_prio=0 tid=0x00007fe470098800 nid=0x50 runnable 

"G1 Concurrent Refinement Thread#2" os_prio=0 tid=0x00007fe470096800 nid=0x4f runnable 

"G1 Concurrent Refinement Thread#3" os_prio=0 tid=0x00007fe470094800 nid=0x4e runnable 

"G1 Concurrent Refinement Thread#4" os_prio=0 tid=0x00007fe470092800 nid=0x4d runnable 

"G1 Concurrent Refinement Thread#5" os_prio=0 tid=0x00007fe470091000 nid=0x4c runnable 

"G1 Concurrent Refinement Thread#6" os_prio=0 tid=0x00007fe47008f000 nid=0x4b runnable 

"G1 Concurrent Refinement Thread#7" os_prio=0 tid=0x00007fe47008d000 nid=0x4a runnable 

"G1 Concurrent Refinement Thread#8" os_prio=0 tid=0x00007fe47008b000 nid=0x49 runnable 

"G1 Concurrent Refinement Thread#9" os_prio=0 tid=0x00007fe470089000 nid=0x48 runnable 

"G1 Concurrent Refinement Thread#10" os_prio=0 tid=0x00007fe470087800 nid=0x47 runnable 

"G1 Concurrent Refinement Thread#11" os_prio=0 tid=0x00007fe470085800 nid=0x46 runnable 

"G1 Concurrent Refinement Thread#12" os_prio=0 tid=0x00007fe470083800 nid=0x45 runnable 

"G1 Concurrent Refinement Thread#13" os_prio=0 tid=0x00007fe470081800 nid=0x44 runnable 

"VM Periodic Task Thread" os_prio=0 tid=0x00007fe4714d0000 nid=0x6c waiting on condition 

JNI global references: 9510java

pan3793 Dec 19, 2022
Collaborator

Who created "pool-36-thread" pool? The driver was suspended by this thread pool

imperio-wxm Dec 19, 2022
Author

我的spark程序是包在spring框架中，pool-36-thread是手动在spring中声明的一个线程池，这个线程池的销毁放在了spring bean的@PreDestroy里面，看起来SparkSQLEngine中的stop并不能触发spring框架的shutdown，从而卸载bean

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

executors idle移除后，driver资源无法释放 #4014

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 1 comment 3 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

executors idle移除后，driver资源无法释放 #4014

Uh oh!

Uh oh!

imperio-wxm Dec 19, 2022

Replies: 1 comment · 3 replies

Uh oh!

pan3793 Dec 19, 2022 Collaborator

Uh oh!

imperio-wxm Dec 19, 2022 Author

Uh oh!

pan3793 Dec 19, 2022 Collaborator

Uh oh!

imperio-wxm Dec 19, 2022 Author

imperio-wxm
Dec 19, 2022

Replies: 1 comment 3 replies

pan3793
Dec 19, 2022
Collaborator

imperio-wxm Dec 19, 2022
Author

pan3793 Dec 19, 2022
Collaborator

imperio-wxm Dec 19, 2022
Author