You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[SPARK-25543][K8S] Print debug message iff execIdsRemovedInThisRound is not empty.
## What changes were proposed in this pull request?
Spurious logs like /sec.
2018-09-26 09:33:57 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors with ids from Spark that were either found to be deleted or non-existent in the cluster.
2018-09-26 09:33:58 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors with ids from Spark that were either found to be deleted or non-existent in the cluster.
2018-09-26 09:33:59 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors with ids from Spark that were either found to be deleted or non-existent in the cluster.
2018-09-26 09:34:00 DEBUG ExecutorPodsLifecycleManager:58 - Removed executors with ids from Spark that were either found to be deleted or non-existent in the cluster.
The fix is easy, first check if there are any removed executors, before producing the log message.
## How was this patch tested?
Tested by manually deploying to a minikube cluster.
Closesapache#22565 from ScrapCodes/spark-25543/k8s/debug-log-spurious-warning.
Authored-by: Prashant Sharma <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
Copy file name to clipboardExpand all lines: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/ExecutorPodsLifecycleManager.scala
+5-2Lines changed: 5 additions & 2 deletions
Original file line number
Diff line number
Diff line change
@@ -100,8 +100,11 @@ private[spark] class ExecutorPodsLifecycleManager(
100
100
}
101
101
}
102
102
}
103
-
logDebug(s"Removed executors with ids ${execIdsRemovedInThisRound.mkString(",")}"+
104
-
s" from Spark that were either found to be deleted or non-existent in the cluster.")
103
+
104
+
if (execIdsRemovedInThisRound.nonEmpty) {
105
+
logDebug(s"Removed executors with ids ${execIdsRemovedInThisRound.mkString(",")}"+
106
+
s" from Spark that were either found to be deleted or non-existent in the cluster.")
0 commit comments