Skip to content
This repository was archived by the owner on Jan 9, 2020. It is now read-only.

Commit a6291c6

Browse files
Hong Zhiguoash211
authored andcommitted
Allow spark driver find shuffle pods in specified namespace (#357)
The conf property spark.kubernetes.shuffle.namespace is used to specify the namesapce of shuffle pods. In normal cases, only one "shuffle daemonset" is deployed and shared by all spark pods. The spark driver should be able to list and watch shuffle pods in the namespace specified by user. Note: by default, spark driver pod doesn't have authority to list and watch shuffle pods in another namespace. Some action is needed to grant it the authority. For example, below ABAC policy works. ``` {"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "kind": "Policy", "spec": {"group": "system:serviceaccounts", "namespace": "SHUFFLE_NAMESPACE", "resource": "pods", "readonly": true}} ```
1 parent fdd50f1 commit a6291c6

File tree

1 file changed

+3
-1
lines changed
  • resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/kubernetes

1 file changed

+3
-1
lines changed

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/kubernetes/ShufflePodCache.scala

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,8 @@ private[spark] class ShufflePodCache (
3737

3838
def start(): Unit = {
3939
// seed the initial cache.
40-
val pods = client.pods().withLabels(dsLabels.asJava).list()
40+
val pods = client.pods()
41+
.inNamespace(dsNamespace).withLabels(dsLabels.asJava).list()
4142
pods.getItems.asScala.foreach {
4243
pod =>
4344
if (Readiness.isReady(pod)) {
@@ -50,6 +51,7 @@ private[spark] class ShufflePodCache (
5051

5152
watcher = client
5253
.pods()
54+
.inNamespace(dsNamespace)
5355
.withLabels(dsLabels.asJava)
5456
.watch(new Watcher[Pod] {
5557
override def eventReceived(action: Watcher.Action, p: Pod): Unit = {

0 commit comments

Comments
 (0)