You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
This repository was archived by the owner on Jan 9, 2020. It is now read-only.
Allow spark driver find shuffle pods in specified namespace (#357)
The conf property spark.kubernetes.shuffle.namespace is used to
specify the namesapce of shuffle pods.
In normal cases, only one "shuffle daemonset" is deployed and
shared by all spark pods.
The spark driver should be able to list and watch shuffle pods
in the namespace specified by user.
Note: by default, spark driver pod doesn't have authority to
list and watch shuffle pods in another namespace. Some action
is needed to grant it the authority. For example, below ABAC
policy works.
```
{"apiVersion": "abac.authorization.kubernetes.io/v1beta1", "kind":
"Policy", "spec": {"group": "system:serviceaccounts", "namespace":
"SHUFFLE_NAMESPACE",
"resource": "pods", "readonly": true}}
```
Copy file name to clipboardExpand all lines: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/kubernetes/ShufflePodCache.scala
+3-1Lines changed: 3 additions & 1 deletion
Original file line number
Diff line number
Diff line change
@@ -37,7 +37,8 @@ private[spark] class ShufflePodCache (
0 commit comments