Current spark.app.name is too long for K8S environment. Set spark.kubernetes.driver.resourceNamePrefix when master is K8S

[minyk]

In Spark, spark.app.name is the part of K8S service object for driver otherwise spark.kubernetes.driver.resourceNamePrefix is set. And K8S service name is used to make domain name so service name is limited to 63 length.

Currently Kyuubi's sparksql eninge name is something like kyuubi-user-spark-sql-admin-default-f8221ce2-dc94-4f15-a44f-04cc0cc5cfa0-86a5c882ed35143e-driver, so it cannot be used for domain name, and spark generate service name:

    val randomServiceId = KubernetesUtils.uniqueID(clock = clock)
    val shorterServiceName = s"spark-$randomServiceId$DRIVER_SVC_POSTFIX"

In the result, Spark Driver's service name is like this: spark-54032b82ed3515c8-driver-svc and we loose many informations and cannot discover user's spark driver.

I think kyuubi set spark.kubernetes.driver.resourceNamePrefix to kyuubi-user-spark-sql-admin-default when Spark master is K8S, whole service discovery is a lot easier.

How about this?

[yaooqinn]

are we able to find the driver pod via kyuubi-unique-tag, also cc @zhouyifan279 @zwangsheng

[minyk]

we have a plan to use reverse proxy like https://github.com/datapunchorg/spark-ui-reverse-proxy, so k8s service name is the crucial point.

[yaooqinn]

OK, PRs are welcome

[minyk]

Our product's next milestone will be later Sept ~ early Oct. I will file a issue and PR after that. Thank you.

[yaooqinn]

OK, thanks~

[minyk]

We're already using SessionConfAdvisor to provide some spark configurations. after some discussions we're quite sure SessionConfAdvisor is the right way to deliver these k8s related configs rather than kyuubi itself. Maybe we can contribute some kind of KubernetesSessionConfAdvisor which proivdes extra spark configurations for K8S environment. How about this idea?

[yaooqinn]

Current spark.app.name is too long for K8S environment

This is a bug we shall fix

Maybe we can contribute some kind of KubernetesSessionConfAdvisor which proivdes extra spark configurations for K8S environment

Give a conf advisor to benefit k8s users sound good to me

[minyk]

Short note of current status:
Spark is using spark.kubernetes.driver.resourceNamePrefix only for testing purpose:
https://github.com/apache/spark/blame/master/resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/KubernetesConf.scala#L92 (maybe afraid of the name collision?)

So we're finding another way.

[yaooqinn]

there is also a config for executor?

[minyk]

most similar one is spark.kubernetes.executor.podNamePrefix, to determine the pod name of executor.