Upgrade: new session are routed to the old spark driver

[user-GitHub-user]

kyuubi version: 1.7.0
1.start node1 kyuubi server
2.connect use zk mode
3. run ./bin/kyuubi-ctl delete server --host "node1" --port "10009"
4.quit connect
5. start node2 kyuubi server and version still 1.7.0
6. other connect use zk mode

The connection from Step 6 will use the spark driver from Step 2 , It looks like the connection from Step 2 didn't close automatically when I quit, and shouldn't the connection from Step 6 be using new spark driver?

Do I have to change the version？But what if I just want to migrate from yarn to k8s or local?

[pan3793]

Please read the article to learn the concept of "Engine Share Level"

[pan3793]

No, Kyuubi's version is a variable in routing rules

[user-GitHub-user]

No, Kyuubi's version is a variable in routing rules

but if i not change the kyuubi version, new kyuubi server still use 1.6.0 ,then new connect reuse the spark driver, Even if I change spark master from yarn to local[*] in new kyuubi server , it still run in yarn

[pan3793]

it's by design, use the CONNECTION share level if you don't want the engine shared across sessions

[user-GitHub-user]

it's by design, use the CONNECTION share level if you don't want the engine shared across sessions

I was just testing whether the upgraded function is normal, and I found this problem. Maybe it is not a big problem, but I think the priority of delete server operation should be higher than the Engine Share Level. Thank you for your answer

[pan3793]

To be clear, the engine's lifecycle is not managed by the server, the engine will self-terminate when there is no active session and exceeds the idle time threshold.

[pan3793]

what if I just want to migrate from yarn to k8s or local

Kyuubi respects your Spark Client and Spark Configurations, please read the doc to learn how to manage Spark configurations. And you can check the whole spark-submit command in Kyuubi Server's log