How to close the Kyuubi session and spark cluster? #4449
-
|
I am currently using JayDeBeApi to connect to kyuubi service on K8S to run some SQL queries on Spark. The usage is like this import jaydebeapi
connection = jaydebeapi.connect("org.apache.kyuubi.jdbc.KyuubiHiveDriver", <kyuubi service endpoint>, jars = [<kyuubi_jdbc_hive_jar>])
cursor = connection.cursor()
cursor.execute(<sql query>)
cursor.close()
connection.close()I meet a problem when doing this, which I cannot close the Kyuubi session and the Spark cluster. |
Beta Was this translation helpful? Give feedback.
Replies: 3 comments 6 replies
-
|
If you want to terminate the spark app when closing the connection, you need to set the share level to CONNECTION, see https://kyuubi.readthedocs.io/en/master/deployment/engine_lifecycle.html. |
Beta Was this translation helpful? Give feedback.
-
|
If you choose USER / GROUP share level, engines will share across connections, it's unsafe to shut down in one connection. However, you can use kyuubi.operation.language=scala to call spark.stop |
Beta Was this translation helpful? Give feedback.
-
|
Thank you. I'd like to try this. Is there any example or tutorial of how to use it? |
Beta Was this translation helpful? Give feedback.
-
|
Just execute two statements(same as normal SQL) sequentially |
Beta Was this translation helpful? Give feedback.
-
|
Thank you @pan3793. cursor.execute("SET kyuubi.operation.language=scala")
cursor.execute("spark.stop")This can delete the Spark Context, but raise an error How can I fix it? Besides, I also want to delete the Kyuubi driver. Is there a similar way to do it? |
Beta Was this translation helpful? Give feedback.
-
|
it's kind of hacky way, so exception is expected |
Beta Was this translation helpful? Give feedback.
-
|
Another thread shuts down the currently running connection, which should do just that |
Beta Was this translation helpful? Give feedback.
-
|
Setting |
Beta Was this translation helpful? Give feedback.
If you want to terminate the spark app when closing the connection, you need to set the share level to CONNECTION, see https://kyuubi.readthedocs.io/en/master/deployment/engine_lifecycle.html.