Spark application on the existing Kyuubi cluster

[mmuru]

Hi Kyuubi Team,

I would like to leverage an existing Kyuubi cluster for the spark ETL job application. Ideally, how to connect or submit a spark application? If it is possible, please provide me the steps or points me to the documentation.

Thanks in advance.

Best,
Muru

[bowenliang123]

Some suggestions for submitting Spark job via REST API .

• enable REST protocol in frontend configs including setting kyuubi.frontend.protocols(https://kyuubi.readthedocs.io/en/master/deployment/settings.html#frontend)
• create a new batch with POST /batches with existed resource file and class name
• monitor and get batch information with other REST API , GET /bathes/${batchId}

Uploading resource file for batch creation via REST API is still in working, (#4144 ).

[mmuru]

Hi @bowenliang123

Thank you for the information. Is it the same for PySpark job application too? Please, can you clarify it? In the meantime, I will check out the suggestions.

[bowenliang123]

I didn't try that with Pyspark, but seems PYSPARK in batchType is already supported in BatchesSubmission.
https://github.com/apache/kyuubi/blob/master/kyuubi-server/src/main/scala/org/apache/kyuubi/operation/BatchJobSubmission.scala#L87

Also uploading the resource file (as for spark-submit in Spark) in batch creation via REST API will be supported in #4106 soon.