Separation of SparkOperationEvent from kyuubi-spark-sql-engine

[minyk]

Hi, all

We're trying to get some event from spark. So I first try to use SparkListenerSQLExecutionStart and SparkListenerSQLExecutionEnd, but some queries are hitting twice; first at result = spark.sql(statement) and second at result.collect(). Later I noticed this is absolutely normal because of spark's lazy execution behavior.

Now we're trying to get event from Kyuubi's SparkOperationEvent, which is more reliable, but we cannot find SparkOperationEvent class (NoClassDefFound) from our SparkListener implementation as normal. I think there're some kind of class loader isolation between Spark app(e.g. kyuubi-spark-sql-engine) jar and Spark's extra lib(e.g. our listener) jars. If I make symlink of the kyuubi-spark-sql-engine.jar to spark/jars directory, our listener is running as designed.

Am I right? if so, separate simple jar which contains SparkOperationEvent is very helpful for making kyuubi extensions easy and light.

Thank you.

[yaooqinn]

Am I right? if so, separate simple jar which contains SparkOperationEvent is very helpful for making kyuubi extensions easy and light.

this is the right way to go, we are trying to make kyuubi-events a common extension for such a purpose