Couldn't run the spline codes via utility file

[ArunSP23EY]

Hi,

I'm looking for an alternate for the below libraries in Spark version, the one i got from the GtHub was all Scala libraries.

I have added the below mentioned code to capture the notebook info and trigger Spline Listeners, I have used a logic in before this code, in a way that it should run only when certain condition is met. Once the conditions are met, I'm calling the spline enabler added via a utility as mentioned below with a parameter value

def runSplineProcess( input: str):
  if True:
  dbutils.notebook.run("Notebookpath", 60, {"scalaTextWidget": input}) 

The spline triggers are in Scala code, but the notebook is in spark code and when i run the notebook via a job, the notebook fails stating Scala libraries as unknown. If executed without the parameter value, no issue. The notebook runs successfully, creating a lineage in Spline UI.

Could someone help me here?

The Scala code used to trigger Spline Listeners

%scala
import org.apache.commons.configuration._
import za.co.absa.spline.harvester.postprocessing.AbstractPostProcessingFilter 
import za.co.absa.spline.harvester.conf.DefaultSplineConfigurer 
import za.co.absa.spline.harvester.conf.StandardSplineConfigurationStack
import za.co.absa.spline.harvester.HarvestingContext
import za.co.absa.spline.producer.model.v1_1._
import za.co.absa.spline.harvester.ExtraMetadataImplicits._
import za.co.absa.spline.harvester.SparkLineageInitializer._

val notebookInfo = {
  val ctx = dbutils.notebook.getContext
  val tags = ctx.tags
  val notebookName = ctx.notebookPath.get.split("/").last
  val notebookURL = s"${tags("browserHostName")}/?o=${tags("orgId")}${tags("browserHash")}"
  val user = tags("user")
  val mounts = dbutils.fs.ls("/mnt").map(_.path)
  Map(
    "notebookURL" -> notebookURL,
    "user" -> user,
    "name" -> notebookName,
    "mounts" -> mounts,
    "timestamp" -> System.currentTimeMillis
  )
}

val splineConf = StandardSplineConfigurationStack(spark)

spark.enableLineageTracking(new DefaultSplineConfigurer(spark, splineConf) {
  override def postProcessingFilter = new AbstractPostProcessingFilter {
    override def processExecutionPlan(plan: ExecutionPlan, ctx: HarvestingContext): ExecutionPlan = {
      plan.withAddedExtra(Map(
        "notebookInfo" -> notebookInfo
      ))
    }
  }
}) 

[cerveada]

The spline triggers are in Scala code, but the notebook is in spark code and when i run the notebook via a job, the notebook fails stating Scala libraries as unknown.

What do you mean by spark code?

[ArunSP23EY]

Hi Alex, Thanks i was able to track down the issue. Yes, it was an issue with the tags, when i execute a notebook via a job the browser information was not captured but on a manual run, the browser in which im executing the notebook was captured and hence the error.

However, the Lineage is not being reflected when i run the databricks notebook via a job. the lineage should be imported to a custom UI page.

The spline triggers:

%scala
System.setProperty("spline.mode", "REQUIRED")
System.setProperty("spark.sql.queryExecutionListeners", "za.co.absa.spline.harvester.listener.SplineQueryExecutionListener")
System.setProperty("spline.producer.url","https://functionsapp_url/api/func/producer")

Issue: When executed manually, the lineage is created/published in the target UI, but the same notebook via a job fails to create one.

[cerveada]

I am not Alex ;)

Are you sure the tags issue is fixed? It could cause the spline to not turn on.
Are there any errors in the Databricks log? If yes, please post them here.

[ArunSP23EY]

Apologies @cerveada,

Yes, the tags issue is fixed. I have changed the code, so that the error was resolved.

The databricks notebook has completed successfully. There are no errors but at the same time the lineage is not created when executed as a job.

The same notebook creates a lineage when executed manually.

I use the azure functions app as my producer url, as mentioned in above code, with manual run there's an event captured in functions app, which is not happening when the notebook is executed via a job.

[ArunSP23EY]

Only difference i notice in the manual run and automated run (via job) is the output of the cell where i declare the below,

%scala
System.setProperty("spline.mode", "REQUIRED")
System.setProperty("spark.sql.queryExecutionListeners", "za.co.absa.spline.harvester.listener.SplineQueryExecutionListener")
System.setProperty("spline.producer.url","https://functionapp.azurewebsites.net/api/func/producer")

Output:
Manual run: res0: String = https://functionsapp.azurewebsites.net/api/func/producer
Job: res0: String = null

Any idea on why am i getting a null value when executed via a job?

[cerveada]

I am bit confused. I thought you are using programatic initialisation (calling spark.enableLineageTracking), but here you set spark.sql.queryExecutionListeners property which is codeless initialisation. You should not do both at the same time.

The codeless approach doesn't work on Databricks so you should not use it in this case.

see
https://github.com/AbsaOSS/spline-spark-agent#initialization

also this guide for databricks
https://github.com/AbsaOSS/spline-getting-started/tree/main/spline-on-databricks