Passing on SparkContext for lineaging and auditing

[rks0191]

Well Iam planning on a design which could be generalized.

I am creating a framework which does multiple Transformations along with Auditing/lineaging. So the idea here is to ensure that the transformations happen in another module/EMR cluster(Maybe using PySpark/Scala Spark) entirely and lineaging happens in a different Emr Cluster. However To ensure that the auditing happens, I would need to send the SparkSessions across both the Spark jobs. Is there a better way to do the same?

[wajda]

Let me understand your question. So you run Spark jobs on multiple EMR clusters and want to track lineage centrally for all clusters, is that correct?

[rks0191]

Yes you are right. Iam planning on a single lineage tracker across multiple spark jobs.

[wajda]

For lineage tracking purpose you don't need to pass a Spark session around. All you need is deploying a Spark server in a way that it's visible from both your EMR clusters, so that the Spline agents sitting inside your Spark jobs could send the metadata to the server.