Add basic Iceberg/Hive dataset parsing in Spark instrumentation#8152

Closed

yiliangzhou wants to merge 21 commits intomasterfrom

liangzhou.yi/djm-spark-table-lineage

Commits on Dec 20, 2024

a spark instrumentation poc to collect table dataset run by spark sql
yiliangzhou
committed

Commits on Jan 2, 2025

compile sparklistener on spark 2.4 and use reflection to parse logical plan from spark 3
yiliangzhou
committed
add spark datasets to application span
yiliangzhou
committed
parse ReplaceData node from LogicalPlan to support dataset referenced in sql update and delete
yiliangzhou
committed

Commits on Jan 3, 2025

add hive table parsing in spark listener for insert and select
yiliangzhou
committed

Commits on Jan 6, 2025

serialize hive/iceberg table stats and properties as json string for easier parsing downstream
yiliangzhou
committed

Commits on Jan 14, 2025