[Spark]Refactor Spark project structure to combine both Dsv1 connector and kernel backed Dsv2 connector #5320

huan233usc · 2025-10-09T19:59:10Z

Which Delta project/connector is this regarding?

Description

This PR refactors the Delta Spark build system to support a modular architecture that separates V1 and V2 implementations while maintaining a single published delta-spark jar that could use connector for both V1 and V2 and with public entry point DeltaCatalog and DeltaSparkSessionExtension unchanged.

Architecture Changes

Module Structure:

sparkV1 (internal): Delta Spark V1 implementation (production code only, no tests)
sparkV1Shaded (internal): V1 without DeltaLog/Snapshot/OptimisticTransaction classes, used to avoid V2 connector accidentally depend on legacy V1 representation of delta log.
sparkV2 (internal): Kernel-based Delta Spark implementation (formerly kernelSpark)
spark (combined): Final published module that merges V1 + V2 + storage into delta-spark.jar

Details Changes

Rename old catalog plugin:

Introduced Abstract* base classes in sparkV1 and Legacy* subclasses for backward compatibility
New unified classes in spark-combined:
- DeltaCatalog extends AbstractDeltaCatalog
- DeltaSparkSessionExtension extends AbstractDeltaSparkSessionExtension
  This makes public entry point DeltaCatalog and DeltaSparkSessionExtension unchanged.

Single Jar rules all connectors:

Internal modules (sparkV1, sparkV2, sparkV1Shaded) marked with skipReleaseSettings - not published to Maven
spark (combined) uses packageBin / mappings to merge classes from internal modules
Duplicate class detection added to prevent overlapping code
POM post-processing excludes internal module dependencies

Test Configuration:

Tests run in spark (combined) module with Test / baseDirectory pointing to spark/ directory
I could run tests for both V1/V2 connector

How was this patch tested?

All existing unit tests pass

Does this PR introduce any user-facing changes?

No user-facing changes. This is an internal build system refactoring:

No

tdas · 2025-10-10T08:01:03Z

build.sbt

+// Module 3: delta-spark-v2 (kernel-spark based, depends on v1-shaded)
+// ============================================================
+lazy val `delta-spark-v2` = (project in file("kernel-spark"))
+  .dependsOn(`delta-spark-v1-shaded`) // Only depends on shaded v1 (no DeltaLog)


exactly what i had in mind.

tdas · 2025-10-10T08:05:21Z

build.sbt

+    // Test sources point to original spark/src/test/ (no file movement)
+    Test / unmanagedSourceDirectories ++= Seq(
+      baseDirectory.value.getParentFile / "spark" / "src" / "test" / "scala",
+      baseDirectory.value.getParentFile / "spark" / "src" / "test" / "java"
+    ),
+    Test / unmanagedResourceDirectories += 
+      baseDirectory.value.getParentFile / "spark" / "src" / "test" / "resources",
+
+    // Include spark-version-specific test sources
+    Test / unmanagedSourceDirectories ++= {
+      val sparkVer = sparkVersion.value
+      if (sparkVer.startsWith("3.5")) {
+        Seq(baseDirectory.value.getParentFile / "spark" / "src" / "test" / "scala-spark-3.5")
+      } else if (sparkVer.startsWith("4.0")) {
+        Seq(baseDirectory.value.getParentFile / "spark" / "src" / "test" / "scala-spark-master")
+      } else {
+        Seq.empty
+      }
+    },


it might be simpler to actually move the files, but this is okay for the first cut.

tdas · 2025-10-10T08:10:59Z

build.sbt

+// ============================================================
+// Module 1: delta-spark-v1 (prod code only, no tests)
+// ============================================================
+lazy val `delta-spark-v1` = (project in file("spark"))


why are you using "`" (backticks) ?
why not just name it sparkV1 (it all already delta)?

and fundamentally.. this is scala... and scala variables. nobody uses - in variable names :)

Ah I see, will rename

tdas · 2025-10-10T08:12:58Z

build.sbt

+// ============================================================
+lazy val spark = (project in file("spark-combined"))
+  .dependsOn(`delta-spark-shaded`)  // Direct dependency on shaded (for delegation classes)
+  .dependsOn(`delta-spark-v1` % "test->test")  // Test utilities from v1


why do you need this? v1 has no tests.

tdas · 2025-10-10T08:16:50Z

build.sbt

+    // This module contains delegation code like:
+    // - DeltaCatalog (delegates to V1 or V2)
+    // - DeltaSparkSessionExtension (registers both)


do we really have to shade anything?

These comments and many of those are stale, I will fix them (after fixing tests) and let you know then

…java - build.sbt: Removed old kernelSpark module (replaced by sparkV2 in our refactor) - StreamingHelperTest.java: Use deltaLog.initialCatalogTable() instead of Option.empty()

gengliangwang · 2025-10-17T05:03:31Z

spark/src/main/scala/io/delta/sql/DeltaSparkSessionExtension.scala

 */
-class DeltaSparkSessionExtension extends (SparkSessionExtensions => Unit) {
+class LegacyDeltaSparkSessionExtension extends AbstractDeltaSparkSessionExtension
+class AbstractDeltaSparkSessionExtension extends (SparkSessionExtensions => Unit) {


Let's add a blank line between the two classes.
Also, add comment for each classs

gengliangwang · 2025-10-17T05:03:47Z

spark/src/main/scala/org/apache/spark/sql/delta/catalog/DeltaCatalog.scala

-class DeltaCatalog extends DelegatingCatalogExtension
+class LegacyDeltaCatalog extends AbstractDeltaCatalog
+
+class AbstractDeltaCatalog extends DelegatingCatalogExtension


Let's add comments

gengliangwang · 2025-10-17T05:04:27Z

spark-combined/src/main/java/org/apache/spark/sql/delta/catalog/DeltaCatalog.java

Shall we add a README.md under spark-combined?

Also, shall we call it spark-unified instead of spark-combined?

Or, we can have

spark

sparkV1

sparkV2

zikangh · 2025-10-17T20:59:32Z

build.sbt

+
+      // Filter out DeltaLog, Snapshot, OptimisticTransaction classes
+      v1Mappings.filterNot { case (file, path) =>
+        path.contains("org/apache/spark/sql/delta/DeltaLog") ||


What do you think about org/apache/spark/sql/delta/actions/actions.scala? Should we filter this out too?

yes, this is very v1 specific.

huan233usc added 12 commits October 9, 2025 12:57

save

e6e374f

save

9560950

save

8b9fda1

save

4f45983

fix

d6dc7ef

fix

3a6472b

fix

b55bfaa

fix

29a9dbe

fix

5cbf64f

fix

9a6acdd

fix

f8d4862

fix

6c7c972

tdas reviewed Oct 10, 2025

View reviewed changes

huan233usc added 2 commits October 10, 2025 10:36

fix

565f9cb

fix

74a1f5c

huan233usc changed the title ~~[POC]New spark structure~~ [WIP][POC]New spark structure Oct 10, 2025

huan233usc added 4 commits October 10, 2025 11:01

fix

c6c306d

fix

dbf686b

fix

2929d72

fix

642a42a

huan233usc force-pushed the new branch from 278d77b to 642a42a Compare October 11, 2025 03:51

huan233usc added 5 commits October 10, 2025 21:04

fix

e485cda

fix

f6ef579

fix

5f0dbc8

fix

812ba1d

fix

7dd8d18

huan233usc added 4 commits October 15, 2025 13:45

save

6fea63c

save

ba9f416

save

42d09ae

revert to a working version

94dcf97

huan233usc force-pushed the new branch from 82350cc to 94dcf97 Compare October 16, 2025 20:05

huan233usc added 16 commits October 16, 2025 13:41

update

4f07eb9

Merge master: resolve conflicts in build.sbt and StreamingHelperTest.…

758d35e

…java - build.sbt: Removed old kernelSpark module (replaced by sparkV2 in our refactor) - StreamingHelperTest.java: Use deltaLog.initialCatalogTable() instead of Option.empty()

fix

322cb0b

simplify

573ed9e

simplify

85173cc

fix comments

b52d0a8

fix import

7860a01

fix import

6650ff7

remove unnecessary change

68802d9

remove unnecessary change

4f48696

fix comments

3479837

fix test

9a039ab

fix test

43ced15

simplify

64d5723

simplify

8cceaed

simplify

8afe5fd

huan233usc changed the title ~~[WIP][POC]New spark structure~~ [Spark]Refactor Spark project structure to combine both Dsv1 connector and kernel backed Dsv2 connector Oct 17, 2025

huan233usc requested review from allisonport-db, gengliangwang, scottsand-db and tdas October 17, 2025 04:39

gengliangwang reviewed Oct 17, 2025

View reviewed changes

zikangh reviewed Oct 17, 2025

View reviewed changes

[Spark]Refactor Spark project structure to combine both Dsv1 connector and kernel backed Dsv2 connector #5320

Are you sure you want to change the base?

[Spark]Refactor Spark project structure to combine both Dsv1 connector and kernel backed Dsv2 connector #5320

Uh oh!

Conversation

huan233usc commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Which Delta project/connector is this regarding?

Description

Architecture Changes

Details Changes

How was this patch tested?

Does this PR introduce any user-facing changes?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tdas Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

huan233usc commented Oct 9, 2025 •

edited

Loading

tdas Oct 10, 2025 •

edited

Loading