[AURON apache#2030] Add Native Scan Support for Apache Hudi Copy-On-Write Tables. (apache#2031)

slfan1989 · web-flow · commit 10cf0fdf2b1d · 2026-03-20T10:33:34.000+08:00
### Which issue does this PR close? Closes apache#2030 ### Rationale for this change This PR adds native scan support for Hudi Copy-On-Write (COW) tables, enabling Auron to accelerate Hudi table reads by converting `FileSourceScanExec` operations to native Parquet/ORC scan implementations. ### What changes are included in this PR? #### 1. **New Module: `thirdparty/auron-hudi`** - **`HudiConvertProvider`**: Implements `AuronConvertProvider` SPI to intercept and convert Hudi `FileSourceScanExec` to native scans - Detects Hudi file formats (`HoodieParquetFileFormat`, `HoodieOrcFileFormat`) - Converts to `NativeParquetScanExec` or `NativeOrcScanExec` - Handles timestamp fallback logic automatically - **`HudiScanSupport`**: Core detection and validation logic - File format recognition with `NewHoodie*` format rejection - Table type resolution via multi-source metadata fallback: - Options → Catalog → `.hoodie/hoodie.properties` - MOR table detection and rejection - Time travel query detection (via `as.of.instant`, `as.of.timestamp` options) - FileIndex class hierarchy verification #### 2. **Configuration** - Added `spark.auron.enable.hudi.scan` config option (default: `true`) - Respects existing Parquet/ORC timestamp scanning configurations - Runtime Spark version validation (3.0–3.5 only) #### 3. **Build & Integration** - **Maven**: New profile `hudi-0.15` with enforcer rules - Validates `hudiEnabled=true` property - Restricts Spark to 3.0–3.5 - Pins Hudi version to 0.15.0 - **Build Script**: Enhanced `auron-build.sh` - Added `--hudi <VERSION>` parameter - Version compatibility validation - Auto-enables `hudiEnabled` property - **CI/CD**: New workflow `.github/workflows/hudi.yml` - Matrix testing: Spark 3.0–3.5 × JDK 8/17/21 × Scala 2.12 - Independent Hudi test pipeline ### Are there any user-facing changes? ## New Configuration Option ```scala // Enable Hudi native scan (enabled by default) spark.conf.set("spark.auron.enable.hudi.scan", "true") ``` ### How was this patch tested? Add Junit Test. Signed-off-by: slfan1989 <slfan1989@apache.org>
diff --git a/.github/workflows/hudi.yml b/.github/workflows/hudi.yml
@@ -0,0 +1,108 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+name: Hudi
+
+on:
+  workflow_dispatch:
+  push:
+    branches:
+      - master
+      - branch-*
+  pull_request:
+    branches:
+      - master
+      - branch-*
+
+concurrency:
+  group: hudi-${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  test-hudi:
+    name: Test Hudi (${{ matrix.sparkver }} / JDK${{ matrix.javaver }} / Scala${{ matrix.scalaver }})
+    runs-on: ubuntu-24.04
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - sparkver: "3.0"
+            scalaver: "2.12"
+            javaver: "8"
+            hudiver: "0.15"
+          - sparkver: "3.1"
+            scalaver: "2.12"
+            javaver: "8"
+            hudiver: "0.15"
+          - sparkver: "3.2"
+            scalaver: "2.12"
+            javaver: "8"
+            hudiver: "0.15"
+          - sparkver: "3.3"
+            scalaver: "2.12"
+            javaver: "8"
+            hudiver: "0.15"
+          - sparkver: "3.4"
+            scalaver: "2.12"
+            javaver: "17"
+            hudiver: "0.15"
+          - sparkver: "3.5"
+            scalaver: "2.12"
+            javaver: "17"
+            hudiver: "0.15"
+          - sparkver: "3.5"
+            scalaver: "2.12"
+            hudiver: "0.15"
+            javaver: "21"
+
+    steps:
+      - name: Checkout Auron
+        uses: actions/checkout@v6
+
+      - name: Setup Java and Maven cache
+        uses: actions/setup-java@v5
+        with:
+          distribution: 'adopt-hotspot'
+          java-version: ${{ matrix.javaver }}
+          cache: 'maven'
+
+      - name: Build dependencies (skip tests)
+        run: >
+          ./build/mvn -B install
+          -pl thirdparty/auron-hudi
+          -am
+          -Pscala-${{ matrix.scalaver }}
+          -Pspark-${{ matrix.sparkver }}
+          -Phudi-${{ matrix.hudiver }}
+          -Prelease
+          -DskipTests
+
+      - name: Test Hudi Module
+        run: >
+          ./build/mvn -B test
+          -pl thirdparty/auron-hudi
+          -Pscala-${{ matrix.scalaver }}
+          -Pspark-${{ matrix.sparkver }}
+          -Phudi-${{ matrix.hudiver }}
+          -Prelease
+
+      - name: Upload reports
+        if: failure()
+        uses: actions/upload-artifact@v6
+        with:
+          name: auron-hudi-${{ matrix.sparkver }}-hudi${{ matrix.hudiver }}-jdk${{ matrix.javaver }}-test-report
+          path: thirdparty/auron-hudi/target/surefire-reports
diff --git a/auron-build.sh b/auron-build.sh
@@ -38,6 +38,7 @@ SUPPORTED_UNIFFLE_VERSIONS=("0.10")
 SUPPORTED_PAIMON_VERSIONS=("1.2")
 SUPPORTED_FLINK_VERSIONS=("1.18")
 SUPPORTED_ICEBERG_VERSIONS=("1.10.1")
+SUPPORTED_HUDI_VERSIONS=("0.15")
 
 # -----------------------------------------------------------------------------
 # Function: print_help
@@ -64,6 +65,7 @@ print_help() {
     IFS=','; echo "  --uniffle <VERSION>      Specify Uniffle version (e.g. ${SUPPORTED_UNIFFLE_VERSIONS[*]})"; unset IFS
     IFS=','; echo "  --paimon <VERSION>       Specify Paimon version (e.g. ${SUPPORTED_PAIMON_VERSIONS[*]})"; unset IFS
     IFS=','; echo "  --iceberg <VERSION>      Specify Iceberg version (e.g. ${SUPPORTED_ICEBERG_VERSIONS[*]})"; unset IFS
+    IFS=','; echo "  --hudi <VERSION>         Specify Hudi version (e.g. ${SUPPORTED_HUDI_VERSIONS[*]})"; unset IFS
 
     echo "  -h, --help               Show this help message"
     echo
@@ -78,7 +80,8 @@ print_help() {
          "--celeborn ${SUPPORTED_CELEBORN_VERSIONS[*]: -1}" \
          "--uniffle ${SUPPORTED_UNIFFLE_VERSIONS[*]: -1}" \
          "--paimon ${SUPPORTED_PAIMON_VERSIONS[*]: -1}" \
-         "--iceberg ${SUPPORTED_ICEBERG_VERSIONS[*]: -1}"
+         "--iceberg ${SUPPORTED_ICEBERG_VERSIONS[*]: -1}" \
+         "--hudi ${SUPPORTED_HUDI_VERSIONS[*]: -1}"
     exit 0
 }
 
@@ -135,6 +138,7 @@ CELEBORN_VER=""
 UNIFFLE_VER=""
 PAIMON_VER=""
 ICEBERG_VER=""
+HUDI_VER=""
 
 # -----------------------------------------------------------------------------
 # Section: Argument Parsing
@@ -301,6 +305,27 @@ while [[ $# -gt 0 ]]; do
                 exit 1
             fi
             ;;
+        --hudi)
+            if [[ -n "$2" && "$2" != -* ]]; then
+                HUDI_VER="$2"
+                if ! check_supported_version "$HUDI_VER" "${SUPPORTED_HUDI_VERSIONS[@]}"; then
+                  print_invalid_option_error Hudi "$HUDI_VER" "${SUPPORTED_HUDI_VERSIONS[@]}"
+                fi
+                if [ -z "$SPARK_VER" ]; then
+                  echo "ERROR: Building hudi requires spark at the same time, and only Spark versions 3.0 to 3.5 are supported."
+                  exit 1
+                fi
+                if [ "$SPARK_VER" != "3.0" ] && [ "$SPARK_VER" != "3.1" ] && [ "$SPARK_VER" != "3.2" ] && [ "$SPARK_VER" != "3.3" ] && [ "$SPARK_VER" != "3.4" ] && [ "$SPARK_VER" != "3.5" ]; then
+                  echo "ERROR: Building hudi requires spark versions are 3.0 to 3.5."
+                  exit 1
+                fi
+                shift 2
+            else
+                IFS=','; echo "ERROR: Missing argument for --hudi," \
+                "specify one of: ${SUPPORTED_HUDI_VERSIONS[*]}" >&2; unset IFS
+                exit 1
+            fi
+            ;;
         --flinkver)
             if [[ -n "$2" && "$2" != -* ]]; then
                 FLINK_VER="$2"
@@ -437,6 +462,9 @@ fi
 if [[ -n "$ICEBERG_VER" ]]; then
     BUILD_ARGS+=("-Piceberg-$ICEBERG_VER")
 fi
+if [[ -n "$HUDI_VER" ]]; then
+    BUILD_ARGS+=("-Phudi-$HUDI_VER")
+fi
 
 # Configure Maven build threads:
 # - local builds default to Maven's single-threaded behavior
@@ -473,6 +501,7 @@ get_build_info() {
     "paimon.version") echo "${PAIMON_VER}" ;;
     "flink.version") echo "${FLINK_VER}" ;;
     "iceberg.version") echo "${ICEBERG_VER}" ;;
+    "hudi.version") echo "${HUDI_VER}" ;;
     "build.timestamp") echo "$(date -u +"%Y-%m-%dT%H:%M:%SZ")" ;;
     *) echo "" ;;
   esac
diff --git a/dev/reformat b/dev/reformat
@@ -146,7 +146,7 @@ sparkver=spark-3.5
 prepare_for_spark "${sparkver}"
 for celebornver in celeborn-0.5 celeborn-0.6
 do
-  run_maven_format -P"${sparkver}" -Pceleborn,"${celebornver}" -Puniffle,uniffle-0.10 -Ppaimon,paimon-1.2 -Pflink-1.18 -Piceberg-1.10.1
+  run_maven_format -P"${sparkver}" -Pceleborn,"${celebornver}" -Puniffle,uniffle-0.10 -Ppaimon,paimon-1.2 -Pflink-1.18 -Piceberg-1.10.1 -Phudi-0.15
 
 done
 
diff --git a/pom.xml b/pom.xml
@@ -1319,6 +1319,16 @@
       </properties>
     </profile>
 
+    <profile>
+      <id>hudi-0.15</id>
+      <modules>
+        <module>thirdparty/auron-hudi</module>
+      </modules>
+      <properties>
+        <hudiVersion>0.15.0</hudiVersion>
+      </properties>
+    </profile>
+
     <profile>
       <id>flink-1.18</id>
       <modules>
diff --git a/spark-extension/src/main/java/org/apache/auron/spark/configuration/SparkAuronConfiguration.java b/spark-extension/src/main/java/org/apache/auron/spark/configuration/SparkAuronConfiguration.java
@@ -327,6 +327,12 @@ public class SparkAuronConfiguration extends AuronConfiguration {
             .withDescription("Enable Iceberg scan operation conversion to native Auron implementations.")
             .withDefaultValue(true);
 
+    public static final ConfigOption<Boolean> ENABLE_HUDI_SCAN = new SQLConfOption<>(Boolean.class)
+            .withKey("auron.enable.hudi.scan")
+            .withCategory("Operator Supports")
+            .withDescription("Enable Hudi scan operation conversion to native Auron implementations.")
+            .withDefaultValue(true);
+
     public static final ConfigOption<Boolean> ENABLE_PROJECT = new SQLConfOption<>(Boolean.class)
             .withKey("auron.enable.project")
             .withCategory("Operator Supports")
diff --git a/spark-extension/src/main/scala/org/apache/spark/sql/auron/AuronConverters.scala b/spark-extension/src/main/scala/org/apache/spark/sql/auron/AuronConverters.scala
@@ -161,7 +161,10 @@ object AuronConverters extends Logging {
       case e: BroadcastExchangeExec if enableBroadcastExchange =>
         tryConvert(e, convertBroadcastExchangeExec)
       case e: FileSourceScanExec if enableScan => // scan
-        tryConvert(e, convertFileSourceScanExec)
+        extConvertProviders.find(p => p.isEnabled && p.isSupported(e)) match {
+          case Some(provider) => tryConvert(e, provider.convert)
+          case None => tryConvert(e, convertFileSourceScanExec)
+        }
       case e: ProjectExec if enableProject => // project
         tryConvert(e, convertProjectExec)
       case e: FilterExec if enableFilter => // filter
diff --git a/thirdparty/auron-hudi/pom.xml b/thirdparty/auron-hudi/pom.xml
@@ -0,0 +1,116 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+  ~ Licensed to the Apache Software Foundation (ASF) under one or more
+  ~ contributor license agreements.  See the NOTICE file distributed with
+  ~ this work for additional information regarding copyright ownership.
+  ~ The ASF licenses this file to You under the Apache License, Version 2.0
+  ~ (the "License"); you may not use this file except in compliance with
+  ~ the License.  You may obtain a copy of the License at
+  ~
+  ~    http://www.apache.org/licenses/LICENSE-2.0
+  ~
+  ~ Unless required by applicable law or agreed to in writing, software
+  ~ distributed under the License is distributed on an "AS IS" BASIS,
+  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+  ~ See the License for the specific language governing permissions and
+  ~ limitations under the License.
+  -->
+<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
+  <modelVersion>4.0.0</modelVersion>
+
+  <parent>
+    <groupId>org.apache.auron</groupId>
+    <artifactId>auron-parent_${scalaVersion}</artifactId>
+    <version>${project.version}</version>
+    <relativePath>../../pom.xml</relativePath>
+  </parent>
+
+  <artifactId>auron-hudi_${scalaVersion}</artifactId>
+  <packaging>jar</packaging>
+  <name>Apache Auron Hudi ${hudiVersion} ${scalaVersion}</name>
+
+  <dependencies>
+    <dependency>
+      <groupId>org.apache.auron</groupId>
+      <artifactId>spark-extension_${scalaVersion}</artifactId>
+      <version>${project.version}</version>
+    </dependency>
+    <dependency>
+      <groupId>org.apache.hudi</groupId>
+      <artifactId>hudi-spark${shortSparkVersion}-bundle_${scalaVersion}</artifactId>
+      <version>${hudiVersion}</version>
+      <scope>test</scope>
+    </dependency>
+    <dependency>
+      <groupId>org.apache.spark</groupId>
+      <artifactId>spark-sql_${scalaVersion}</artifactId>
+      <scope>provided</scope>
+    </dependency>
+    <dependency>
+      <groupId>org.scalatest</groupId>
+      <artifactId>scalatest_${scalaVersion}</artifactId>
+      <scope>test</scope>
+    </dependency>
+    <dependency>
+      <groupId>org.apache.spark</groupId>
+      <artifactId>spark-core_${scalaVersion}</artifactId>
+      <type>test-jar</type>
+      <scope>test</scope>
+    </dependency>
+    <dependency>
+      <groupId>org.apache.spark</groupId>
+      <artifactId>spark-sql_${scalaVersion}</artifactId>
+      <type>test-jar</type>
+      <scope>test</scope>
+    </dependency>
+    <dependency>
+      <groupId>org.apache.spark</groupId>
+      <artifactId>spark-catalyst_${scalaVersion}</artifactId>
+      <type>test-jar</type>
+      <scope>test</scope>
+    </dependency>
+    <dependency>
+      <groupId>org.apache.spark</groupId>
+      <artifactId>spark-hive_${scalaVersion}</artifactId>
+      <scope>test</scope>
+    </dependency>
+    <dependency>
+      <groupId>org.apache.auron</groupId>
+      <artifactId>spark-extension-shims-spark_${scalaVersion}</artifactId>
+      <version>${project.version}</version>
+      <scope>test</scope>
+    </dependency>
+  </dependencies>
+
+  <build>
+    <plugins>
+      <plugin>
+        <groupId>org.apache.maven.plugins</groupId>
+        <artifactId>maven-enforcer-plugin</artifactId>
+        <version>${maven-enforcer-plugin.version}</version>
+        <executions>
+          <execution>
+            <id>hudi-spark-version-compat</id>
+            <goals>
+              <goal>enforce</goal>
+            </goals>
+            <configuration>
+              <rules>
+                <requireProperty>
+                  <property>shortSparkVersion</property>
+                  <regex>^(3\.0|3\.1|3\.2|3\.3|3\.4|3\.5)$</regex>
+                  <regexMessage>Hudi integration supports Spark 3.0-3.5 only. Current: ${shortSparkVersion}</regexMessage>
+                </requireProperty>
+                <requireProperty>
+                  <property>hudiVersion</property>
+                  <regex>^0\.15\.0$</regex>
+                  <regexMessage>Hudi integration supports only Hudi 0.15.0. Current: ${hudiVersion}</regexMessage>
+                </requireProperty>
+              </rules>
+            </configuration>
+          </execution>
+        </executions>
+      </plugin>
+    </plugins>
+  </build>
+</project>
diff --git a/thirdparty/auron-hudi/src/main/resources/META-INF/services/org.apache.spark.sql.auron.AuronConvertProvider b/thirdparty/auron-hudi/src/main/resources/META-INF/services/org.apache.spark.sql.auron.AuronConvertProvider
@@ -0,0 +1,18 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+org.apache.spark.sql.auron.hudi.HudiConvertProvider
diff --git a/thirdparty/auron-hudi/src/main/scala/org/apache/spark/sql/auron/hudi/HudiConvertProvider.scala b/thirdparty/auron-hudi/src/main/scala/org/apache/spark/sql/auron/hudi/HudiConvertProvider.scala
diff --git a/thirdparty/auron-hudi/src/main/scala/org/apache/spark/sql/auron/hudi/HudiScanSupport.scala b/thirdparty/auron-hudi/src/main/scala/org/apache/spark/sql/auron/hudi/HudiScanSupport.scala
diff --git a/thirdparty/auron-hudi/src/test/resources/log4j2.properties b/thirdparty/auron-hudi/src/test/resources/log4j2.properties
diff --git a/thirdparty/auron-hudi/src/test/scala/org/apache/spark/sql/auron/hudi/HudiScanSupportSuite.scala b/thirdparty/auron-hudi/src/test/scala/org/apache/spark/sql/auron/hudi/HudiScanSupportSuite.scala
