Skip to content

Commit 4bbaa9d

Browse files
committed
[SPARK-53571] Add integration-test-mac-spark4-iceberg GitHub Action job
### What changes were proposed in this pull request? This PR aims to add `integration-test-mac-spark4-iceberg` GitHub Action job. In addition, the existing GitHub Action job `integration-test-mac-iceberg` is renamed to `integration-test-mac-spark3-iceberg` to be clear. ### Why are the changes needed? Apache Iceberg 1.10 is released with Apache Spark 4 support. We need to have a test coverage for this combination. https://github.com/apache/iceberg/releases/tag/apache-iceberg-1.10.0 ### Does this PR introduce _any_ user-facing change? No, this is a test infra change. ### How was this patch tested? Pass the CIs. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #239 from dongjoon-hyun/SPARK-53571. Authored-by: Dongjoon Hyun <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
1 parent 93d527d commit 4bbaa9d

File tree

1 file changed

+30
-1
lines changed

1 file changed

+30
-1
lines changed

.github/workflows/build_and_test.yml

Lines changed: 30 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -225,7 +225,7 @@ jobs:
225225
cd -
226226
swift test --no-parallel -c release
227227
228-
integration-test-mac-iceberg:
228+
integration-test-mac-spark3-iceberg:
229229
runs-on: macos-15
230230
timeout-minutes: 20
231231
env:
@@ -254,6 +254,35 @@ jobs:
254254
swift test --filter DataFrameWriterV2Tests -c release
255255
swift test --filter IcebergTest -c release
256256
257+
integration-test-mac-spark4-iceberg:
258+
runs-on: macos-15
259+
timeout-minutes: 20
260+
env:
261+
SPARK_LOCAL_IP: localhost
262+
SPARK_ICEBERG_TEST_ENABLED: "true"
263+
steps:
264+
- uses: actions/checkout@v5
265+
- uses: swift-actions/setup-swift@d10500c1ac8822132eebbd74c48c3372c71d7ff5
266+
with:
267+
swift-version: "6.1"
268+
- name: Install Java
269+
uses: actions/setup-java@v4
270+
with:
271+
distribution: zulu
272+
java-version: 17
273+
- name: Build
274+
run: swift test --filter NOTHING -c release
275+
- name: Test
276+
run: |
277+
curl -LO https://www.apache.org/dyn/closer.lua/spark/spark-4.0.1/spark-4.0.1-bin-hadoop3.tgz?action=download
278+
tar xvfz spark-4.0.1-bin-hadoop3.tgz && rm spark-4.0.1-bin-hadoop3.tgz
279+
mv spark-4.0.1-bin-hadoop3 /tmp/spark
280+
cd /tmp/spark/sbin
281+
./start-connect-server.sh --packages org.apache.spark:spark-connect_2.13:4.0.1,org.apache.iceberg:iceberg-spark-runtime-4.0_2.13:1.10.0 -c spark.sql.catalog.local=org.apache.iceberg.spark.SparkCatalog -c spark.sql.catalog.local.type=hadoop -c spark.sql.catalog.local.warehouse=/tmp/spark/warehouse -c spark.sql.defaultCatalog=local
282+
cd -
283+
swift test --filter DataFrameWriterV2Tests -c release
284+
swift test --filter IcebergTest -c release
285+
257286
linter:
258287
runs-on: ubuntu-latest
259288
timeout-minutes: 20

0 commit comments

Comments
 (0)