Skip to content

[VL][CI] Migrate Spark 4.1 tests to CentOS 9 #11519

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
PHILO-HE merged 13 commits into from
Feb 10, 2026
+18 −6
Merged

[VL][CI] Migrate Spark 4.1 tests to CentOS 9 #11519

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
2fb560d
[CI] Add CentOS 9 support
ReemaAlzaid Jan 29, 2026
97c5e0d
fix: use JDK 8 for Spark 3.4/3.5 centOS 9 jobs
ReemaAlzaid Jan 29, 2026
75bf8ee
Merge branch 'main' into ci/add-centos9-support
ReemaAlzaid Jan 29, 2026
9c4759c
Move Spark 4.1 tests to CentOS 9
ReemaAlzaid Jan 31, 2026
d7daf1b
Merge branch 'main' into ci/add-centos9-support
ReemaAlzaid Jan 31, 2026
ac38d66
fix: use build centos 7 for spark41
ReemaAlzaid Feb 2, 2026
7621ace
Apply suggestion from @zhouyuan
ReemaAlzaid Feb 2, 2026
c39c0c4
Merge branch 'main' into ci/add-centos9-support
ReemaAlzaid Feb 2, 2026
546cea5
fix: add alternatives --install for python 3.11 setup
ReemaAlzaid Feb 2, 2026
06f0501
Merge branch 'apache:main' into ci/add-centos9-support
ReemaAlzaid Feb 5, 2026
6add1ea
exclude America/Coyhaique timezone from tryCastSuite and castwithansi
ReemaAlzaid Feb 5, 2026
d17c8fe
fix format
ReemaAlzaid Feb 6, 2026
6180296
Merge branch 'main' into ci/add-centos9-support
ReemaAlzaid Feb 6, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 6 additions & 4 deletions .github/workflows/velox_backend_x86.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1425,7 +1425,7 @@ jobs:
runs-on: ubuntu-22.04
env:
SPARK_TESTING: true
container: apache/gluten:centos-8-jdk17
container: apache/gluten:centos-9-jdk17
steps:
- uses: actions/checkout@v2
- name: Download All Artifacts
Expand All @@ -1440,8 +1440,10 @@ jobs:
path: /root/.m2/repository/org/apache/arrow/
- name: Prepare
run: |
dnf module -y install python39 && \
alternatives --set python3 /usr/bin/python3.9 && \
dnf install -y python3.11 python3.11-pip python3.11-devel && \
ls -la /usr/bin/python3.11 && \
alternatives --install /usr/bin/python3 python3 /usr/bin/python3.11 1 && \
alternatives --set python3 /usr/bin/python3.11 && \
pip3 install setuptools==77.0.3 && \
pip3 install pyspark==3.5.5 cython && \
pip3 install pandas==2.2.3 pyarrow==20.0.0
Expand Down Expand Up @@ -1482,7 +1484,7 @@ jobs:
runs-on: ubuntu-22.04
env:
SPARK_TESTING: true
container: apache/gluten:centos-8-jdk17
container: apache/gluten:centos-9-jdk17
steps:
- uses: actions/checkout@v2
- name: Download All Artifacts
Expand Down
7 changes: 6 additions & 1 deletion ...src/test/scala/org/apache/spark/sql/catalyst/expressions/GlutenCastWithAnsiOffSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,11 @@ class GlutenCastWithAnsiOffSuite extends CastWithAnsiOffSuite with GlutenTestsTr

// SystemV timezones are a legacy way of specifying timezones in Unix-like OS.
// It is not supported by Velox.
for (tz <- ALL_TIMEZONES.filterNot(_.getId.contains("SystemV"))) {
for (
tz <- ALL_TIMEZONES
.filterNot(_.getId.contains("SystemV"))
.filterNot(_.getId.contains("America/Coyhaique"))
) {
withSQLConf(
SQLConf.SESSION_LOCAL_TIMEZONE.key -> tz.getId
) {
Expand Down Expand Up @@ -184,6 +188,7 @@ class GlutenCastWithAnsiOffSuite extends CastWithAnsiOffSuite with GlutenTestsTr
.filterNot(_.getId.contains("America/Ciudad_Juarez"))
.filterNot(_.getId.contains("Antarctica/Vostok"))
.filterNot(_.getId.contains("Pacific/Kanton"))
.filterNot(_.getId.contains("America/Coyhaique"))
.filterNot(_.getId.contains("Asia/Tehran"))
.filterNot(_.getId.contains("Iran")),
prefix = "CastSuiteBase-cast-string-to-timestamp",
Expand Down
7 changes: 6 additions & 1 deletion ...spark41/src/test/scala/org/apache/spark/sql/catalyst/expressions/GlutenTryCastSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,11 @@ class GlutenTryCastSuite extends TryCastSuite with GlutenTestsTrait {

// SystemV timezones are a legacy way of specifying timezones in Unix-like OS.
// It is not supported by Velox.
for (tz <- ALL_TIMEZONES.filterNot(_.getId.contains("SystemV"))) {
for (
tz <- ALL_TIMEZONES
.filterNot(_.getId.contains("SystemV"))
.filterNot(_.getId.contains("America/Coyhaique"))
) {
withSQLConf(
SQLConf.SESSION_LOCAL_TIMEZONE.key -> tz.getId
) {
Expand Down Expand Up @@ -118,6 +122,7 @@ class GlutenTryCastSuite extends TryCastSuite with GlutenTestsTrait {
.filterNot(_.getId.contains("SystemV"))
.filterNot(_.getId.contains("Europe/Kyiv"))
.filterNot(_.getId.contains("America/Ciudad_Juarez"))
.filterNot(_.getId.contains("America/Coyhaique"))
.filterNot(_.getId.contains("Antarctica/Vostok"))
.filterNot(_.getId.contains("Pacific/Kanton"))
.filterNot(_.getId.contains("Asia/Tehran"))
Expand Down