Skip to content

Commit 8550b44

Browse files
LuciferYangdongjoon-hyun
authored andcommitted
[SPARK-51532][INFRA] Restore the installation of Python 3.11 in macOS daily test
### What changes were proposed in this pull request? This pr enables macOS to use `actions/setup-python` to install Python 3.11, so that the relevant modules can be tested using Python 3.11. ### Why are the changes needed? Although the macOS daily test uses macOS-15, which comes with Python 3.9 to Python 3.13 installed by default, the `python3` command points to Python 3.13 by default instead of Python 3.11, which is required for the tests. This leads to some unexpected behaviors during testing. For example, I encountered the following logs while testing: - https://github.com/apache/spark/actions/runs/13876211339/job/38828852614 ``` - udaf/udaf-group-analytics.sql - Grouped Aggregate Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!! - udaf/udaf-group-by-ordinal.sql - Grouped Aggregate Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!! - udaf/udaf-group-by.sql - Grouped Aggregate Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!! - udaf/udaf-grouping-set.sql - Grouped Aggregate Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!! - udaf/udaf.sql - Grouped Aggregate Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!! - udf/postgreSQL/udf-aggregates_part1.sql - Regular Python UDF - udf/postgreSQL/udf-aggregates_part1.sql - Regular Python UDF_analyzer_test - udf/postgreSQL/udf-aggregates_part1.sql - Scala UDF - udf/postgreSQL/udf-aggregates_part1.sql - Scala UDF_analyzer_test - udf/postgreSQL/udf-aggregates_part1.sql - Scalar Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!! - udf/postgreSQL/udf-aggregates_part1.sql - Scalar Pandas UDF_analyzer_test is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!! - udf/postgreSQL/udf-aggregates_part2.sql - Regular Python UDF - udf/postgreSQL/udf-aggregates_part2.sql - Regular Python UDF_analyzer_test - udf/postgreSQL/udf-aggregates_part2.sql - Scala UDF - udf/postgreSQL/udf-aggregates_part2.sql - Scala UDF_analyzer_test - udf/postgreSQL/udf-aggregates_part2.sql - Scalar Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!! - udf/postgreSQL/udf-aggregates_part2.sql - Scalar Pandas UDF_analyzer_test is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!! - udf/postgreSQL/udf-aggregates_part3.sql - Regular Python UDF - udf/postgreSQL/udf-aggregates_part3.sql - Regular Python UDF_analyzer_test - udf/postgreSQL/udf-aggregates_part3.sql - Scala UDF - udf/postgreSQL/udf-aggregates_part3.sql - Scala UDF_analyzer_test - udf/postgreSQL/udf-aggregates_part3.sql - Scalar Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!! - udf/postgreSQL/udf-aggregates_part3.sql - Scalar Pandas UDF_analyzer_test is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!! - udf/postgreSQL/udf-aggregates_part4.sql - Regular Python UDF - udf/postgreSQL/udf-aggregates_part4.sql - Regular Python UDF_analyzer_test - udf/postgreSQL/udf-aggregates_part4.sql - Scala UDF - udf/postgreSQL/udf-aggregates_part4.sql - Scala UDF_analyzer_test - udf/postgreSQL/udf-aggregates_part4.sql - Scalar Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!! - udf/postgreSQL/udf-aggregates_part4.sql - Scalar Pandas UDF_analyzer_test is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!! ``` This is because the relevant modules were installed using Python 3.11, but the `python3` is actually Python 3.13. Therefore, this pull request restores the installation process for Python 3.11 on macOS, so that the tests also use Python 3.11. ### Does this PR introduce _any_ user-facing change? No, just for test ### How was this patch tested? - Pass GitHGub Actions - check maven test on GitHub Actions: https://github.com/LuciferYang/spark/actions/runs/13892314705/attempts/1, confirm that the tests meet expectations and there are no longer ignore behaviors similar to `Scalar Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!!` ### Was this patch authored or co-authored using generative AI tooling? No Closes #50293 from LuciferYang/install-python311-for-macos. Authored-by: yangjie01 <yangjie01@baidu.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>
1 parent 646b84b commit 8550b44

File tree

1 file changed

+1
-3
lines changed

1 file changed

+1
-3
lines changed

.github/workflows/maven_test.yml

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -169,9 +169,7 @@ jobs:
169169
# We should install one Python that is higher than 3+ for SQL and Yarn because:
170170
# - SQL component also has Python related tests, for example, IntegratedUDFTestUtils.
171171
# - Yarn has a Python specific test too, for example, YarnClusterSuite.
172-
# macos (14) already has its Python installed, see also SPARK-47096 and
173-
# https://github.com/actions/runner-images/blob/main/images/macos/macos-14-Readme.md
174-
if: contains(inputs.os, 'ubuntu') && (contains(matrix.modules, 'resource-managers#yarn') || (contains(matrix.modules, 'sql#core')) || contains(matrix.modules, 'connect'))
172+
if: contains(matrix.modules, 'resource-managers#yarn') || (contains(matrix.modules, 'sql#core')) || contains(matrix.modules, 'connect')
175173
with:
176174
python-version: '3.11'
177175
architecture: x64

0 commit comments

Comments
 (0)