Commit 8550b44
[SPARK-51532][INFRA] Restore the installation of Python 3.11 in macOS daily test
### What changes were proposed in this pull request?
This pr enables macOS to use `actions/setup-python` to install Python 3.11, so that the relevant modules can be tested using Python 3.11.
### Why are the changes needed?
Although the macOS daily test uses macOS-15, which comes with Python 3.9 to Python 3.13 installed by default, the `python3` command points to Python 3.13 by default instead of Python 3.11, which is required for the tests. This leads to some unexpected behaviors during testing. For example, I encountered the following logs while testing:
- https://github.com/apache/spark/actions/runs/13876211339/job/38828852614
```
- udaf/udaf-group-analytics.sql - Grouped Aggregate Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!!
- udaf/udaf-group-by-ordinal.sql - Grouped Aggregate Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!!
- udaf/udaf-group-by.sql - Grouped Aggregate Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!!
- udaf/udaf-grouping-set.sql - Grouped Aggregate Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!!
- udaf/udaf.sql - Grouped Aggregate Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!!
- udf/postgreSQL/udf-aggregates_part1.sql - Regular Python UDF
- udf/postgreSQL/udf-aggregates_part1.sql - Regular Python UDF_analyzer_test
- udf/postgreSQL/udf-aggregates_part1.sql - Scala UDF
- udf/postgreSQL/udf-aggregates_part1.sql - Scala UDF_analyzer_test
- udf/postgreSQL/udf-aggregates_part1.sql - Scalar Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!!
- udf/postgreSQL/udf-aggregates_part1.sql - Scalar Pandas UDF_analyzer_test is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!!
- udf/postgreSQL/udf-aggregates_part2.sql - Regular Python UDF
- udf/postgreSQL/udf-aggregates_part2.sql - Regular Python UDF_analyzer_test
- udf/postgreSQL/udf-aggregates_part2.sql - Scala UDF
- udf/postgreSQL/udf-aggregates_part2.sql - Scala UDF_analyzer_test
- udf/postgreSQL/udf-aggregates_part2.sql - Scalar Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!!
- udf/postgreSQL/udf-aggregates_part2.sql - Scalar Pandas UDF_analyzer_test is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!!
- udf/postgreSQL/udf-aggregates_part3.sql - Regular Python UDF
- udf/postgreSQL/udf-aggregates_part3.sql - Regular Python UDF_analyzer_test
- udf/postgreSQL/udf-aggregates_part3.sql - Scala UDF
- udf/postgreSQL/udf-aggregates_part3.sql - Scala UDF_analyzer_test
- udf/postgreSQL/udf-aggregates_part3.sql - Scalar Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!!
- udf/postgreSQL/udf-aggregates_part3.sql - Scalar Pandas UDF_analyzer_test is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!!
- udf/postgreSQL/udf-aggregates_part4.sql - Regular Python UDF
- udf/postgreSQL/udf-aggregates_part4.sql - Regular Python UDF_analyzer_test
- udf/postgreSQL/udf-aggregates_part4.sql - Scala UDF
- udf/postgreSQL/udf-aggregates_part4.sql - Scala UDF_analyzer_test
- udf/postgreSQL/udf-aggregates_part4.sql - Scalar Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!!
- udf/postgreSQL/udf-aggregates_part4.sql - Scalar Pandas UDF_analyzer_test is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!!
```
This is because the relevant modules were installed using Python 3.11, but the `python3` is actually Python 3.13. Therefore, this pull request restores the installation process for Python 3.11 on macOS, so that the tests also use Python 3.11.
### Does this PR introduce _any_ user-facing change?
No, just for test
### How was this patch tested?
- Pass GitHGub Actions
- check maven test on GitHub Actions: https://github.com/LuciferYang/spark/actions/runs/13892314705/attempts/1, confirm that the tests meet expectations and there are no longer ignore behaviors similar to `Scalar Pandas UDF is skipped because pyspark,pandas and/or pyarrow were not available in [python3]. !!! IGNORED !!!`
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #50293 from LuciferYang/install-python311-for-macos.
Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>1 parent 646b84b commit 8550b44
1 file changed
+1
-3
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
169 | 169 | | |
170 | 170 | | |
171 | 171 | | |
172 | | - | |
173 | | - | |
174 | | - | |
| 172 | + | |
175 | 173 | | |
176 | 174 | | |
177 | 175 | | |
| |||
0 commit comments