Skip to content

Commit f25b381

Browse files
zhengruifengzhengruifeng
committed
[SPARK-54828][PYTHON][INFRA] Set up a scheduled builder for free-threaded Python 3.14
### What changes were proposed in this pull request? Set up a scheduled builder for free-threaded Python 3.14 ### Why are the changes needed? to monitor the support of no-gil ### Does this PR introduce _any_ user-facing change? no ### How was this patch tested? PR builder with ``` default: '{"PYSPARK_IMAGE_TO_TEST": "python-314-nogil", "PYTHON_TO_TEST": "python3.14t", "PYTHON_GIL": "0"}' ``` https://github.com/zhengruifeng/spark/actions/runs/20477205646/job/58844457177 note that some packages are not compatible with no-gil, and some tests are skipped with `PYTHON_GIL=0` ### Was this patch authored or co-authored using generative AI tooling? no Closes #53588 from zhengruifeng/python_314_no_gil. Authored-by: Ruifeng Zheng <[email protected]> Signed-off-by: Ruifeng Zheng <[email protected]>
1 parent e316d28 commit f25b381

File tree

3 files changed

+143
-0
lines changed

3 files changed

+143
-0
lines changed

‎.github/workflows/build_infra_images_cache.yml‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -41,6 +41,7 @@ on:
4141
- 'dev/spark-test-image/python-313/Dockerfile'
4242
- 'dev/spark-test-image/python-313-nogil/Dockerfile'
4343
- 'dev/spark-test-image/python-314/Dockerfile'
44+
- 'dev/spark-test-image/python-314-nogil/Dockerfile'
4445
- 'dev/spark-test-image/numpy-213/Dockerfile'
4546
- '.github/workflows/build_infra_images_cache.yml'
4647
# Create infra image when cutting down branches/tags
@@ -258,6 +259,19 @@ jobs:
258259
- name: Image digest (PySpark with Python 3.14)
259260
if: hashFiles('dev/spark-test-image/python-314/Dockerfile') != ''
260261
run: echo ${{ steps.docker_build_pyspark_python_314.outputs.digest }}
262+
- name: Build and push (PySpark with Python 3.14 no GIL)
263+
if: hashFiles('dev/spark-test-image/python-314-nogil/Dockerfile') != ''
264+
id: docker_build_pyspark_python_314_nogil
265+
uses: docker/build-push-action@v6
266+
with:
267+
context: ./dev/spark-test-image/python-314-nogil/
268+
push: true
269+
tags: ghcr.io/apache/spark/apache-spark-github-action-image-pyspark-python-314-nogil-cache:${{ github.ref_name }}-static
270+
cache-from: type=registry,ref=ghcr.io/apache/spark/apache-spark-github-action-image-pyspark-python-314-nogil-cache:${{ github.ref_name }}
271+
cache-to: type=registry,ref=ghcr.io/apache/spark/apache-spark-github-action-image-pyspark-python-314-nogil-cache:${{ github.ref_name }},mode=max
272+
- name: Image digest (PySpark with Python 3.14 no GIL)
273+
if: hashFiles('dev/spark-test-image/python-314-nogil/Dockerfile') != ''
274+
run: echo ${{ steps.docker_build_pyspark_python_314_nogil.outputs.digest }}
261275
- name: Build and push (PySpark with Numpy 2.1.3)
262276
if: hashFiles('dev/spark-test-image/numpy-213/Dockerfile') != ''
263277
id: docker_build_pyspark_numpy_213

‎.github/workflows/build_python_3.14_nogil.yml‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one
3+
# or more contributor license agreements. See the NOTICE file
4+
# distributed with this work for additional information
5+
# regarding copyright ownership. The ASF licenses this file
6+
# to you under the Apache License, Version 2.0 (the
7+
# "License"); you may not use this file except in compliance
8+
# with the License. You may obtain a copy of the License at
9+
#
10+
# http://www.apache.org/licenses/LICENSE-2.0
11+
#
12+
# Unless required by applicable law or agreed to in writing,
13+
# software distributed under the License is distributed on an
14+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
15+
# KIND, either express or implied. See the License for the
16+
# specific language governing permissions and limitations
17+
# under the License.
18+
#
19+
20+
name: "Build / Python-only (master, Python 3.14, no GIL)"
21+
22+
on:
23+
schedule:
24+
- cron: '0 20 */3 * *'
25+
workflow_dispatch:
26+
27+
jobs:
28+
run-build:
29+
permissions:
30+
packages: write
31+
name: Run
32+
uses: ./.github/workflows/build_and_test.yml
33+
if: github.repository == 'apache/spark'
34+
with:
35+
java: 17
36+
branch: master
37+
hadoop: hadoop3
38+
envs: >-
39+
{
40+
"PYSPARK_IMAGE_TO_TEST": "python-314-nogil",
41+
"PYTHON_TO_TEST": "python3.14t",
42+
"PYTHON_GIL": "0"
43+
}
44+
jobs: >-
45+
{
46+
"pyspark": "true",
47+
"pyspark-pandas": "true"
48+
}

‎dev/spark-test-image/python-314-nogil/Dockerfile‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
#
2+
# Licensed to the Apache Software Foundation (ASF) under one or more
3+
# contributor license agreements. See the NOTICE file distributed with
4+
# this work for additional information regarding copyright ownership.
5+
# The ASF licenses this file to You under the Apache License, Version 2.0
6+
# (the "License"); you may not use this file except in compliance with
7+
# the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing, software
12+
# distributed under the License is distributed on an "AS IS" BASIS,
13+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
# See the License for the specific language governing permissions and
15+
# limitations under the License.
16+
#
17+
18+
# Image for building and testing Spark branches. Based on Ubuntu 22.04.
19+
# See also in https://hub.docker.com/_/ubuntu
20+
FROM ubuntu:jammy-20240911.1
21+
LABEL org.opencontainers.image.authors="Apache Spark project <[email protected]>"
22+
LABEL org.opencontainers.image.licenses="Apache-2.0"
23+
LABEL org.opencontainers.image.ref.name="Apache Spark Infra Image For PySpark with Python 3.13 (no GIL)"
24+
# Overwrite this label to avoid exposing the underlying Ubuntu OS version label
25+
LABEL org.opencontainers.image.version=""
26+
27+
ENV FULL_REFRESH_DATE=20251224
28+
29+
ENV DEBIAN_FRONTEND=noninteractive
30+
ENV DEBCONF_NONINTERACTIVE_SEEN=true
31+
32+
RUN apt-get update && apt-get install -y \
33+
build-essential \
34+
ca-certificates \
35+
curl \
36+
gfortran \
37+
git \
38+
gnupg \
39+
libcurl4-openssl-dev \
40+
libfontconfig1-dev \
41+
libfreetype6-dev \
42+
libfribidi-dev \
43+
libgit2-dev \
44+
libharfbuzz-dev \
45+
libjpeg-dev \
46+
liblapack-dev \
47+
libopenblas-dev \
48+
libpng-dev \
49+
libpython3-dev \
50+
libssl-dev \
51+
libtiff5-dev \
52+
libwebp-dev \
53+
libxml2-dev \
54+
openjdk-17-jdk-headless \
55+
pkg-config \
56+
qpdf \
57+
tzdata \
58+
software-properties-common \
59+
wget \
60+
zlib1g-dev
61+
62+
# Install Python 3.14 (no GIL)
63+
RUN add-apt-repository ppa:deadsnakes/ppa
64+
RUN apt-get update && apt-get install -y \
65+
python3.14-nogil \
66+
&& apt-get autoremove --purge -y \
67+
&& apt-get clean \
68+
&& rm -rf /var/lib/apt/lists/*
69+
70+
71+
ARG BASIC_PIP_PKGS="numpy pyarrow>=22.0.0 six==1.16.0 pandas==2.3.3 scipy plotly<6.0.0 mlflow>=2.8.1 coverage matplotlib openpyxl memory-profiler>=0.61.0 scikit-learn>=1.3.2"
72+
ARG CONNECT_PIP_PKGS="grpcio==1.76.0 grpcio-status==1.76.0 protobuf==6.33.0 googleapis-common-protos==1.71.0 zstandard==0.25.0 graphviz==0.20.3"
73+
74+
75+
# Install Python 3.14 packages
76+
RUN curl -sS https://bootstrap.pypa.io/get-pip.py | python3.14t
77+
# TODO: Add BASIC_PIP_PKGS and CONNECT_PIP_PKGS when it supports Python 3.14 free threaded
78+
# TODO: Add lxml, grpcio, grpcio-status back when they support Python 3.14 free threaded
79+
RUN python3.14t -m pip install --ignore-installed 'blinker>=1.6.2' # mlflow needs this
80+
RUN python3.14t -m pip install 'numpy>=2.1' 'pyarrow>=19.0.0' 'six==1.16.0' 'pandas==2.3.3' scipy coverage matplotlib openpyxl jinja2 && \
81+
python3.14t -m pip cache purge

0 commit comments

Comments
 (0)