Skip to content

Commit 43727a2

Browse files
AlenkaFraulcd
AlenkaF
and
raulcd
authored
GH-47797: [CI][Python] Update Python installs for free-threaded wheel tasks (#47993)
### Rationale for this change Musllinux and Win wheel tasks are building Python from source. That can be changed. ### What changes are included in this PR? - usage of one of the binaries provided by [astral-sh](https://github.com/astral-sh/python-build-standalone/releases) in the free-threaded musllinux wheel tasks - usage of Python install manager for the free-threaded and regular Win wheels build (test stays as is). ### Are these changes tested? Yes, with extended wheel builds. ### Are there any user-facing changes? No. * GitHub Issue: #47797 Lead-authored-by: AlenkaF <frim.alenka@gmail.com> Co-authored-by: Alenka Frim <AlenkaF@users.noreply.github.com> Co-authored-by: Raúl Cumplido <raulcumplido@gmail.com> Signed-off-by: Raúl Cumplido <raulcumplido@gmail.com>
1 parent 649a372 commit 43727a2

7 files changed

+62
-108
lines changed

‎ci/docker/python-free-threaded-wheel-musllinux-test-imports.dockerfile‎

Lines changed: 16 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -19,38 +19,32 @@ ARG base
1919
FROM ${base}
2020

2121
ARG python_version=3.13
22+
ARG arch=aarch64
23+
ARG build_date
2224

23-
RUN apk add --no-cache \
25+
RUN apk update && \
26+
apk add --no-cache \
2427
bash \
25-
build-base \
26-
bzip2-dev \
27-
g++ \
28+
curl \
2829
git \
29-
libffi-dev \
30-
libnsl-dev \
31-
libtirpc-dev \
32-
linux-headers \
33-
ncurses-dev \
34-
openssl-dev \
35-
pkgconf \
30+
tar \
3631
tzdata \
37-
zlib-dev
32+
zstd
3833

39-
# Install Python without GIL
34+
# Install Python with free-threading from python-build-standalone
35+
# See available releases at: https://github.com/astral-sh/python-build-standalone/releases
4036
RUN set -e; \
4137
case "${python_version}" in \
4238
3.13) python_patch_version="3.13.9";; \
4339
3.14) python_patch_version="3.14.0";; \
4440
esac && \
45-
wget https://github.com/python/cpython/archive/refs/tags/v${python_patch_version}.tar.gz && \
46-
tar -xzf v${python_patch_version}.tar.gz && \
47-
rm v${python_patch_version}.tar.gz && \
48-
cd cpython-${python_patch_version}/ && \
49-
./configure --disable-gil --with-ensurepip && \
50-
make -j && \
51-
make install && \
52-
cd ../ && \
53-
rm -rf cpython-${python_patch_version}/
41+
curl -L -o python.tar.zst \
42+
https://github.com/astral-sh/python-build-standalone/releases/download/${build_date}/cpython-${python_patch_version}+${build_date}-${arch}-unknown-linux-musl-freethreaded+lto-full.tar.zst && \
43+
mkdir -p /opt/python && \
44+
tar -xf python.tar.zst -C /opt/python --strip-components=1 && \
45+
rm python.tar.zst
46+
47+
ENV PATH="/opt/python/install/bin:${PATH}"
5448

5549
ENV ARROW_PYTHON_VENV /arrow-dev
5650
RUN python${python_version}t -m venv ${ARROW_PYTHON_VENV}

‎ci/docker/python-free-threaded-wheel-musllinux-test-unittests.dockerfile‎

Lines changed: 16 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -19,38 +19,34 @@ ARG base
1919
FROM ${base}
2020

2121
ARG python_version=3.13
22+
ARG arch=aarch64
23+
ARG build_date
2224

23-
RUN apk add --no-cache \
25+
RUN apk update && \
26+
apk add --no-cache \
2427
bash \
2528
build-base \
26-
bzip2-dev \
29+
curl \
2730
g++ \
2831
git \
29-
libffi-dev \
30-
libnsl-dev \
31-
libtirpc-dev \
32-
linux-headers \
33-
ncurses-dev \
34-
openssl-dev \
35-
pkgconf \
32+
tar \
3633
tzdata \
37-
zlib-dev
34+
zstd
3835

39-
# Install Python without GIL
36+
# Install Python with free-threading from python-build-standalone
37+
# See available releases at: https://github.com/astral-sh/python-build-standalone/releases
4038
RUN set -e; \
4139
case "${python_version}" in \
4240
3.13) python_patch_version="3.13.9";; \
4341
3.14) python_patch_version="3.14.0";; \
4442
esac && \
45-
wget https://github.com/python/cpython/archive/refs/tags/v${python_patch_version}.tar.gz && \
46-
tar -xzf v${python_patch_version}.tar.gz && \
47-
rm v${python_patch_version}.tar.gz && \
48-
cd cpython-${python_patch_version}/ && \
49-
./configure --disable-gil --with-ensurepip && \
50-
make -j && \
51-
make install && \
52-
cd ../ && \
53-
rm -rf cpython-${python_patch_version}/
43+
curl -L -o python.tar.zst \
44+
https://github.com/astral-sh/python-build-standalone/releases/download/${build_date}/cpython-${python_patch_version}+${build_date}-${arch}-unknown-linux-musl-freethreaded+lto-full.tar.zst && \
45+
mkdir -p /opt/python && \
46+
tar -xf python.tar.zst -C /opt/python --strip-components=1 && \
47+
rm python.tar.zst
48+
49+
ENV PATH="/opt/python/install/bin:${PATH}"
5450

5551
ENV ARROW_PYTHON_VENV /arrow-dev
5652
RUN python${python_version}t -m venv ${ARROW_PYTHON_VENV}

‎ci/docker/python-free-threaded-wheel-windows-vs2022.dockerfile‎

Lines changed: 0 additions & 53 deletions
This file was deleted.

‎ci/docker/python-wheel-windows-test-vs2022.dockerfile‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -42,4 +42,4 @@ RUN %PYTHON_CMD% -m pip install -U pip setuptools
4242
COPY python/requirements-wheel-test.txt C:/arrow/python/
4343
RUN %PYTHON_CMD% -m pip install -r C:/arrow/python/requirements-wheel-test.txt
4444

45-
ENV PYTHON=$python
45+
ENV PYTHON=${python}

‎ci/docker/python-wheel-windows-vs2022-base.dockerfile‎

Lines changed: 15 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -74,13 +74,25 @@ RUN `
7474

7575
# Install choco CLI
7676
#
77-
# Switch into Powershell just for this command because choco only provides a
78-
# Powershell installation script. After, we switch back to cmd.
77+
# Switch into Powershell just for the following commands because choco and Python install manager (MSIX)
78+
# only provide a Powershell installation scripts. After, we switch back to cmd.
7979
#
8080
# See https://chocolatey.org/install#completely-offline-install
8181
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]
8282
RUN `
83-
Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))
83+
Set-ExecutionPolicy Bypass -Scope Process -Force; `
84+
[System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; `
85+
iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))
86+
87+
# Install the Python install manager
88+
#
89+
# See https://docs.python.org/dev/using/windows.html#python-install-manager and
90+
# https://www.python.org/ftp/python/pymanager/
91+
RUN `
92+
$pymanager_url = 'https://www.python.org/ftp/python/pymanager/python-manager-25.0.msix'; `
93+
Invoke-WebRequest -Uri $pymanager_url -OutFile 'C:\Windows\pymanager.msix'; `
94+
Add-AppxPackage C:\Windows\pymanager.msix
95+
8496
SHELL ["cmd", "/S", "/C"]
8597

8698
# Install CMake and other tools

‎ci/docker/python-wheel-windows-vs2022.dockerfile‎

Lines changed: 7 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -23,16 +23,15 @@ FROM ${base}
2323

2424
# Define the full version number otherwise choco falls back to patch number 0 (3.10 => 3.10.0)
2525
ARG python=3.10
26-
RUN (if "%python%"=="3.10" setx PYTHON_VERSION "3.10.11" && setx PYTHON_CMD "py -3.10") & \
27-
(if "%python%"=="3.11" setx PYTHON_VERSION "3.11.9" && setx PYTHON_CMD "py -3.11") & \
28-
(if "%python%"=="3.12" setx PYTHON_VERSION "3.12.10" && setx PYTHON_CMD "py -3.12") & \
29-
(if "%python%"=="3.13" setx PYTHON_VERSION "3.13.9" && setx PYTHON_CMD "py -3.13") & \
30-
(if "%python%"=="3.14" setx PYTHON_VERSION "3.14.0" && setx PYTHON_CMD "py -3.14")
3126

32-
RUN choco install -r -y --pre --no-progress python --version=%PYTHON_VERSION%
33-
RUN %PYTHON_CMD% -m pip install -U pip setuptools
27+
ARG python_variant=default
28+
ENV PYTHON_VERSION=${python}
29+
ENV PYTHON_VARIANT=${python_variant}
30+
RUN pymanager install --version %PYTHON_VERSION% --variant %PYTHON_VARIANT%
31+
32+
RUN py -%PYTHON_VERSION% -m pip install -U pip setuptools
3433

3534
COPY python/requirements-wheel-build.txt C:/arrow/python/
36-
RUN %PYTHON_CMD% -m pip install -r C:/arrow/python/requirements-wheel-build.txt
35+
RUN py -%PYTHON_VERSION% -m pip install -r C:/arrow/python/requirements-wheel-build.txt
3736

3837
ENV PYTHON=${python}

‎compose.yaml‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1228,6 +1228,8 @@ services:
12281228
args:
12291229
base: "${ARCH}/alpine:${ALPINE_LINUX}"
12301230
python_version: ${PYTHON}
1231+
build_date: "20251014" # python-build-standalone release date
1232+
arch: ${ARCH_ALIAS}
12311233
context: .
12321234
dockerfile: ci/docker/python-free-threaded-wheel-musllinux-test-imports.dockerfile
12331235
cache_from:
@@ -1270,6 +1272,8 @@ services:
12701272
args:
12711273
base: "${ARCH}/alpine:${ALPINE_LINUX}"
12721274
python_version: ${PYTHON}
1275+
build_date: "20251014" # python-build-standalone release date
1276+
arch: ${ARCH_ALIAS}
12731277
context: .
12741278
dockerfile: ci/docker/python-free-threaded-wheel-musllinux-test-unittests.dockerfile
12751279
cache_from:
@@ -1384,6 +1388,7 @@ services:
13841388
args:
13851389
base: ${REPO}:python-wheel-windows-vs2022-base-vcpkg-${VCPKG}-${PYTHON_WHEEL_WINDOWS_IMAGE_REVISION}
13861390
python: ${PYTHON}
1391+
python_variant: default
13871392
context: .
13881393
dockerfile: ci/docker/python-wheel-windows-vs2022.dockerfile
13891394
# This should make the pushed images reusable, but the image gets rebuilt.
@@ -1400,8 +1405,9 @@ services:
14001405
args:
14011406
base: ${REPO}:python-wheel-windows-vs2022-base-vcpkg-${VCPKG}-${PYTHON_WHEEL_WINDOWS_IMAGE_REVISION}
14021407
python: ${PYTHON}
1408+
python_variant: freethreaded
14031409
context: .
1404-
dockerfile: ci/docker/python-free-threaded-wheel-windows-vs2022.dockerfile
1410+
dockerfile: ci/docker/python-wheel-windows-vs2022.dockerfile
14051411
# This should make the pushed images reusable, but the image gets rebuilt.
14061412
# Uncomment if no local cache is available.
14071413
# cache_from:

0 commit comments

Comments
 (0)