Skip to content

Commit e4b2810

Browse files
kevinjqliukevinjqliu
authored
[infra] Nightly pypi build for pyiceberg_core (#948)
Inspired by [pyiceberg#1601](apache/iceberg-python#1601) & https://github.com/apache/iceberg-python/blob/main/.github/workflows/nightly-pypi-build.yml This PR adds nightly build of `pyiceberg_core` to testpypi. I have successfully tested the github action on my fork by pushing to my own package, * Action run: https://github.com/kevinjqliu/iceberg-rust/actions/runs/13381378488 * testpypi artifact: https://test.pypi.org/project/pyiceberg-core-kevinliu/ I've also verified library versions locally ``` (3.12.8) ➜ python git:(kevinjqliu/nightly-pyiceberg-core) pip install -i https://test.pypi.org/simple/ pyiceberg-core-kevinliu --prefer-binary --force Looking in indexes: https://test.pypi.org/simple/ Collecting pyiceberg-core-kevinliu Using cached https://test-files.pythonhosted.org/packages/4b/c7/771857185d3d96e4ce0b59293540d0e1df9a8a39dbc938e0770328504c25/pyiceberg_core_kevinliu-0.4.0.dev20250218012743-cp39-abi3-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl.metadata (1.7 kB) Using cached https://test-files.pythonhosted.org/packages/4b/c7/771857185d3d96e4ce0b59293540d0e1df9a8a39dbc938e0770328504c25/pyiceberg_core_kevinliu-0.4.0.dev20250218012743-cp39-abi3-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl (1.3 MB) Installing collected packages: pyiceberg-core-kevinliu Attempting uninstall: pyiceberg-core-kevinliu Found existing installation: pyiceberg_core-kevinliu 0.4.0.dev20250218012743 Uninstalling pyiceberg_core-kevinliu-0.4.0.dev20250218012743: Successfully uninstalled pyiceberg_core-kevinliu-0.4.0.dev20250218012743 Successfully installed pyiceberg-core-kevinliu-0.4.0.dev20250218012743 pip show pyiceberg-core-kevinliu (3.12.8) ➜ python git:(kevinjqliu/nightly-pyiceberg-core) pip show pyiceberg-core-kevinliu Name: pyiceberg_core-kevinliu Version: 0.4.0.dev20250218012743 Summary: Home-page: https://rust.iceberg.apache.org Author: Author-email: License: Apache-2.0 Location: /Users/kevinliu/.pyenv/versions/3.12.8/lib/python3.12/site-packages Requires: Required-by: ```
1 parent c6cf0e9 commit e4b2810

File tree

3 files changed

+156
-28
lines changed

3 files changed

+156
-28
lines changed

‎.github/actions/overwrite-package-version/action.yml‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
name: 'Update Package Version'
19+
description: 'Updates pyproject.toml version with timestamp'
20+
runs:
21+
using: "composite"
22+
steps:
23+
- name: Setup Python
24+
uses: actions/setup-python@v5
25+
with:
26+
python-version: '3.12'
27+
28+
- name: Install toml
29+
run: pip install toml
30+
shell: bash
31+
32+
- name: Get and update version
33+
shell: bash
34+
run: |
35+
CURRENT_VERSION=$(python -c "import toml; print(toml.load('bindings/python/pyproject.toml')['project']['version'])")
36+
TIMESTAMP=$(date +%Y%m%d%H%M%S)
37+
NEW_VERSION="${CURRENT_VERSION}.dev${TIMESTAMP}"
38+
NEW_VERSION=$NEW_VERSION python -c "
39+
import toml
40+
import os
41+
config = toml.load('bindings/python/pyproject.toml')
42+
config['project']['version'] = os.environ['NEW_VERSION']
43+
with open('bindings/python/pyproject.toml', 'w') as f:
44+
toml.dump(config, f)
45+
print(f'Updated version to: {config[\"project\"][\"version\"]}')
46+
"

‎.github/workflows/release_python.yml‎

Lines changed: 0 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -113,31 +113,3 @@ jobs:
113113
with:
114114
skip-existing: true
115115
packages-dir: bindings/python/dist
116-
117-
testpypi-publish:
118-
name: Publish Python 🐍 distribution 📦 to TestPypi
119-
needs: [ sdist, wheels ]
120-
runs-on: ubuntu-latest
121-
# Only publish to TestPyPi if the tag is a pre-release
122-
if: ${{ startsWith(github.ref, 'refs/tags/') && contains(github.ref, '-')}}
123-
124-
environment:
125-
name: testpypi
126-
url: https://test.pypi.org/p/pyiceberg_core
127-
128-
permissions:
129-
id-token: write # IMPORTANT: mandatory for trusted publishing
130-
131-
steps:
132-
- name: Download all the dists
133-
uses: actions/download-artifact@v4
134-
with:
135-
pattern: wheels-*
136-
merge-multiple: true
137-
path: bindings/python/dist
138-
- name: Publish to TestPyPI
139-
uses: pypa/gh-action-pypi-publish@release/v1
140-
with:
141-
repository-url: https://test.pypi.org/legacy/
142-
skip-existing: true
143-
packages-dir: bindings/python/dist

‎.github/workflows/release_python_nightly.yml‎

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
# Licensed to the Apache Software Foundation (ASF) under one
2+
# or more contributor license agreements. See the NOTICE file
3+
# distributed with this work for additional information
4+
# regarding copyright ownership. The ASF licenses this file
5+
# to you under the Apache License, Version 2.0 (the
6+
# "License"); you may not use this file except in compliance
7+
# with the License. You may obtain a copy of the License at
8+
#
9+
# http://www.apache.org/licenses/LICENSE-2.0
10+
#
11+
# Unless required by applicable law or agreed to in writing,
12+
# software distributed under the License is distributed on an
13+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
14+
# KIND, either express or implied. See the License for the
15+
# specific language governing permissions and limitations
16+
# under the License.
17+
18+
name: "Nightly PyPI Build"
19+
20+
on:
21+
schedule:
22+
- cron: "0 0 * * *" # Runs at midnight UTC every day
23+
workflow_dispatch: # Allows manual triggering
24+
25+
env:
26+
rust_msrv: "1.77.1"
27+
28+
permissions:
29+
contents: read
30+
31+
jobs:
32+
sdist:
33+
if: github.repository == 'apache/iceberg-rust' # Only run for apache repo
34+
runs-on: ubuntu-latest
35+
steps:
36+
- uses: actions/checkout@v4
37+
38+
- uses: ./.github/actions/overwrite-package-version # overwrite the pacakge version with current timestamp
39+
40+
- uses: PyO3/maturin-action@v1
41+
with:
42+
working-directory: "bindings/python"
43+
command: sdist
44+
args: -o dist
45+
- name: Upload sdist
46+
uses: actions/upload-artifact@v4
47+
with:
48+
name: wheels-sdist
49+
path: bindings/python/dist
50+
51+
wheels:
52+
if: github.repository == 'apache/iceberg-rust' # Only run for apache repo
53+
runs-on: "${{ matrix.os }}"
54+
strategy:
55+
matrix:
56+
include:
57+
- { os: windows-latest }
58+
- { os: macos-latest, target: "universal2-apple-darwin" }
59+
- { os: ubuntu-latest, target: "x86_64" }
60+
- { os: ubuntu-latest, target: "aarch64", manylinux: "manylinux_2_28" }
61+
- { os: ubuntu-latest, target: "armv7l" }
62+
steps:
63+
- uses: actions/checkout@v4
64+
- uses: ./.github/actions/overwrite-package-version # overwrite the pacakge version with current timestamp
65+
- uses: actions/setup-python@v5
66+
with:
67+
python-version: 3.9
68+
- name: Setup Rust toolchain
69+
uses: ./.github/actions/setup-builder
70+
with:
71+
rust-version: ${{ env.rust_msrv }}
72+
- uses: PyO3/maturin-action@v1
73+
with:
74+
target: ${{ matrix.target }}
75+
manylinux: ${{ matrix.manylinux || 'auto' }}
76+
working-directory: "bindings/python"
77+
command: build
78+
args: --release -o dist
79+
- name: Upload wheels
80+
uses: actions/upload-artifact@v4
81+
with:
82+
name: wheels-${{ matrix.os }}-${{ matrix.target }}
83+
path: bindings/python/dist
84+
85+
testpypi-publish:
86+
needs: [ sdist, wheels ]
87+
runs-on: ubuntu-latest
88+
89+
environment:
90+
name: testpypi
91+
url: https://test.pypi.org/p/pyiceberg_core
92+
93+
permissions:
94+
id-token: write # IMPORTANT: mandatory for trusted publishing
95+
96+
steps:
97+
- name: Download all the dists
98+
uses: actions/download-artifact@v4
99+
with:
100+
pattern: wheels-*
101+
merge-multiple: true
102+
path: bindings/python/dist
103+
- name: List downloaded artifacts
104+
run: ls -R bindings/python/dist
105+
- name: Publish to TestPyPI
106+
uses: pypa/gh-action-pypi-publish@release/v1
107+
with:
108+
repository-url: https://test.pypi.org/legacy/
109+
skip-existing: true
110+
packages-dir: bindings/python/dist

0 commit comments

Comments
 (0)