Skip to content

Commit 6ac2a82

Browse files
glide-theglide-the
authored
CICD github action build publish pypi、Release Tag (#28)
## Add CI/CD build process, which includes the following tasks: compiling and building, releasing to test PyPI, testing with pytest, releasing to PyPI, and publishing a Release Tag. ### your configuration env to github Repository secrets [using-secrets-in-github-actions](https://docs.github.com/en/actions/security-guides/using-secrets-in-github-actions) ### pypi pypi token generate for docs [publishers](https://blog.pypi.org/posts/2023-04-20-introducing-trusted-publishers/) or [publisher](https://docs.pypi.org/trusted-publishers/adding-a-publisher/) #### Environment - PYPI_API_TOKEN - TEST_PYPI_API_TOKEN ### ZhipuAI ZhipuAI integration_tests for docs [bigmodel](https://open.bigmodel.cn/) #### Environment - ZHIPUAI_API_KEY - ZHIPUAI_BASE_URL ### Release Tag [release-action](https://github.com/ncipollo/release-action) [managing-your-personal-access-tokens](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/managing-your-personal-access-tokens) 增加CICD构建流程，包含了如下作业，编译构建、发布test pypi、测试pytest、发布pypi、发布Release Tag 需要配置仓库变量才能使用，可以在这里查看配置方法 [using-secrets-in-github-actions](https://docs.github.com/en/actions/security-guides/using-secrets-in-github-actions) ### pypi pypi token生成可以选择配置变量也可以选择pipy加入开发人员[publisher](https://docs.pypi.org/trusted-publishers/adding-a-publisher/) #### Environment - PYPI_API_TOKEN - TEST_PYPI_API_TOKEN ### ZhipuAI 智谱的sdk配置需要从官网获取 [bigmodel](https://open.bigmodel.cn/) #### Environment - ZHIPUAI_API_KEY - ZHIPUAI_BASE_URL
1 parent 386a38c commit 6ac2a82

24 files changed

+1214
-292
lines changed

‎.github/actions/poetry_setup/action.yml‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
# An action for setting up poetry install with caching.
2+
# Using a custom action since the default action does not
3+
# take poetry install groups into account.
4+
# Action code from:
5+
# https://github.com/actions/setup-python/issues/505#issuecomment-1273013236
6+
name: poetry-install-with-caching
7+
description: Poetry install with support for caching of dependency groups.
8+
9+
inputs:
10+
python-version:
11+
description: Python version, supporting MAJOR.MINOR only
12+
required: true
13+
14+
poetry-version:
15+
description: Poetry version
16+
required: true
17+
18+
cache-key:
19+
description: Cache key to use for manual handling of caching
20+
required: true
21+
22+
working-directory:
23+
description: Directory whose poetry.lock file should be cached
24+
required: true
25+
26+
runs:
27+
using: composite
28+
steps:
29+
- uses: actions/setup-python@v5
30+
name: Setup python ${{ inputs.python-version }}
31+
id: setup-python
32+
with:
33+
python-version: ${{ inputs.python-version }}
34+
35+
- uses: actions/cache@v4
36+
id: cache-bin-poetry
37+
name: Cache Poetry binary - Python ${{ inputs.python-version }}
38+
env:
39+
SEGMENT_DOWNLOAD_TIMEOUT_MIN: "1"
40+
with:
41+
path: |
42+
/opt/pipx/venvs/poetry
43+
# This step caches the poetry installation, so make sure it's keyed on the poetry version as well.
44+
key: bin-poetry-${{ runner.os }}-${{ runner.arch }}-py-${{ inputs.python-version }}-${{ inputs.poetry-version }}
45+
46+
- name: Refresh shell hashtable and fixup softlinks
47+
if: steps.cache-bin-poetry.outputs.cache-hit == 'true'
48+
shell: bash
49+
env:
50+
POETRY_VERSION: ${{ inputs.poetry-version }}
51+
PYTHON_VERSION: ${{ inputs.python-version }}
52+
run: |
53+
set -eux
54+
55+
# Refresh the shell hashtable, to ensure correct `which` output.
56+
hash -r
57+
58+
# `actions/cache@v3` doesn't always seem able to correctly unpack softlinks.
59+
# Delete and recreate the softlinks pipx expects to have.
60+
rm /opt/pipx/venvs/poetry/bin/python
61+
cd /opt/pipx/venvs/poetry/bin
62+
ln -s "$(which "python$PYTHON_VERSION")" python
63+
chmod +x python
64+
cd /opt/pipx_bin/
65+
ln -s /opt/pipx/venvs/poetry/bin/poetry poetry
66+
chmod +x poetry
67+
68+
# Ensure everything got set up correctly.
69+
/opt/pipx/venvs/poetry/bin/python --version
70+
/opt/pipx_bin/poetry --version
71+
72+
- name: Install poetry
73+
if: steps.cache-bin-poetry.outputs.cache-hit != 'true'
74+
shell: bash
75+
env:
76+
POETRY_VERSION: ${{ inputs.poetry-version }}
77+
PYTHON_VERSION: ${{ inputs.python-version }}
78+
# Install poetry using the python version installed by setup-python step.
79+
run: pipx install "poetry==$POETRY_VERSION" --python '${{ steps.setup-python.outputs.python-path }}' --verbose
80+
81+
- name: Restore pip and poetry cached dependencies
82+
uses: actions/cache@v4
83+
env:
84+
SEGMENT_DOWNLOAD_TIMEOUT_MIN: "4"
85+
WORKDIR: ${{ inputs.working-directory == '' && '.' || inputs.working-directory }}
86+
with:
87+
path: |
88+
~/.cache/pip
89+
~/.cache/pypoetry/virtualenvs
90+
~/.cache/pypoetry/cache
91+
~/.cache/pypoetry/artifacts
92+
${{ env.WORKDIR }}/.venv
93+
key: py-deps-${{ runner.os }}-${{ runner.arch }}-py-${{ inputs.python-version }}-poetry-${{ inputs.poetry-version }}-${{ inputs.cache-key }}-${{ hashFiles(format('{0}/**/poetry.lock', env.WORKDIR)) }}

‎.github/workflows/_release.yml‎

Lines changed: 260 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,260 @@
1+
name: release
2+
run-name: Release ${{ inputs.working-directory }} by @${{ github.actor }}
3+
on:
4+
workflow_call:
5+
inputs:
6+
working-directory:
7+
required: true
8+
type: string
9+
description: "From which folder this pipeline executes"
10+
workflow_dispatch:
11+
inputs:
12+
working-directory:
13+
required: true
14+
type: string
15+
default: '.'
16+
description: "From which folder this pipeline executes"
17+
env:
18+
PYTHON_VERSION: "3.8"
19+
POETRY_VERSION: "1.7.1"
20+
21+
jobs:
22+
build:
23+
if: github.ref == 'refs/heads/main'
24+
environment: Scheduled testing publish
25+
runs-on: ubuntu-latest
26+
27+
outputs:
28+
pkg-name: ${{ steps.check-version.outputs.pkg-name }}
29+
version: ${{ steps.check-version.outputs.version }}
30+
31+
steps:
32+
- uses: actions/checkout@v4
33+
34+
- name: Set up Python + Poetry ${{ env.POETRY_VERSION }}
35+
uses: "./.github/actions/poetry_setup"
36+
with:
37+
python-version: ${{ env.PYTHON_VERSION }}
38+
poetry-version: ${{ env.POETRY_VERSION }}
39+
working-directory: ${{ inputs.working-directory }}
40+
cache-key: release
41+
42+
# We want to keep this build stage *separate* from the release stage,
43+
# so that there's no sharing of permissions between them.
44+
# The release stage has trusted publishing and GitHub repo contents write access,
45+
# and we want to keep the scope of that access limited just to the release job.
46+
# Otherwise, a malicious `build` step (e.g. via a compromised dependency)
47+
# could get access to our GitHub or PyPI credentials.
48+
#
49+
# Per the trusted publishing GitHub Action:
50+
# > It is strongly advised to separate jobs for building [...]
51+
# > from the publish job.
52+
# https://github.com/pypa/gh-action-pypi-publish#non-goals
53+
- name: Build project for distribution
54+
run: poetry build
55+
working-directory: ${{ inputs.working-directory }}
56+
57+
- name: Upload build
58+
uses: actions/upload-artifact@v4
59+
with:
60+
name: dist
61+
path: ${{ inputs.working-directory }}/dist/
62+
63+
- name: Check Version
64+
id: check-version
65+
shell: bash
66+
working-directory: ${{ inputs.working-directory }}
67+
run: |
68+
echo pkg-name="$(poetry version | cut -d ' ' -f 1)" >> $GITHUB_OUTPUT
69+
echo version="$(poetry version --short)" >> $GITHUB_OUTPUT
70+
71+
test-pypi-publish:
72+
needs:
73+
- build
74+
uses:
75+
./.github/workflows/_test_release.yml
76+
with:
77+
working-directory: ${{ inputs.working-directory }}
78+
secrets: inherit
79+
80+
pre-release-checks:
81+
needs:
82+
- build
83+
- test-pypi-publish
84+
environment: Scheduled testing publish
85+
runs-on: ubuntu-latest
86+
steps:
87+
- uses: actions/checkout@v4
88+
89+
# We explicitly *don't* set up caching here. This ensures our tests are
90+
# maximally sensitive to catching breakage.
91+
#
92+
# For example, here's a way that caching can cause a falsely-passing test:
93+
# - Make the langchain package manifest no longer list a dependency package
94+
# as a requirement. This means it won't be installed by `pip install`,
95+
# and attempting to use it would cause a crash.
96+
# - That dependency used to be required, so it may have been cached.
97+
# When restoring the venv packages from cache, that dependency gets included.
98+
# - Tests pass, because the dependency is present even though it wasn't specified.
99+
# - The package is published, and it breaks on the missing dependency when
100+
# used in the real world.
101+
102+
- name: Set up Python + Poetry ${{ env.POETRY_VERSION }}
103+
uses: "./.github/actions/poetry_setup"
104+
with:
105+
python-version: ${{ env.PYTHON_VERSION }}
106+
poetry-version: ${{ env.POETRY_VERSION }}
107+
working-directory: ${{ inputs.working-directory }}
108+
109+
- name: Import published package
110+
shell: bash
111+
working-directory: ${{ inputs.working-directory }}
112+
env:
113+
PKG_NAME: ${{ needs.build.outputs.pkg-name }}
114+
VERSION: ${{ needs.build.outputs.version }}
115+
# Here we use:
116+
# - The default regular PyPI index as the *primary* index, meaning
117+
# that it takes priority (https://pypi.org/simple)
118+
# - The test PyPI index as an extra index, so that any dependencies that
119+
# are not found on test PyPI can be resolved and installed anyway.
120+
# (https://test.pypi.org/simple). This will include the PKG_NAME==VERSION
121+
# package because VERSION will not have been uploaded to regular PyPI yet.
122+
# - attempt install again after 5 seconds if it fails because there is
123+
# sometimes a delay in availability on test pypi
124+
run: |
125+
poetry run pip install \
126+
--extra-index-url https://test.pypi.org/simple/ \
127+
"$PKG_NAME==$VERSION" || \
128+
( \
129+
sleep 5 && \
130+
poetry run pip install \
131+
--extra-index-url https://test.pypi.org/simple/ \
132+
"$PKG_NAME==$VERSION" \
133+
)
134+
135+
# Replace all dashes in the package name with underscores,
136+
# since that's how Python imports packages with dashes in the name.
137+
IMPORT_NAME="$(echo "$PKG_NAME" | sed s/-/_/g)"
138+
139+
poetry run python -c "import $IMPORT_NAME; print(dir($IMPORT_NAME))"
140+
141+
- name: Import test dependencies
142+
run: poetry install --with test
143+
working-directory: ${{ inputs.working-directory }}
144+
145+
# Overwrite the local version of the package with the test PyPI version.
146+
- name: Import published package (again)
147+
working-directory: ${{ inputs.working-directory }}
148+
shell: bash
149+
env:
150+
PKG_NAME: ${{ needs.build.outputs.pkg-name }}
151+
VERSION: ${{ needs.build.outputs.version }}
152+
run: |
153+
poetry run pip install \
154+
--extra-index-url https://test.pypi.org/simple/ \
155+
"$PKG_NAME==$VERSION"
156+
157+
- name: Run unit tests
158+
run: make tests
159+
env:
160+
ZHIPUAI_API_KEY: ${{ secrets.ZHIPUAI_API_KEY }}
161+
ZHIPUAI_BASE_URL: ${{ secrets.ZHIPUAI_BASE_URL }}
162+
working-directory: ${{ inputs.working-directory }}
163+
164+
- name: Run integration tests
165+
env:
166+
ZHIPUAI_API_KEY: ${{ secrets.ZHIPUAI_API_KEY }}
167+
ZHIPUAI_BASE_URL: ${{ secrets.ZHIPUAI_BASE_URL }}
168+
run: make integration_tests
169+
working-directory: ${{ inputs.working-directory }}
170+
171+
publish:
172+
needs:
173+
- build
174+
- test-pypi-publish
175+
- pre-release-checks
176+
environment: Scheduled testing publish
177+
runs-on: ubuntu-latest
178+
permissions:
179+
# This permission is used for trusted publishing:
180+
# https://blog.pypi.org/posts/2023-04-20-introducing-trusted-publishers/
181+
#
182+
# Trusted publishing has to also be configured on PyPI for each package:
183+
# https://docs.pypi.org/trusted-publishers/adding-a-publisher/
184+
id-token: write
185+
186+
defaults:
187+
run:
188+
working-directory: ${{ inputs.working-directory }}
189+
190+
steps:
191+
- uses: actions/checkout@v4
192+
193+
- name: Set up Python + Poetry ${{ env.POETRY_VERSION }}
194+
uses: "./.github/actions/poetry_setup"
195+
with:
196+
python-version: ${{ env.PYTHON_VERSION }}
197+
poetry-version: ${{ env.POETRY_VERSION }}
198+
working-directory: ${{ inputs.working-directory }}
199+
cache-key: release
200+
201+
- uses: actions/download-artifact@v4
202+
with:
203+
name: dist
204+
path: ${{ inputs.working-directory }}/dist/
205+
206+
- name: Publish package distributions to PyPI
207+
uses: pypa/gh-action-pypi-publish@release/v1
208+
with:
209+
packages-dir: ${{ inputs.working-directory }}/dist/
210+
verbose: true
211+
print-hash: true
212+
password: ${{ secrets.PYPI_API_TOKEN }}
213+
# We overwrite any existing distributions with the same name and version.
214+
# This is *only for CI use* and is *extremely dangerous* otherwise!
215+
# https://github.com/pypa/gh-action-pypi-publish#tolerating-release-package-file-duplicates
216+
skip-existing: true
217+
218+
mark-release:
219+
needs:
220+
- build
221+
- test-pypi-publish
222+
- pre-release-checks
223+
- publish
224+
environment: Scheduled testing publish
225+
runs-on: ubuntu-latest
226+
permissions:
227+
# This permission is needed by `ncipollo/release-action` to
228+
# create the GitHub release.
229+
contents: write
230+
231+
defaults:
232+
run:
233+
working-directory: ${{ inputs.working-directory }}
234+
235+
steps:
236+
- uses: actions/checkout@v4
237+
238+
- name: Set up Python + Poetry ${{ env.POETRY_VERSION }}
239+
uses: "./.github/actions/poetry_setup"
240+
with:
241+
python-version: ${{ env.PYTHON_VERSION }}
242+
poetry-version: ${{ env.POETRY_VERSION }}
243+
working-directory: ${{ inputs.working-directory }}
244+
cache-key: release
245+
246+
- uses: actions/download-artifact@v4
247+
with:
248+
name: dist
249+
path: ${{ inputs.working-directory }}/dist/
250+
251+
- name: Create Release
252+
uses: ncipollo/release-action@v1
253+
if: ${{ inputs.working-directory == '.' }}
254+
with:
255+
artifacts: "dist/*"
256+
token: ${{ secrets.GITHUB_TOKEN }}
257+
draft: false
258+
generateReleaseNotes: true
259+
tag: v${{ needs.build.outputs.version }}
260+
commit: main

0 commit comments

Comments
 (0)