Skip to content

Commit fe96604

Browse files
BewareMyPowerBewareMyPower
authored
Add the release process for Python client (#68)
Fixes #64 It also adds a `upload-pypi.sh` that downloads the wheels from the Apache's SVN project and upload them to PyPI.
1 parent fe6dc94 commit fe96604

File tree

4 files changed

+306
-1
lines changed

4 files changed

+306
-1
lines changed

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -110,7 +110,7 @@ python3 ./tests/pulsar_test.py 'PulsarTest.test_tls_auth'
110110
Pulsar Python Client uses [pydoctor](https://github.com/twisted/pydoctor) to generate API docs. To generate by yourself, run the following command in the root path of this repository:
111111

112112
```bash
113-
pip3 install pydoctor
113+
sudo python3 -m pip install pydoctor
114114
pydoctor --make-html \
115115
--html-viewsource-base=https://github.com/apache/pulsar-client-python/tree/<release-version-tag> \
116116
--docformat=numpy --theme=readthedocs \

‎RELEASE.md‎

Lines changed: 253 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,253 @@
1+
<!--
2+
3+
Licensed to the Apache Software Foundation (ASF) under one
4+
or more contributor license agreements. See the NOTICE file
5+
distributed with this work for additional information
6+
regarding copyright ownership. The ASF licenses this file
7+
to you under the Apache License, Version 2.0 (the
8+
"License"); you may not use this file except in compliance
9+
with the License. You may obtain a copy of the License at
10+
11+
http://www.apache.org/licenses/LICENSE-2.0
12+
13+
Unless required by applicable law or agreed to in writing,
14+
software distributed under the License is distributed on an
15+
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
16+
KIND, either express or implied. See the License for the
17+
specific language governing permissions and limitations
18+
under the License.
19+
20+
-->
21+
22+
# Release Process
23+
24+
This page contains instructions for Pulsar committers on how to perform a release for the Pulsar Python client.
25+
26+
## Preparation
27+
28+
> **Note**
29+
>
30+
> The term `major/minor releases` used throughout this document is defined as follows:
31+
> - Major releases refer to feature releases, such as 3.0.0, 3.1.0, and so on.
32+
> - Minor releases refer to bug-fix releases, such as 3.0.1, 3.0.2, and so on.
33+
>
34+
> This guide use `X.Y.Z` or `X.Y` to represent the actual versions like `3.0.0` or `3.0`.
35+
36+
For major releases, you should create a new branch named `branch-X.Y` once all PRs with the X.Y.0 milestone are merged. If some PRs with the X.Y.0 milestone are still working in progress and might take much time to complete, you can move them to the next milestone if they are not important. In this case, you'd better notify the author in the PR.
37+
38+
For minor releases, if there are no disagreements, you should cherry-pick all merged PRs with the `release/X.Y.Z` labels into `branch-X.Y`. After these PRs are cherry-picked, you should add the `cherry-picked/branch-X.Y` labels.
39+
40+
Sometimes some PRs cannot be cherry-picked cleanly, you might need to create a separate PR and move the `release/X.Y.Z` label from the original PR to it. In this case, you can ask the author to help create the new PR.
41+
42+
For PRs that are still open, you can choose to delay them to the next release or ping other committers to review so that they can be merged.
43+
44+
## Requirements
45+
46+
If you haven't already done it, [create and publish the GPG key](https://pulsar.apache.org/contribute/create-gpg-keys/) to sign the release artifacts.
47+
48+
## Upgrade the C++ client dependency
49+
50+
During the development, the C++ client dependency might be downloaded from an unofficial release. But when releasing the Python client, the dependency must be downloaded from an official release. You should modify the base url in [dep-url.sh](./build-support/dep-url.sh).
51+
52+
Example: https://github.com/apache/pulsar-client-python/pull/62
53+
54+
## Cut the candidate release
55+
56+
After all necessary PRs are cherry-picked to `branch-X.Y`, you should cut the release by pushing a tag.
57+
58+
```bash
59+
git checkout branch-X.Y
60+
git push origin branch-X.Y
61+
# N starts with 1
62+
git tag vX.Y.Y-candidate-N
63+
git push origin vX.Y.Y-candidate-N
64+
```
65+
66+
Then, [create a new milestone](https://github.com/apache/pulsar-client-python/milestones/new) for the next major release.
67+
68+
## Sign and stage the candidate release
69+
70+
After a tag is pushed, a workflow will be triggered to build the Python wheels in GitHub Actions. You can find it in https://github.com/apache/pulsar-client-python/actions/workflows/ci-build-release-wheels.yaml
71+
72+
For example, https://github.com/apache/pulsar-client-python/actions/runs/3709463737 is the workflow of `v3.0.0-candidate-3`, the workflow id is 3709463737. Remember the workflow id, which will be passed as an argument of `stage-release.sh` in the following step. Once the workflow is completed, the wheels will be available for downloading.
73+
74+
Generate a GitHub token by following the guide [here](https://github.com/settings/tokens). The `repo` and `workflow` checkboxes must be selected. Then export the token as the environment variable:
75+
76+
```bash
77+
export GITHUB_TOKEN=<your-token>
78+
```
79+
80+
Edit `~/.gnupg/gpg.conf` to ensure the default GPG key is from your Apache mail (`<your-name>@apache.org`):
81+
82+
```
83+
default-key <key fingerprint>
84+
```
85+
86+
Make sure `curl`, `jq`, `unzip`, `gpg`, `shasum` commands are available. Then you should run the following commands in an empty directory:
87+
88+
```bash
89+
svn co https://dist.apache.org/repos/dist/dev/pulsar pulsar-dist-dev-keys --depth empty
90+
cd pulsar-dist-dev-keys
91+
svn mkdir pulsar-client-python-X.Y.Z-candidate-N && cd pulsar-client-python-X.Y.Z-candidate-N
92+
# PROJECT_DIR is the directory of the pulsar-client-python repository
93+
$PROJECT_DIR/build-support/stage-release.sh vX.Y.Z-candidate-N $WORKFLOW_ID
94+
svn add *
95+
svn ci -m "Staging artifacts and signature for Python client X.Y.Z-candidate-N"
96+
```
97+
98+
## Start the vote for the candidate
99+
100+
Send an email to [email protected] to start the vote for the candidate:
101+
102+
```
103+
To: [email protected]
104+
Subject: [VOTE] Pulsar Client Python Release X.Y.Z Candidate N
105+
106+
This is the third release candidate for Apache Pulsar Client Python,
107+
version X.Y.Z.
108+
109+
It fixes the following issues:
110+
https://github.com/apache/pulsar-client-python/milestone/<milestone-id>?closed=1
111+
112+
*** Please download, test and vote on this release. This vote will
113+
stay open for at least 72 hours ***
114+
115+
Python wheels:
116+
https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-python-X.Y.Z-candidate-N/
117+
118+
The supported python versions are 3.7, 3.8, 3.9 and 3.10. The
119+
supported platforms and architectures are:
120+
- Windows x86_64 (windows/)
121+
- glibc-based Linux x86_64 (linux-glibc-x86_64/)
122+
- glibc-based Linux arm64 (linux-glibc-arm64/)
123+
- musl-based Linux x86_64 (linux-musl-x86_64/)
124+
- musl-based Linux arm64 (linux-musl-arm64/)
125+
- macOS universal 2 (macos/)
126+
127+
The tag to be voted upon: vX.Y.Z-candidate-N
128+
(<commit-id>)
129+
https://github.com/apache/pulsar-client-python/releases/tag/vX.Y.Z-candidate-N
130+
131+
Pulsar's KEYS file containing PGP keys you use to sign the release:
132+
https://dist.apache.org/repos/dist/dev/pulsar/KEYS
133+
134+
Please download the Python wheels and follow the README to test.
135+
```
136+
137+
Once there are at least 3 binding +1s, the vote will be ready to close and you can continue the following steps. If there is something wrong with the candidate, you need to fix it and repeat the steps from the **Cut the next candidate** section again.
138+
139+
## Move main branch to next version
140+
141+
You need to move the main version to the next iteration.
142+
143+
```bash
144+
git checkout main
145+
sed -i.bak 's/X.Y.0a1/X.Y+1.0a1/' pulsar/__about__.py
146+
git add pulsar/__about__.py
147+
git commit -m "Bumped version to X.Y.0a1"
148+
```
149+
150+
Since this needs to be merged into main, you need to follow the regular process and create a Pull Request on GitHub.
151+
152+
## Promote the release
153+
154+
Ask a PMC member to promote the release:
155+
156+
```bash
157+
svn move -m "Release Apache Pulsar Client Python X.Y.Z" \
158+
https://dist.apache.org/repos/dist/dev/pulsar/pulsar-client-python-X.Y.Z-candidate-N \
159+
https://dist.apache.org/repos/dist/release/pulsar/pulsar-client-python-X.Y.Z
160+
```
161+
162+
## Upload the wheels to PyPI
163+
164+
1. You need to create an account on PyPI: https://pypi.org/account/register/
165+
2. Ask anyone that has been a release manager before to add you as a maintainer for [pulsar-client](https://pypi.org/manage/project/pulsar-client/releases/) on PyPI
166+
3. Once you have completed the following steps in this section, you can check if the wheels are uploaded successfully in Download files. Remember to switch to the correct version in Release history).
167+
168+
Then, upload the wheels to PyPI:
169+
170+
```bash
171+
sudo python3 -m pip install twine
172+
./build-support/upload-pypi.sh https://dist.apache.org/repos/dist/release/pulsar/pulsar-client-python-X.Y.Z
173+
```
174+
175+
To verify the wheels have been uploaded successfully, you can try installing the wheel:
176+
177+
```bash
178+
python3 -m pip install pulsar-client==X.Y.Z
179+
```
180+
181+
## Write release notes
182+
183+
Push the official tag:
184+
185+
```bash
186+
git checkout vX.Y.Z-candidate-N
187+
git tag vX.Y.Z
188+
git push origin vX.Y.Z
189+
```
190+
191+
Then, [create a release](https://github.com/apache/pulsar-client-python/releases/new). Choose the `vX.Y.Z` tag and click the `Generate release notes` button to generate the release note automatically. Here is an example release note: https://github.com/apache/pulsar-client-python/releases/tag/v3.0.0
192+
193+
Then, create a PR in [`pulsar-site`](https://github.com/apache/pulsar-site) repo to update the website. Here is an example: https://github.com/apache/pulsar-site/pull/343
194+
195+
## Generate the API documents
196+
197+
For minor releases, skip this section. For major releases, you should generate the HTML files into the [`pulsar-site`](https://github.com/apache/pulsar-site) repo:
198+
199+
```bash
200+
git clone [email protected]:apache/pulsar-client-python.git
201+
cd pulsar-client-python
202+
git checkout vX.Y.0
203+
# It's better to replace this URL with the URL of your own fork
204+
git clone [email protected]:apache/pulsar-site.git
205+
sudo python3 -m pip install pydoctor
206+
pydoctor --make-html \
207+
--html-viewsource-base=https://github.com/apache/pulsar-client-python/tree/vX.Y.0 \
208+
--docformat=numpy --theme=readthedocs \
209+
--intersphinx=https://docs.python.org/3/objects.inv \
210+
--html-output=./pulsar-site/site2/website-next/static/api/python/X.Y.x \
211+
pulsar-client-python/pulsar
212+
cd pulsar-site
213+
git checkout -b py-docs-X.Y
214+
git add .
215+
git commit -m "Generate Python client X.Y.0 doc"
216+
git push origin py-docs-X.Y
217+
```
218+
219+
Then open a PR like: https://github.com/apache/pulsar-site/pull/342
220+
221+
## Announce the release
222+
223+
Use your Apache email account (`[email protected]`) to send an email like:
224+
225+
```
226+
To: [email protected], [email protected], [email protected]
227+
Subject: [ANNOUNCE] Apache Pulsar Client Python X.Y.Z released
228+
229+
The Apache Pulsar team is proud to announce Apache Pulsar Client
230+
Python version X.Y.Z.
231+
232+
Pulsar is a highly scalable, low latency messaging platform running on
233+
commodity hardware. It provides simple pub-sub semantics over topics,
234+
guaranteed at-least-once delivery of messages, automatic cursor management for
235+
subscribers, and cross-datacenter replication.
236+
237+
You can download the source code and the Python wheels in:
238+
https://archive.apache.org/dist/pulsar/pulsar-client-python-X.Y.Z/
239+
240+
The Python wheels were uploaded to PyPI as well so that they can be
241+
installed by `pip install pulsar-client==X.Y.Z`.
242+
243+
Release Notes are at:
244+
https://pulsar.apache.org/release-notes/versioned/client-python-X.Y.Z/
245+
246+
We would like to thank the contributors that made the release possible.
247+
248+
Regards,
249+
250+
The Pulsar Team
251+
```
252+
253+
Send the email in plain text mode since the [email protected] mailing list will reject messages with text/html content. In Gmail, there's an option to set Plain text mode in the ⋮/ More options menu.

‎build-support/stage-release.sh‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,9 @@ WORKFLOW_ID=$2
3535

3636
# Download the source tar
3737
curl -O -L https://github.com/apache/pulsar-client-python/archive/refs/tags/$TAG.tar.gz
38+
39+
# Remove the "-candidate-N" suffix
40+
VERSION=$(echo $VERSION | sed 's/-candidate-.*//')
3841
mv $TAG.tar.gz pulsar-client-python-$VERSION.tar.gz
3942

4043
# Download the Python wheels

‎build-support/upload-pypi.sh‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
#!/usr/bin/env bash
2+
#
3+
# Licensed to the Apache Software Foundation (ASF) under one
4+
# or more contributor license agreements. See the NOTICE file
5+
# distributed with this work for additional information
6+
# regarding copyright ownership. The ASF licenses this file
7+
# to you under the Apache License, Version 2.0 (the
8+
# "License"); you may not use this file except in compliance
9+
# with the License. You may obtain a copy of the License at
10+
#
11+
# http://www.apache.org/licenses/LICENSE-2.0
12+
#
13+
# Unless required by applicable law or agreed to in writing,
14+
# software distributed under the License is distributed on an
15+
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
16+
# KIND, either express or implied. See the License for the
17+
# specific language governing permissions and limitations
18+
# under the License.
19+
#
20+
21+
set -e
22+
23+
if [[ $# -lt 1 ]]; then
24+
echo "Usage: $0 BASE_URL"
25+
exit 1
26+
fi
27+
28+
BASE_URL=$1
29+
30+
download_wheels() {
31+
URL=$BASE_URL/$1
32+
LIST=$(curl -L $URL | grep ".whl<" | sed 's/^.*>pulsar\(.*\)\.whl.*/pulsar\1.whl/')
33+
for WHEEL in $LIST; do
34+
if [[ ! -f $WHEEL ]]; then
35+
echo "Download $URL/$WHEEL"
36+
curl -O -L $URL/$WHEEL
37+
fi
38+
done
39+
}
40+
41+
download_wheels linux-glibc-arm64
42+
download_wheels linux-glibc-x86_64
43+
download_wheels linux-musl-arm64
44+
download_wheels linux-musl-x86_64
45+
download_wheels macos
46+
download_wheels windows
47+
48+
twine upload pulsar_client-*.whl
49+
rm -f pulsar_client-*.whl

0 commit comments

Comments
 (0)