Skip to content

Manage dependencies with PDM #250

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
vbaltrusaitis-reef merged 22 commits into from
Feb 5, 2024
Merged

Manage dependencies with PDM #250

Show file tree
Hide file tree
Changes from 14 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
8fc6e4d
Build with PDM
vbaltrusaitis-reef Jan 16, 2024
fba1446
Fix docker tests
vbaltrusaitis-reef Jan 24, 2024
e416ee2
Minor fixes
vbaltrusaitis-reef Jan 31, 2024
03ff618
Use pdm in push_docker GH action
vbaltrusaitis-reef Jan 31, 2024
639f957
Check b2sdk dependency before making a release commit
vbaltrusaitis-reef Jan 31, 2024
9662aea
Build the docs with pdm
vbaltrusaitis-reef Jan 31, 2024
c6a20e2
Describe pdm use in CONTRIBUTING.md
vbaltrusaitis-reef Jan 31, 2024
b1e8028
Remove setup.py
vbaltrusaitis-reef Jan 31, 2024
a54e375
In nox sessions install as non-editable
vbaltrusaitis-reef Jan 31, 2024
2cb2de1
Format with yapf
vbaltrusaitis-reef Feb 1, 2024
38e191d
Install editable by default
vbaltrusaitis-reef Feb 1, 2024
d352158
Dump license before building docker
vbaltrusaitis-reef Feb 1, 2024
2a7c12d
Add changelog entry
vbaltrusaitis-reef Feb 1, 2024
de594eb
Style fix
vbaltrusaitis-reef Feb 1, 2024
d9da353
Multi-stage docker build
vbaltrusaitis-reef Feb 2, 2024
cae6b12
Format with yapf
vbaltrusaitis-reef Feb 2, 2024
774919d
Update pdm.lock to match pyproject.toml
vbaltrusaitis-reef Feb 2, 2024
0b95e45
Fix bundling
vbaltrusaitis-reef Feb 2, 2024
92ca332
Merge remote-tracking branch 'backblaze/master'
vbaltrusaitis-reef Feb 2, 2024
7e69959
Improve changelog entry
vbaltrusaitis-reef Feb 5, 2024
82aec06
Install b2 in /opt/b2 in Docker image
vbaltrusaitis-reef Feb 5, 2024
3463fff
Merge remote-tracking branch 'backblaze/master' into pdm
vbaltrusaitis-reef Feb 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/workflows/cd.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ jobs:
with:
python-version: ${{ env.PYTHON_DEFAULT_VERSION }}
- name: Install dependencies
run: python -m pip install --upgrade nox pip setuptools
run: python -m pip install --upgrade nox pdm
- name: Build the distribution
id: build
run: nox -vs build >> $GITHUB_OUTPUT
Expand Down Expand Up @@ -64,7 +64,7 @@ jobs:
run: |
apt-get -y update
apt-get -y install patchelf
python -m pip install --upgrade nox pip setuptools
python -m pip install --upgrade nox pdm
git config --global --add safe.directory '*'
- name: Bundle the distribution
id: bundle
Expand Down Expand Up @@ -97,7 +97,7 @@ jobs:
with:
python-version: ${{ env.PYTHON_DEFAULT_VERSION }}
- name: Install dependencies
run: python -m pip install --upgrade nox pip setuptools
run: python -m pip install --upgrade nox pdm
- name: Bundle the distribution
id: bundle
shell: bash
Expand Down
53 changes: 25 additions & 28 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,11 @@ jobs:
with:
ignore_words_list: datas re-use
- name: Set up Python ${{ env.PYTHON_DEFAULT_VERSION }}
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_DEFAULT_VERSION }}
- name: Install dependencies
run: python -m pip install --upgrade nox pip setuptools
run: python -m pip install --upgrade nox pdm
- name: Run linters
run: nox -vs lint
- name: Validate new changelog entries
Expand All @@ -45,7 +45,7 @@ jobs:
with:
python-version: ${{ env.PYTHON_DEFAULT_VERSION }}
- name: Install dependencies
run: python -m pip install --upgrade nox pip setuptools
run: python -m pip install --upgrade nox pdm
- name: Build the distribution
run: nox -vs build >> $GITHUB_OUTPUT
cleanup_buckets:
Expand All @@ -67,7 +67,7 @@ jobs:
cache: "pip"
- name: Install dependencies
if: ${{ env.B2_TEST_APPLICATION_KEY != '' && env.B2_TEST_APPLICATION_KEY_ID != '' }} # TODO: skip this whole job instead
run: python -m pip install --upgrade nox pip setuptools
run: python -m pip install --upgrade nox pdm
- name: Find and remove old buckets
if: ${{ env.B2_TEST_APPLICATION_KEY != '' && env.B2_TEST_APPLICATION_KEY_ID != '' }} # TODO: skip this whole job instead
run: nox -vs cleanup_buckets
Expand All @@ -81,31 +81,30 @@ jobs:
fail-fast: false
matrix:
os: ["ubuntu-latest", "macos-latest", "windows-latest"]
# pypy version pin was required due 7.3.13 being broken https://foss.heptapod.net/pypy/pypy/-/issues/4021
python-version: ["3.7", "3.8", "3.9", "3.10", "3.11", "3.12", "pypy-3.10-nightly"]
python-version: ["3.7", "3.8", "3.9", "3.10", "3.11", "3.12", "pypy3.9", "pypy3.10"]
exclude:
- os: "macos-latest"
python-version: "pypy-3.10-nightly"
python-version: "pypy3.10"
- os: "windows-latest"
python-version: "pypy-3.10-nightly"
python-version: "pypy3.10"
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
- name: Install dependencies
run: python -m pip install --upgrade nox pip setuptools
run: python -m pip install --upgrade nox pdm
- name: Run unit tests
run: nox -vs unit
run: nox -vs unit -p ${{ matrix.python-version }}
- name: Run integration tests (without secrets)
run: nox -vs integration -- -m "not require_secrets"
run: nox -vs integration -p ${{ matrix.python-version }} -- -m "not require_secrets"
- name: Run integration tests (with secrets)
# Limit CI workload by running integration tests with secrets only on edge Python versions.
if: ${{ env.B2_TEST_APPLICATION_KEY != '' && env.B2_TEST_APPLICATION_KEY_ID != '' && contains(fromJSON('["3.7", "pypy-3.10-nightly", "3.12"]'), matrix.python-version) }}
run: nox -vs integration -- -m "require_secrets" --cleanup
if: ${{ env.B2_TEST_APPLICATION_KEY != '' && env.B2_TEST_APPLICATION_KEY_ID != '' && contains(fromJSON('["3.7", "pypy3.10", "3.12"]'), matrix.python-version) }}
run: nox -vs integration -p ${{ matrix.python-version }} -- -m "require_secrets" --cleanup
test-docker:
needs: cleanup_buckets
env:
Expand All @@ -117,13 +116,11 @@ jobs:
with:
fetch-depth: 0
- name: Set up Python ${{ env.PYTHON_DEFAULT_VERSION }}
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_DEFAULT_VERSION }}
- name: setup sudo NOX_PYTHONS
run: echo NOX_PYTHONS=$(sudo python3 --version | cut -d ' ' -f 2) >> "$GITHUB_ENV"
- name: Install dependencies
run: sudo python -m pip install --upgrade nox pip setuptools
run: sudo python -m pip install --upgrade nox pdm
- name: Generate Dockerfile
run: nox -vs generate_dockerfile
- name: Set up QEMU
Expand All @@ -139,7 +136,7 @@ jobs:
platforms: linux/amd64
- name: Run tests with docker
if: ${{ env.B2_TEST_APPLICATION_KEY != '' && env.B2_TEST_APPLICATION_KEY_ID != '' }}
run: sudo NOX_PYTHONS=$NOX_PYTHONS B2_TEST_APPLICATION_KEY=${{ env.B2_TEST_APPLICATION_KEY }} B2_TEST_APPLICATION_KEY_ID=${{ env.B2_TEST_APPLICATION_KEY_ID }} nox -vs docker_test -- backblazeit/b2:test
run: nox -vs docker_test -- backblazeit/b2:test
test-linux-bundle:
needs: cleanup_buckets
env:
Expand All @@ -158,7 +155,7 @@ jobs:
run: |
apt-get -y update
apt-get -y install patchelf
python -m pip install --upgrade nox pip setuptools
python -m pip install --upgrade nox pdm
git config --global --add safe.directory '*'
- name: Bundle the distribution
id: bundle
Expand All @@ -167,10 +164,10 @@ jobs:
id: hashes
run: nox -vs make_dist_digest
- name: Run integration tests (without secrets)
run: nox -vs integration -- --sut=${{ steps.bundle.outputs.sut_path }} -m "not require_secrets"
run: nox -vs integration -p ${{ env.PYTHON_DEFAULT_VERSION }} -- --sut=${{ steps.bundle.outputs.sut_path }} -m "not require_secrets"
- name: Run integration tests (with secrets)
if: ${{ env.B2_TEST_APPLICATION_KEY != '' && env.B2_TEST_APPLICATION_KEY_ID != '' }}
run: nox -vs integration -- --sut=${{ steps.bundle.outputs.sut_path }} -m "require_secrets" --cleanup
run: nox -vs integration -p ${{ env.PYTHON_DEFAULT_VERSION }} -- --sut=${{ steps.bundle.outputs.sut_path }} -m "require_secrets" --cleanup
- name: Upload assets
if: failure()
uses: actions/upload-artifact@v2
Expand All @@ -193,11 +190,11 @@ jobs:
with:
fetch-depth: 0
- name: Set up Python ${{ env.PYTHON_DEFAULT_VERSION }}
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_DEFAULT_VERSION }}
- name: Install dependencies
run: python -m pip install --upgrade nox pip setuptools
run: python -m pip install --upgrade nox pdm
- name: Bundle the distribution
id: bundle
shell: bash
Expand All @@ -206,10 +203,10 @@ jobs:
id: hashes
run: nox -vs make_dist_digest
- name: Run integration tests (without secrets)
run: nox -vs integration -- --sut=${{ steps.bundle.outputs.sut_path }} -m "not require_secrets"
run: nox -vs integration -p ${{ env.PYTHON_DEFAULT_VERSION }} -- --sut=${{ steps.bundle.outputs.sut_path }} -m "not require_secrets"
- name: Run integration tests (with secrets)
if: ${{ env.B2_TEST_APPLICATION_KEY != '' && env.B2_TEST_APPLICATION_KEY_ID != '' }}
run: nox -vs integration -- --sut=${{ steps.bundle.outputs.sut_path }} -m "require_secrets" --cleanup
run: nox -vs integration -p ${{ env.PYTHON_DEFAULT_VERSION }} -- --sut=${{ steps.bundle.outputs.sut_path }} -m "require_secrets" --cleanup
- name: Upload assets
if: failure()
uses: actions/upload-artifact@v2
Expand All @@ -225,7 +222,7 @@ jobs:
with:
fetch-depth: 0
- name: Set up Python ${{ env.PYTHON_DEFAULT_VERSION }}
uses: actions/setup-python@v4
uses: actions/setup-python@v5
with:
python-version: ${{ env.PYTHON_DEFAULT_VERSION }}
- name: Install dependencies
Expand All @@ -234,6 +231,6 @@ jobs:
run: |
sudo apt-get update -y
sudo apt-get install -y graphviz plantuml
python -m pip install --upgrade nox pip setuptools
python -m pip install --upgrade nox pdm
- name: Build the docs
run: nox --non-interactive -vs doc
8 changes: 4 additions & 4 deletions .github/workflows/push_docker.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,11 @@ jobs:
with:
python-version: ${{ env.PYTHON_DEFAULT_VERSION }}
- name: Install dependencies
run: python -m pip install --upgrade nox pip setuptools
run: python -m pip install --upgrade nox pdm
- name: Build Dockerfile
run: nox -vs generate_dockerfile
- name: Dump licenses file
run: nox -vs dump_license
- name: Set up QEMU
uses: docker/setup-qemu-action@v2
- name: Set up Docker Buildx
Expand All @@ -34,11 +36,9 @@ jobs:
with:
username: ${{ secrets.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: install setuptools_scm
run: pip install setuptools_scm
- name: get version
id: package_version
run: echo package_version=`python -m setuptools_scm` >> $GITHUB_OUTPUT
run: echo package_version=`pdm show --version --quiet` >> $GITHUB_OUTPUT
- name: echo
run: echo ${{ steps.package_version.outputs.package_version }}
- name: Build and push
Expand Down
4 changes: 3 additions & 1 deletion .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@
.eggs/
.idea
.nox/
.pdm-build/
.pdm-python
.python-version
b2_cli.log
b2.egg-info
Expand All @@ -14,4 +16,4 @@ venv
doc/source/main_help.rst
Dockerfile
b2/licenses_output.txt
*.spec
*.spec
12 changes: 7 additions & 5 deletions .readthedocs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,11 @@ version: 2
build:
os: ubuntu-22.04
tools:
python: "3.11"
python: "3.12"
jobs:
post_create_environment:
- pip install pdm
- pdm export --format requirements --prod --group doc --output requirements-doc.txt --no-hashes
Copy link

@mjurbanski-reef mjurbanski-reef Feb 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why no hashes?

Copy link
Author

@vbaltrusaitis-reef vbaltrusaitis-reef Feb 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you have a non-PyPI dependency (like a dependency on a git repo), pip install -r requirements.doc will error out:

ERROR: Can't verify hashes for these requirements because we don't have a way to hash version control repositories:
    b2sdk@ git+https://github.com/vbaltrusaitis-reef/b2-sdk-python@ac1e1355f1aaac10a398618214021cc7ecb6a1eb from git+https://github.com/vbaltrusaitis-reef/b2-sdk-python@ac1e1355f1aaac10a398618214021cc7ecb6a1eb (from -r requirements-doc.txt (line 21))

To be fair, non-PyPI dependencies shouldn't get into master (and definitely definitely not release), so this is only a convenience for testing. But for this use-case this seems pretty harmless, I'm now on the fence if I should remove --no-hashes.

Copy link

@mjurbanski-reef mjurbanski-reef Feb 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

don't think it matters much - this is run on readthedocs side afterall and only for docs generation;
so the worst case (i.e. someone hacked pypi depdency) our docs get defaced - but at that point it would very cheap price to pay for knowing one of our dependencies were hacked :D


# Build documentation in the docs/ directory with Sphinx
sphinx:
Expand All @@ -20,8 +24,6 @@ formats: all
# Optionally set the version of Python and requirements required to build your docs
python:
install:
- requirements: requirements.txt
- requirements: requirements-doc.txt
- method: pip
path: .
extra_requirements:
- doc
path: .
13 changes: 10 additions & 3 deletions CONTRIBUTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,9 +55,9 @@ for a given github issue).

## Developer Info

You'll need to have [nox](https://github.com/theacodes/nox) installed:
You'll need to have [nox](https://github.com/theacodes/nox) and [pdm](https://pdm-project.org/) installed:

* `pip install nox`
* `pip install nox pdm`

With `nox`, you can run different sessions (default are `lint` and `test`):

Expand All @@ -69,7 +69,6 @@ With `nox`, you can run different sessions (default are `lint` and `test`):
* `generate_dockerfile` -> generate dockerfile
* `docker_test` -> run integration tests against a docker image
* `build_and_test_docker` -> build a docker image and integration tests against it
* `deploy` -> Deploy the distribution to the PyPi.
* `doc` -> Build the documentation.
* `doc_cover` -> Perform coverage analysis for the documentation.

Expand Down Expand Up @@ -105,6 +104,14 @@ With the above setting, session `test` will run on Python 3.9 and 3.10, and all

Given Python interpreters should be installed in the operating system or via [pyenv](https://github.com/pyenv/pyenv).

## Managing dependencies

We use [pdm](https://pdm-project.org/) for managing dependencies and developing locally.
If you want to change any of the project requirements (or requirement bounds) in `pyproject.toml`,
make sure that `pdm.lock` file reflects those changes by using `pdm add`, `pdm update` or other
commands - see [documentation](https://pdm-project.org/latest/). You can verify that lock file
is up to date by running the linter.

## Linting

To run all available linters:
Expand Down
18 changes: 15 additions & 3 deletions Dockerfile.template
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,21 @@ LABEL build-date-iso8601="${build_date}"

WORKDIR /root

# due https://github.com/moby/moby/issues/47021 we cannot have /root/.cache leftover as it causes random errors in CI
RUN --mount=type=bind,source=${tar_path}/${tar_name},target=/tmp/${tar_name} \
pip install --no-cache-dir /tmp/${tar_name}[full] && rm -rf /root/.cache
# In case we're installing from git repo
RUN apt-get update -y && apt-get install git -y

RUN pip install -U pdm

ENV PDM_BUILD_SCM_VERSION=${version}
RUN --mount=type=bind,source=./b2,target=/b2/b2 \
--mount=type=bind,source=./pyproject.toml,target=/b2/pyproject.toml \
--mount=type=bind,source=./pdm.lock,target=/b2/pdm.lock \
Copy link

@mjurbanski-reef mjurbanski-reef Feb 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

at first I was wondering why not use wheel/tar, but it does make sense to use lock indeed 👍

Copy link
Author

@vbaltrusaitis-reef vbaltrusaitis-reef Feb 1, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, lock is half the point. I did consider (and try out) a wheel+requirements approach: install dependencies using exported requirement file (pdm export --no-self [...]) and then install a b2 wheel on top of that with --no-deps . This would get rid of pdm within the container, less files to copy, it's nice. But in this case, I feel like I should definitely export requirements with hashes (see comment about the --no-hashes flag in readthedocs config). However, this would fail the docker build when you have b2sdk from git as a dependency. And that makes for an unpleasant development experience in case you're trying out unreleased b2sdk features, you want to make a draft PR that uses them and see if all tests pass.

This was a bit of a dilemma, I feel that exported requirements + wheel is nicer in many ways.

Copy link

@mjurbanski-reef mjurbanski-reef Feb 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

multi-layered dockerfilerfile is the way to go long term, not necessary now

How big image is after this change? our last b2 image was 170MB, so unless this one is much bigger (>250MB) then I feel we can easily try doing it later down the line.

When using requirements.txt I tried doing multilayer, but other than it being more error prone, the image size didn't differ much at all.

Copy link
Author

@vbaltrusaitis-reef vbaltrusaitis-reef Feb 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked the image size and yes, it increased by a lot after these changes, to 300Mb. So I implemented a multi-stage build and now it's 146M.

--mount=type=bind,source=./LICENSE,target=/b2/LICENSE \
--mount=type=bind,source=./README.md,target=/b2/README.md \
cd /b2 \
&& pdm sync --prod --group full --no-editable --global --project . \
# due https://github.com/moby/moby/issues/47021 we cannot have /root/.cache leftover as it causes random errors in CI
&& rm -rf /root/.cache

ENTRYPOINT ["b2"]
CMD ["--help"]
4 changes: 0 additions & 4 deletions MANIFEST.in
View file
Open in desktop

This file was deleted.

1 change: 1 addition & 0 deletions changelog.d/+pdm.infrastructure.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Use pdm for building, testing and managing dependencies.
Copy link

@mjurbanski-reef mjurbanski-reef Feb 2, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think seeing at issue reported in other repo it is worth mentioning here that tests and other superfluous files have been removed from sdist tarball.

Loading