Skip to content

Commit de707b6

Browse files
sfoster1sfoster1
authored
chore: use hatch as a build backend for python projects (PD 8.5.0) (#18538)
This is a version of #18377 that is targeted on chore_release-pd-8.5.0. This was a bit of a pain because I accidentally clicked the button that merges edge into that pr and the merge had to be unwound, so I'm only like 85% confident everything came with it. This is a long-time-coming change to switch to using [hatch](https://github.com/pypa/hatch) (or more specifically, [hatchling](https://github.com/pypa/hatch/tree/master/backend)) as a [PEP-517 build backend](https://peps.python.org/pep-0517/) for building our python packages. ## What? Why? Python packaging has been in a bit of a churn for a while. The way it used to work is you wrote a script, called setup.py for convention, that would import utilities from the built in package distutils and later its drop in replacement setuptools that provided a function called setup(); then you called setup() with a bunch of descriptive data, and that function would build some kind of python distributable, like an sdist or a wheel (or, back in the day, an egg). People largely hated this because it meant that you would have to run code to build a package, and that's gross, and also there was no way to figure out how to build a package than writing the code - maybe that setup.py had some other dependencies, you'd never know until you tried. The fix for this was to define an official build system API. Python build systems are now split into backends and frontends. Backends are packages that provide a specific programmatic interface for building packages; frontends are applications that transform user input into the appropriate calls to that interface. These two packages are now commonly configured through the use of a toml file called pyproject.toml with a specific format. This happened a while ago, and the python ecosystem moved on to supporting it. Specifically, setuptools switched to focusing on that, and doesn't want to deal with setup.py anymore. Features traditionally provided by setuptools to make setup.py work have been getting deprecated or outright broken. Most recently, that was editable-install support. This is even more of a problem because setuptools is commonly included as a transitive dependency without a version bound, so it can just update on you and break stuff. Switching to use hatchling for building our projects instead of setup.py lets us avoid these breakages by using a more stable build system; it lets us configure our projects using more modern configuration files, that can expand into also configuring our build frontends; and it can all be done without actually changing the way we interact with these projects as devs, because pip is a PEP-517 build frontend, and that's what pipenv uses to install stuff. So we don't have to switch out pipenv (yet) and everything will Just Keep Working. How? First step is to switch all the python projects to use pyproject.tomls instead of setup.py, requesting hatchling as a build backend. There's two things that don't Just Work with this setup: - We get our version from git, which isn't uncommon, but we switch what tag we look at sometimes, which is uncommon. [This hatch plugin](https://github.com/Opentrons/hatch-plugins/tree/main/hatch-vcs-tunable) we wrote lets you override the settings passed to the git version bridge plugin via environment variables - We exactly coversion all our dependencies, and we don't want to edit the files to do it. [This hatch plugin](https://github.com/Opentrons/hatch-plugins/tree/main/hatch-dependency-coversion) we wrote automatically overwrites the versions of specified dependencies at build time We also need to switch Opentrons/buildroot#246 and Opentrons/oe-core#196 to use this. ## What's left? - system builds working - [x] on ot2 Opentrons/buildroot#246 - [x] on flex Opentrons/oe-core#196 - [x] the other python packages - [x] verifying that this works all the way through a built robot version - [x] on ot2, to boot and run - [x] on flex - verifying that push works - [x] on ot2 - [x] on flex - [x] verifying that the interned build for offline analysis works - [ ] verifying that a pypi push works - [x] checking that everybody is ok with it
1 parent d791a79 commit de707b6

File tree

179 files changed

+10967
-8803
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

179 files changed

+10967
-8803
lines changed

‎.github/workflows/g-code-confirm-tests.yaml

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,7 @@ concurrency:
3030
jobs:
3131
confirm-g-code:
3232
strategy:
33+
fail-fast: false
3334
matrix:
3435
command: ['2-modules', 'swift-smoke', 'swift-turbo', 'omega', 'fast']
3536
name: 'Confirm G-Code (${{ matrix.command }})'
@@ -52,4 +53,13 @@ jobs:
5253
run: make -C g-code-testing check-for-missing-comparison-files
5354

5455
- name: 'Run & Compare to comparison files'
55-
run: make -C g-code-testing test-g-code-${{ matrix.command }}
56+
id: run_tests
57+
run: make -C g-code-testing test-g-code-${{ matrix.command }} test_opts="-s"
58+
59+
- name: 'Upload test results'
60+
if: failure() && steps.run_tests.conclusion == 'failure'
61+
uses: actions/upload-artifact@v4
62+
with:
63+
name: g-code-test-results-${{ matrix.command }}
64+
path: g-code-testing/results
65+
retention-days: 14

‎.github/workflows/robot-server-lint-test.yaml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -81,7 +81,7 @@ jobs:
8181
name: Test with opentrons_hardware
8282
run: make -C robot-server test-cov
8383
- name: Ensure assets build
84-
run: make -C robot-server sdist wheel
84+
run: make -C robot-server wheel
8585
- name: Upload coverage report
8686
uses: 'codecov/codecov-action@v3'
8787
with:

‎.github/workflows/shared-data-test-lint-deploy.yaml

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -58,9 +58,9 @@ jobs:
5858

5959
- uses: './.github/actions/python/setup'
6060
with:
61-
project: 'shared-data/python'
61+
project: 'shared-data'
6262
- name: Lint
63-
run: make -C shared-data/python lint
63+
run: make -C shared-data lint-py
6464

6565
python-test:
6666
name: 'shared-data package python ${{ matrix.python }} tests on ${{ matrix.os }}'
@@ -92,7 +92,7 @@ jobs:
9292
python-version: ${{ matrix.python }}
9393
- uses: './.github/actions/python/setup'
9494
with:
95-
project: 'shared-data/python'
95+
project: 'shared-data'
9696
python-version: ${{ matrix.python }}
9797
- name: 'set complex environment variables'
9898
uses: actions/github-script@v6
@@ -101,13 +101,13 @@ jobs:
101101
const { buildComplexEnvVars, } = require(`${process.env.GITHUB_WORKSPACE}/.github/workflows/utils.js`)
102102
buildComplexEnvVars(core, context)
103103
- name: Test
104-
run: make -C shared-data/python test
104+
run: make -C shared-data test-py
105105
- name: Ensure assets build
106-
run: make -C shared-data/python sdist wheel
106+
run: make -C shared-data dist-py
107107
- name: 'Upload coverage report'
108108
uses: codecov/codecov-action@v3
109109
with:
110-
files: ./shared-data/python/coverage.xml
110+
files: ./shared-data/coverage.xml
111111
flags: shared-data
112112

113113
js-test:
@@ -174,7 +174,7 @@ jobs:
174174
python-version: '3.10'
175175
- uses: './.github/actions/python/setup'
176176
with:
177-
project: 'shared-data/python'
177+
project: 'shared-data'
178178
- name: 'set complex environment variables'
179179
uses: actions/github-script@v6
180180
with:
@@ -186,14 +186,14 @@ jobs:
186186
name: 'upload to test pypi'
187187
uses: './.github/actions/python/pypi-deploy'
188188
with:
189-
project: 'shared-data/python'
189+
project: 'shared-data'
190190
repository_url: 'https://test.pypi.org/legacy/'
191191
password: '${{ secrets.TEST_PYPI_DEPLOY_TOKEN_OPENTRONS_SHARED_DATA }}'
192192
- if: startsWith(env.OT_TAG, 'v')
193193
name: 'upload to pypi'
194194
uses: './.github/actions/python/pypi-deploy'
195195
with:
196-
project: 'shared-data/python'
196+
project: 'shared-data'
197197
repository_url: 'https://upload.pypi.org/legacy/'
198198
password: '${{ secrets.PYPI_DEPLOY_TOKEN_OPENTRONS_SHARED_DATA }}'
199199

‎.gitignore

Lines changed: 0 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -120,25 +120,6 @@ calibrations/
120120
*.sublime-project
121121
*.sublime-workspace
122122

123-
# TODO(mc, 2020-10-30): remove this ignore when we need pyproject.toml
124-
# recent versions of setuptools create pyproject.toml automatically
125-
# we're ignoring the file for now to avoid confusion
126-
api/pyproject.toml
127-
robot-server/pyproject.toml
128-
update-server/pyproject.toml
129-
shared-data/python/pyproject.toml
130-
hardware/pyproject.toml
131-
132-
# These are autogenerated by sdist generation for some reason and really
133-
# don't need to exist since they're just copies of the top level license
134-
# file
135-
api/LICENSE
136-
update-server/LICENSE
137-
shared-data/python/LICENSE
138-
shared-data/LICENSE
139-
robot-server/LICENSE
140-
performance-metrics/LICENSE
141-
142123
# typescript incremental files
143124
*.tsbuildinfo
144125

‎Config.in

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
source "$BR2_EXTERNAL_OPENTRONS_MONOREPO_PATH/api/Config.in"
33
source "$BR2_EXTERNAL_OPENTRONS_MONOREPO_PATH/update-server/Config.in"
44
source "$BR2_EXTERNAL_OPENTRONS_MONOREPO_PATH/robot-server/Config.in"
5-
source "$BR2_EXTERNAL_OPENTRONS_MONOREPO_PATH/shared-data/python/Config.in"
5+
source "$BR2_EXTERNAL_OPENTRONS_MONOREPO_PATH/shared-data/Config.in"
66
source "$BR2_EXTERNAL_OPENTRONS_MONOREPO_PATH/system-server/Config.in"
77
source "$BR2_EXTERNAL_OPENTRONS_MONOREPO_PATH/server-utils/Config.in"
88
source "$BR2_EXTERNAL_OPENTRONS_MONOREPO_PATH/hardware/Config.in"

‎Dockerfile

Lines changed: 7 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -9,21 +9,16 @@ COPY LICENSE LICENSE
99

1010
COPY shared-data shared-data
1111

12-
COPY server-utils/setup.py server-utils/setup.py
13-
COPY server-utils/server_utils server-utils/server_utils
12+
COPY server-utils server-utils
1413

15-
COPY api/MANIFEST.in api/MANIFEST.in
16-
COPY api/setup.py api/setup.py
17-
COPY api/pypi-readme.rst api/pypi-readme.rst
18-
COPY api/src/opentrons api/src/opentrons
14+
COPY api api
1915

20-
COPY robot-server/setup.py robot-server/setup.py
21-
COPY robot-server/robot_server robot-server/robot_server
16+
COPY robot-server robot-server
2217

23-
RUN cd shared-data/python && python3 setup.py bdist_wheel -d /dist/
24-
RUN cd server-utils && python3 setup.py bdist_wheel -d /dist/
25-
RUN cd api && python3 setup.py bdist_wheel -d /dist/
26-
RUN cd robot-server && python3 setup.py bdist_wheel -d /dist/
18+
RUN cd shared-data && python3 -m build --outdir=/dist/ --wheel .
19+
RUN cd server-utils && python3 -m build --outdir=/dist/ --wheel .
20+
RUN cd api && python3 -m build --outdir=/dist/ --wheel .
21+
RUN cd robot-server && python3 -m build --outdir=/dist/ --wheel .
2722

2823
FROM base
2924
COPY --from=builder /dist /dist

‎Makefile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -25,7 +25,7 @@ HARDWARE_DIR := hardware
2525
USB_BRIDGE_DIR := usb-bridge
2626
NODE_USB_BRIDGE_CLIENT_DIR := usb-bridge/node-client
2727

28-
PYTHON_DIRS := $(API_DIR) $(UPDATE_SERVER_DIR) $(ROBOT_SERVER_DIR) $(SERVER_UTILS_DIR) $(SHARED_DATA_DIR)/python $(G_CODE_TESTING_DIR) $(HARDWARE_DIR) $(USB_BRIDGE_DIR)
28+
PYTHON_DIRS := $(API_DIR) $(UPDATE_SERVER_DIR) $(ROBOT_SERVER_DIR) $(SERVER_UTILS_DIR) $(SHARED_DATA_DIR) $(G_CODE_TESTING_DIR) $(HARDWARE_DIR) $(USB_BRIDGE_DIR)
2929

3030
# This may be set as an environment variable (and is by CI tasks that upload
3131
# to test pypi) to add a .dev extension to the python package versions. If

‎abr-testing/.gitignore

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
*.py[cod]
2+
*.egg
3+
build
4+
htmlcov
5+
6+
abr_testing/_version.py

‎abr-testing/Makefile

Lines changed: 12 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -3,14 +3,8 @@ include ../scripts/python.mk
33

44
SHX := npx shx
55

6-
ot_project := $(OPENTRONS_PROJECT)
7-
project_rs_default = $(if $(ot_project),$(ot_project),robot-stack)
8-
project_ir_default = $(if $(ot_project),$(ot_project),ot3)
9-
106
package_name = abr_testing
117
package_version = $(call python_package_version,abr-testing,$(project_rs_default))
12-
wheel_file = dist/$(call python_get_wheelname,abr-testing,$(project_rs_default),$(package_name),$(BUILD_NUMBER))
13-
sdist_file = dist/$(call python_get_sdistname,abr-testing,$(project_rs_default),$(package_name))
148

159
# Find the branch, sha, version that will be used to update the VERSION.json file
1610
version_file = $(call python_get_git_version,abr-testing,$(project_rs_default),abr-testing)
@@ -34,10 +28,10 @@ ot_sources := $(ot_py_sources)
3428
# Defined separately than the clean target so the wheel file doesn’t have to
3529
# depend on a PHONY target
3630
clean_cmd = $(SHX) rm -rf build dist .coverage coverage.xml '*.egg-info' '**/__pycache__' '**/*.pyc' '**/.mypy_cache'
37-
31+
wheel_file = $(call python_get_wheelname,abr-testing,$(project_rs_default),abr_testing)
3832

3933
.PHONY: all
40-
all: clean sdist wheel
34+
all: clean wheel
4135

4236
.PHONY: setup
4337
setup:
@@ -53,38 +47,32 @@ clean:
5347
$(clean_cmd)
5448

5549
.PHONY: wheel
56-
wheel: export OPENTRONS_PROJECT=$(project_rs_default)
50+
wheel: export OPENTRONS_PROJECT=$(PROJECT)
51+
wheel: export HATCH_VCS_TUNABLE_TAG_PATTERN=$(call git_tag_regex_for_project,$(PROJECT))
52+
wheel: export HATCH_VCS_TUNABLE_RAW_OPTIONS=$(call hatch_raw_options_for_project,$(PROJECT))
5753
wheel:
5854
rm -rf dist/*.whl
59-
$(python) setup.py $(wheel_opts) bdist_wheel
60-
$(SHX) rm -rf build
61-
$(SHX) ls dist
62-
63-
.PHONY: sdist
64-
sdist: export OPENTRONS_PROJECT=$(project_rs_default)
65-
sdist:
66-
$(clean_cmd)
67-
$(python) setup.py sdist
55+
$(python) -m build --wheel .
6856
$(SHX) rm -rf build
6957
$(SHX) ls dist
7058

7159
.PHONY: lint
7260
lint:
7361
$(python) -m mypy abr_testing tests
74-
$(python) -m black --check abr_testing tests setup.py
75-
$(python) -m flake8 abr_testing tests setup.py
62+
$(python) -m black --check abr_testing tests
63+
$(python) -m flake8 abr_testing tests
7664

7765
.PHONY: format
7866
format:
79-
$(python) -m black abr_testing tests setup.py
67+
$(python) -m black abr_testing tests
8068

8169
.PHONY: test
8270
test:
8371
@echo "No tests yet"
8472

8573
.PHONY: push-no-restart-ot3
86-
push-no-restart-ot3: sdist Pipfile.lock
87-
$(call push-python-sdist,$(host),$(ssh_key),$(ssh_opts),$(sdist_file),/opt/opentrons-robot-server,"abr_testing",,,$(version_file))
74+
push-no-restart-ot3: wheel Pipfile.lock
75+
$(call push-python,$(host),$(ssh_key),$(ssh_opts),$(wheel_file),/opt/opentrons-robot-server/)
8876

8977
.PHONY: push-ot3
9078
push-ot3: push-no-restart-ot3
@@ -98,4 +86,4 @@ PROTOCOL_DIR := abr_testing/protocols
9886
SIMULATION_TOOL := abr_testing/protocol_simulation/abr_sim_check.py
9987
EXTENSION := .py
10088
simulate:
101-
$(python) $(SIMULATION_TOOL)
89+
$(python) $(SIMULATION_TOOL)

‎abr-testing/Pipfile

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ types-httplib2 = "*"
1111
oauth2client = "==4.1.3"
1212
gspread = "==6.0.2"
1313
hardware-testing = {editable = true, path = "../hardware-testing"}
14-
opentrons-shared-data = {editable = true, path = "./../shared-data/python"}
14+
opentrons-shared-data = {editable = true, path = "./../shared-data"}
1515
opentrons-hardware = {editable = true, path = "./../hardware", extras=['FLEX']}
1616
opentrons = {editable = true, path = "./../api", extras=['flex-hardware']}
1717
slackclient = "*"
@@ -36,6 +36,7 @@ flake8-noqa = "~=1.2.1"
3636
requests = "==2.27.1"
3737
types-requests = "==2.25.6"
3838
google-api-python-client-stubs = "*"
39+
build = "~=1.2.0"
3940

4041
[requires]
4142
python_version = "3.10"

0 commit comments

Comments
 (0)