Skip to content

Commit afb2f12

Browse files
arseniiarseniiPaTara43
arseniiarsenii
and
PaTara43
authored
Development (#23)
* longer wait time * speed up unit retrieval with serverside stage aggregation * added execution timing for frequently used db methods * implemented exec timer for async functions * increased units pending revision fetching speed 3x * access collections with properties * removed unused mount * Reworked unit upload and update for x3 speed increase. Removed unused methods. * Version bump * Add WS endpoint for workbench status streaming * Add optional dotenv parsing * Implement SSE stream instead of WS * Fixed type annotation * update black version * reworked connection close pipeline * improved logging * Implement background printing tasks * Update docker-compose.yml * remove .gitignore * mark endpoint as deprecated * optimize txn hash pushing to db * removed polling and optimized state push using async event * bugfix: thread wouldn't unblock until txn posted. implemented async wrapper * implemented single field updates * async short url generator. up to 4s faster passport saving * updated packages * version bump * cached files auto remove * added some docker volumes * added db connection close on quit * update .gitignore * removed duplicate code * removed redundant db field * refactored unit retrieval * implemented workbench shutdown sequence * renamed actions * implemented multiplatform Dockerfile * multiplatform image building * bugfix * bugfix * removed arm v7 form the list of platforms * added comments * updated robonomics-interface version * updated Dockerfiles to matched unified code style * updated Dockerfile names * updated action to reuse in other repos * update type hints to 3.10 format * minor changes * bugfix * dependency update * update base version * update Python version in action * bugfix * updated dependencies * action bugfix attempt * added type annotation * update mypy config * update dep versions in requirements.txt * return caching in action * return caching in action * moved type alias * added back union checking plugin * filter out AssertionErrors * replace variable returns with HTTPExceptions * add black daemon * minor refactoring * fixed typos * fixed typo * minor update to passport code logic * filter access logs for certain endpoints * add healthcheck * added connectivity checks for external services * fix typo * bugfix * Update _Barcode.py (#22) * getting component staged for revision from the parent's id (MAP-257) * rewritten pre-commit config * isort * improved type annotations * bugfix * bugfix MAP-332 * Added ECS logging support for the ELK stack integration * slightly modified README * ECS logs now written to file instead of console * experimental: cache Docker layers * improved logging * updated pc hooks * Implemented SSE notifications * Optimized imports * Allow assigning built units with unreleased passport (MAP-355) * Tagged endpoints * update deps * update deps * notify user of a forbidden transition * add error notifications * fix mypy error * replaced concat with fstring * extracted singular connection func * add fstring instead of concat * added more checks and error notifications * added more checks and error notifications * added datalog success notification * version bump * bugfix * add more notifications * replaced json err responses with HTTP exceptions * more notifications and minor improvements * bring back bg datalog posting * reworked messenger interface, suppressed Runtime Error (temp fix) * Fixed 'RuntimeError: no running event loop' * User notifications translated to russian * Improved error notifications * Minor improvements to Messenger.py * Improved reliability for unit wrap up operation * Improved reliability for operation ending * Minor fixes * Updated dependencies * Added more user notifications * Improved image annotations for passport QR codes * Add unit's combined assembly time to the passport * Faster operation ending on composite units * Attempt to fix QR code side logos * Reworked unit component assignment logic * Avoid reassigning the same unit twice * Remove exception info from the printing error notifications * add prometheus metrics Co-authored-by: Pavel Tarasov <34866682+PaTara43@users.noreply.github.com>
1 parent 87f0015 commit afb2f12

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

46 files changed

+3819
-664
lines changed

‎.github/workflows/Deploy-in-dev-k8s.yml‎

Lines changed: 0 additions & 47 deletions
This file was deleted.

‎.github/workflows/Deploy-in-master-k8s.yml‎

Lines changed: 0 additions & 47 deletions
This file was deleted.

.github/workflows/release.yml renamed to .github/workflows/add-release.yml

File renamed without changes.

‎.github/workflows/docker-image-build-and-push.yml‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
name: Docker image build and push
2+
3+
on:
4+
push:
5+
branches:
6+
- development
7+
- main
8+
pull_request:
9+
branches:
10+
- main
11+
12+
jobs:
13+
build-and-push-image:
14+
runs-on: ubuntu-latest
15+
steps:
16+
- name: Checkout
17+
uses: actions/checkout@v2
18+
19+
- name: Install doctl
20+
uses: digitalocean/action-doctl@v2.1.0
21+
with:
22+
token: ${{ secrets.DIGITALOCEAN_ACCESS_TOKEN }}
23+
24+
- name: Set up QEMU
25+
uses: docker/setup-qemu-action@v1
26+
27+
- name: Set up Docker Buildx
28+
id: buildx
29+
uses: docker/setup-buildx-action@v2
30+
31+
- name: Login to Digital Ocean container registry
32+
run: doctl registry login
33+
34+
- name: Generate Docker meta
35+
id: generate_meta
36+
uses: docker/metadata-action@v3
37+
with:
38+
images: ${{ secrets.DOREGISTRY }}/${{ github.event.repository.name }}
39+
tags: |
40+
type=schedule
41+
type=ref,event=branch
42+
type=ref,event=pr
43+
type=semver,pattern={{version}}
44+
type=semver,pattern={{major}}.{{minor}}
45+
type=semver,pattern={{major}}
46+
type=sha
47+
48+
- name: Cache Docker layers
49+
uses: actions/cache@v3
50+
with:
51+
path: /tmp/.buildx-cache
52+
key: ${{ runner.os }}-buildx-${{ github.event.repository.name }}
53+
restore-keys: |
54+
${{ runner.os }}-buildx-
55+
56+
- name: Build and push
57+
uses: docker/build-push-action@v2
58+
with:
59+
context: .
60+
platforms: linux/amd64,linux/arm64
61+
push: ${{ github.event_name != 'pull_request' }}
62+
tags: ${{ steps.generate_meta.outputs.tags }}
63+
labels: ${{ steps.generate_meta.outputs.labels }}
64+
file: Dockerfile
65+
cache-from: type=local,src=/tmp/.buildx-cache
66+
cache-to: type=local,dest=/tmp/.buildx-cache-new,mode=max
67+
68+
- # Temp fix
69+
# https://github.com/docker/build-push-action/issues/252
70+
# https://github.com/moby/buildkit/issues/1896
71+
name: Move cache
72+
run: |
73+
rm -rf /tmp/.buildx-cache
74+
mv /tmp/.buildx-cache-new /tmp/.buildx-cache

.github/workflows/python-app.yml renamed to .github/workflows/python-app-checks.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ jobs:
1414
- name: Set up python
1515
uses: actions/setup-python@v2
1616
with:
17-
python-version: 3.9
17+
python-version: "3.10"
1818
#----------------------------------------------
1919
# ----- install & configure poetry -----
2020
#----------------------------------------------

‎.pre-commit-config.yaml‎

Lines changed: 68 additions & 51 deletions
Original file line numberDiff line numberDiff line change
@@ -1,67 +1,84 @@
1-
exclude: ".venv|test*|.git"
2-
default_stages: [commit]
3-
fail_fast: false
1+
default_language_version:
2+
python: python3.10
43

54
repos:
6-
- repo: local
5+
- repo: https://github.com/pre-commit/mirrors-isort
6+
rev: v5.10.1
77
hooks:
8-
- id: black-check
9-
name: black-check
10-
entry: black .
11-
language: system
12-
pass_filenames: false
13-
always_run: true
8+
- id: isort
9+
description: Sorts import statements
10+
args:
11+
- --profile=black
12+
- --line-length=120
1413

15-
- repo: local
14+
- repo: https://github.com/myint/eradicate
15+
rev: v2.1.0
1616
hooks:
17-
- id: mypy-check
18-
name: mypy-check
19-
entry: mypy .
20-
language: system
21-
pass_filenames: false
22-
always_run: true
17+
- id: eradicate
18+
description: Eradicate removes unused commented code
19+
args:
20+
- --in-place
2321

24-
- repo: https://gitlab.com/pycqa/flake8
25-
rev: "3.9.2"
22+
- repo: https://github.com/asottile/pyupgrade
23+
rev: v2.34.0
2624
hooks:
27-
- id: flake8
28-
name: flake8-check
29-
description: "flake8 is a command-line utility for enforcing style consistency across Python projects."
30-
entry: flake8
31-
language: python
32-
types: [python]
33-
require_serial: true
34-
args: ["--config=.flake8"]
25+
- id: pyupgrade
26+
name: pyupgrade
27+
description: Pyupgrade replaces deprecated syntax with the newest syntax
28+
args:
29+
- --py310-plus
3530

36-
- repo: local
31+
- repo: https://github.com/psf/black
32+
rev: 22.6.0
3733
hooks:
38-
- id: pytest-check
39-
name: pytest-check
40-
entry: make test
41-
language: system
42-
pass_filenames: false
43-
always_run: true
34+
- id: black
35+
name: black
36+
description: Black is an automatic Python code formatter. It ensures code readability and PEP8 compliance.
37+
args:
38+
- --line-length=120
4439

45-
- repo: https://github.com/myint/eradicate
46-
rev: v2.0.0
40+
- repo: https://github.com/pre-commit/mirrors-mypy
41+
rev: v0.961
4742
hooks:
48-
- id: eradicate
49-
language_version: python3.9
50-
entry: eradicate
51-
files: '\.py$'
52-
args: [ --in-place ]
43+
- id: mypy
44+
name: mypy
45+
additional_dependencies:
46+
- types-PyYAML
47+
- types-requests
48+
- pydantic
49+
args:
50+
- --strict
51+
- --ignore-missing-imports
52+
- --allow-untyped-decorators
53+
- --implicit-reexport
54+
55+
- repo: https://gitlab.com/pycqa/flake8
56+
rev: 3.9.2
57+
hooks:
58+
- id: flake8
59+
name: flake8
60+
description: Flake8 is a command-line utility for enforcing style consistency across Python projects.
61+
additional_dependencies:
62+
- flake8-pep585
63+
- flake8-new-union-types
64+
args:
65+
- --count
66+
- --max-line-length=130
5367

5468
- repo: https://github.com/pre-commit/pre-commit-hooks
55-
rev: v4.0.1
69+
rev: v4.3.0
5670
hooks:
71+
- id: check-added-large-files
72+
description: Ensures no big files are commited
73+
- id: check-json
74+
description: Ensures valid JSON
75+
- id: check-yaml
76+
description: Ensures valid YAML
5777
- id: check-case-conflict
78+
description: Ensures there are no name conflicts in case insesnsitive systems
5879
- id: check-merge-conflict
59-
- id: check-yaml
60-
- id: check-added-large-files
61-
args: ['--maxkb=500']
62-
63-
- repo: https://github.com/Lucas-C/pre-commit-hooks-safety
64-
rev: v1.2.1
65-
hooks:
66-
- id: python-safety-dependencies-check
67-
args: [--json, -i 38678]
80+
description: Ensures there are no merge conflict lines
81+
- id: detect-private-key
82+
description: Detects commited private keys
83+
- id: pretty-format-json
84+
description: Prettifies JSON

‎Dockerfile‎

Lines changed: 37 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -1,17 +1,39 @@
1-
# Commented out to save time on dependency resolving
2-
# Uncomment lines to go full cycle instead of using precompiled requirements.txt
1+
# There are no binaries of this packages as of today for ARM platforms,
2+
# so compilation is required
3+
FROM python:3.10 as dependency-compilation
4+
WORKDIR /tmp
5+
RUN apt-get update && apt-get install -y\
6+
build-essential\
7+
libssl-dev\
8+
libffi-dev\
9+
python3-dev
10+
RUN curl https://sh.rustup.rs -sSf | sh -s -- -y
11+
RUN /root/.cargo/bin/rustup default nightly
12+
RUN pip install --upgrade pip
13+
RUN . /root/.cargo/env && pip install py-bip39-bindings
14+
RUN . /root/.cargo/env && pip install py-ed25519-bindings
15+
RUN . /root/.cargo/env && pip install py-sr25519-bindings
16+
RUN . /root/.cargo/env && pip install cryptography
317

4-
#FROM python:3.9 as requirements-stage
5-
#WORKDIR /tmp
6-
#RUN pip install poetry
7-
#COPY ./pyproject.toml ./poetry.lock* /tmp/
8-
#RUN poetry export -f requirements.txt --output requirements.txt --without-hashes
18+
# At this stage we convert Poetry's dependency file into a more traditional
19+
# requirements.txt to avoid installing Poetry into the final container.
20+
# Although very slow, this cannot be skipped, as we need to resolve dependencies
21+
# for the exact platform the client is using.
22+
FROM python:3.10 as requirements-stage
23+
WORKDIR /tmp
24+
COPY --from=dependency-compilation /root/.cache/pip /root/.cache/pip
25+
RUN pip install poetry
26+
COPY ./pyproject.toml ./poetry.lock* /tmp/
27+
RUN poetry export -f requirements.txt --output requirements.txt --without-hashes
928

10-
FROM python:3.9
11-
WORKDIR /code
12-
#COPY --from=requirements-stage /tmp/requirements.txt /code/requirements.txt
13-
COPY requirements.txt /code/requirements.txt
14-
RUN pip install --no-cache-dir --upgrade -r /code/requirements.txt
15-
COPY ./src /code
16-
WORKDIR /code
17-
CMD ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "5000", "--no-access-log"]
29+
# Final container build. Uses pre-compiled dependencies and requirements.txt
30+
# obtained in the previous steps
31+
FROM python:3.10
32+
WORKDIR /src
33+
COPY --from=requirements-stage /tmp/requirements.txt /src/requirements.txt
34+
COPY --from=dependency-compilation /root/.cache/pip /root/.cache/pip
35+
RUN pip install --no-cache-dir --upgrade -r /src/requirements.txt
36+
COPY ./src /src
37+
HEALTHCHECK --interval=5s --timeout=3s --start-period=5s --retries=12 \
38+
CMD curl --fail http://localhost:5000/docs || exit 1
39+
ENTRYPOINT ["uvicorn", "app:app", "--host", "0.0.0.0", "--port", "5000"]

0 commit comments

Comments
 (0)