[ISV-3309] Add dependency manager. (#454)

* [ISV-3309] Add dependency manager.

* [ISV-3309] Add dependency manager.

* [ISV-3309] Add dependency manager.

* [ISV-3309] Add dependency manager.

* [ISV-3309] Add dependency manager.

* [ISV-3309] Add dependency manager.

* [ISV-3309] Add dependency manager.

* [ISV-3309] Add dependency manager.

* [ISV-3309] Add dependency manager.

* [ISV-3309] Add dependency manager.

* [ISV-3309] Add dependency manager.

* [ISV-3309] Add dependency manager.

* [ISV-3309] Add dependency manager.

* [ISV-3309] Add dependency manager.

* [ISV-3309] Add dependency manager.

* [ISV-3309] Add dependency manager.

* [ISV-3309] Add dependency manager.

* Correct the typo and add pdm installation in the dockerfile.

* [ISV-3309] Add dependency manager.

* Correct the typo and add pdm installation in the dockerfile.

* Correct the typo and add pdm installation in the dockerfile.

* Update the pdm installation in dockerfile.

* Update the pdm installation in dockerfile.

* Update the pdm installation in dockerfile.

* Update the pdm installation in dockerfile.

* Fix the tests.

* Fix the tests.

* Fix the tests.

* Fix the tests.

* Fix the tests.

* Fix the tests.

---------

Co-authored-by: haripate <>

Author: haripate

.github/workflows/validation.yml

@@ -13,8 +13,19 @@ jobs:
     name: Run unit tests and linters
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
-      - uses: fedora-python/[email protected]
+      - name: Checkout repository
+        uses: actions/checkout@v3
+
+      - name: Set up PDM
+        uses: pdm-project/setup-pdm@v3
         with:
-          tox_env: black,test,yamllint
-          dnf_install: krb5-devel krb5-workstation
+          python-version: "3.10"
+
+      - name: Install dependencies
+        run: |
+          pdm sync -dG operator-pipelines-dev
+          pdm sync -dG tox
+
+      - name: Run Tests
+        run: |
+          pdm run -v tox

.gitignore

@@ -146,3 +146,4 @@ dmypy.json
 .*.sw?
 ansible/vault-password*
 
+.pdm-python

docs/developer-guide.md

@@ -266,16 +266,26 @@ oc apply -R -f ansible/roles/operator-pipeline/templates/openshift
 
 ## Making Changes to the Pipeline Image
 
-### Setup
-To install the python package in a development environment, run:
+### Dependency manager
+Operator pipelines project is configured to automatically manage Python dependencies using [PDM][1] tool.
+The pdm automates definition, installation, upgrades and the whole lifecycle of
+dependency in a project. All dependencies are stored in `pyproject.toml` file in a groups
+that corresponds to individual applications within the Operator pipelines project.
 
+Adding, removing and updating of dependency needs to be always done using `pdm` cli.
 ```bash
-pip install ".[dev]"
+pdm add -G operator-pipelines gunicorn==20.1.0
 ```
+After a dependency is installed it is added to pdm.lock file. The lock file
+is always part of git repository.
 
-### Tips
+If you want to install specific group set of dependencies use following command:
+```bash
+pdm install -G operator-pipelines
+```
+Dependencies are stored into virtual environment (.venv) which is automatically created after `pdm install`
+If .venv wasn't created, configure pdm to automatically create it during installation with `pdm config python.use_venv true`.
 
-- If adding a new script in the pipeline image - don't forget to add the entrypoint to setup.py
 
 ### Run Unit Tests, Code Style Checkers, etc.
 
@@ -285,17 +295,26 @@ To run unit tests and code style checkers:
 tox
 ```
 
+### Local development
+
+Setup python virtual environment using pdm.
+```shell
+$ pdm venv create 3.10
+$ pdm install
+$ source .venv/bin/activate
+```
+
 ### Build & Push
 
 1. Ensure you have [buildah](https://github.com/containers/buildah/blob/main/install.md) installed
 
-1. Build the image
+2. Build the image
 
     ```bash
     buildah bud
     ```
 
-1. Push the image to a remote registry, eg. Quay.io.
+3. Push the image to a remote registry, eg. Quay.io.
 
     ```bash
     buildah push <image-digest-from-build-step> <remote-repository>
@@ -306,3 +325,5 @@ tox
     ```bash
     buildah login quay.io
     ```
+
+[1]: https://pdm.fming.dev/latest/

operator-pipeline-images/Dockerfile

@@ -61,6 +61,18 @@ WORKDIR /home/user
 
 COPY ./operator-pipeline-images ./
 
+# install PDM
+RUN pip3 install --upgrade pip setuptools wheel && pip3 install pdm
+# install dependencies in virtual environment
+COPY ./pdm.lock ./pyproject.toml ./README.md ./
+RUN pdm install --no-lock --no-editable \
+        --group operator-pipelines \
+        --group operator-pipelines-dev
+
+ENV VIRTUAL_ENV=./.venv
+RUN python3 -m venv $VIRTUAL_ENV
+ENV PATH="$VIRTUAL_ENV/bin:$PATH"
+
 RUN pip3 install .
 
 # set dir ownership