Merge branch 'master' into extend-locust-tests

Author: bisgaard-itis

.codecov.yml

@@ -31,6 +31,9 @@ component_management:
     - component_id: pkg_aws_library
       paths:
         - packages/aws-library/**
+    - component_id: pkg_celery_library
+      paths:
+        - packages/celery-library/**
     - component_id: pkg_dask_task_models_library
       paths:
         - packages/dask-task-models-library/**

.github/CODEOWNERS

@@ -13,6 +13,7 @@ Makefile                                           @pcrespov @sanderegg
 /api/                                              @sanderegg @pcrespov @matusdrobuliak66
 /ci/                                               @sanderegg @pcrespov
 /docs/                                             @pcrespov
+/packages/celery-library/                          @giancarloromeo
 /packages/common-library/                          @giancarloromeo
 /packages/models-library/                          @sanderegg @pcrespov @matusdrobuliak66 @giancarloromeo
 /packages/postgres-database/                       @matusdrobuliak66

.github/tips.md renamed to .github/README.md

@@ -7,3 +7,15 @@ WARNING: do not name this file as README.md since it will be rendered in the mai
 - ``PULL_REQUEST_TEMPLATE.md``
 - ``ISSUE_TEMPLATE`` folder:
   - Add [top-level syntax](https://docs.github.com/en/communities/using-templates-to-encourage-useful-issues-and-pull-requests/syntax-for-issue-forms#top-level-syntax) to ISSUE_TEMPLATE/*.md files to configure them and view them as [template in the github web](https://github.com/ITISFoundation/osparc-simcore/issues/new/choose)
+
+
+---
+
+## Copilot Usage Tips
+
+1. **Be Specific**: Provide clear and detailed prompts to Copilot for better suggestions.
+2. **Iterate**: Review and refine Copilot's suggestions to ensure they meet project standards.
+3. **Split Tasks**: Break down complex tasks into smaller, manageable parts for better suggestions.
+4. **Test Suggestions**: Always test Copilot-generated code to ensure it works as expected.
+
+-  SEE https://code.visualstudio.com/docs/copilot/copilot-customization#_custom-instructions

.github/codeql/codeql-config.yml

@@ -4,6 +4,7 @@ disable-default-queries: false
 
 paths:
   - packages/aws-library/src
+  - packages/celery-library/src
   - packages/dask-task-models-library/src
   - packages/models-library/src/models_library
   - packages/postgres-database/src/simcore_postgres_database

.github/copilot-instructions.md

@@ -4,52 +4,58 @@ This document provides guidelines and best practices for using GitHub Copilot in
 
 ## General Guidelines
 
-1. **Use Python 3.11**: Ensure that all Python-related suggestions align with Python 3.11 features and syntax.
-2. **Node.js Compatibility**: For Node.js projects, ensure compatibility with the version specified in the project (e.g., Node.js 14 or later).
-3. **Follow Coding Conventions**: Adhere to the coding conventions outlined in the `docs/coding-conventions.md` file.
-4. **Test-Driven Development**: Write unit tests for all new functions and features. Use `pytest` for Python and appropriate testing frameworks for Node.js.
-5. **Environment Variables**: Use environment variables as specified in `docs/env-vars.md` for configuration. Avoid hardcoding sensitive information.
-6. **Documentation**: Prefer self-explanatory code; add documentation only if explicitly requested by the developer.
+1. **Test-Driven Development**: Write unit tests for all new functions and features. Use `pytest` for Python and appropriate testing frameworks for Node.js.
+2. **Environment Variables**: Use [Environment Variables Guide](../docs/env-vars.md) for configuration. Avoid hardcoding sensitive information.
+3. **Documentation**: Prefer self-explanatory code; add documentation only if explicitly requested by the developer.
 
-## Python-Specific Instructions
+---
 
-- Always use type hints and annotations to improve code clarity and compatibility with tools like `mypy`.
-  - An exception to that rule is in `test_*` functions return type hint must not be added
-- Follow the dependency management practices outlined in `requirements/`.
-- Use `ruff` for code formatting and for linting.
-- Use `black` for code formatting and `pylint` for linting.
-- ensure we use `sqlalchemy` >2 compatible code.
-- ensure we use `pydantic` >2 compatible code.
-- ensure we use `fastapi` >0.100 compatible code
-- use f-string formatting
-- Only add comments in function if strictly necessary
-- use relative imports
-- imports should be at top of the file
+## 🛠️Coding Instructions for Python in This Repository
 
+Follow these rules strictly when generating Python code:
 
-### Json serialization
+### 1. Python Version
 
-- Generally use `json_dumps`/`json_loads` from `common_library.json_serialization` to built-in `json.dumps` / `json.loads`.
-- Prefer Pydantic model methods (e.g., `model.model_dump_json()`) for serialization.
+* Use Python 3.11: Ensure all code uses features and syntax compatible with Python 3.11.
 
+### 2. **Type Annotations**
 
-## Node.js-Specific Instructions
+* Always use full type annotations for all functions and class attributes.
+* ❗ **Exception**: Do **not** add return type annotations in `test_*` functions.
 
-- Use ES6+ syntax and features.
-- Follow the `package.json` configuration for dependencies and scripts.
-- Use `eslint` for linting and `prettier` for code formatting.
-- Write modular and reusable code, adhering to the project's structure.
+### 3. **Code Style & Formatting**
 
-## Copilot Usage Tips
+* Follow [Python Coding Conventions](../docs/coding-conventions.md) **strictly**.
+* Format code with `black`.
+* Lint code with `ruff` and `pylint`.
 
-1. **Be Specific**: Provide clear and detailed prompts to Copilot for better suggestions.
-2. **Iterate**: Review and refine Copilot's suggestions to ensure they meet project standards.
-3. **Split Tasks**: Break down complex tasks into smaller, manageable parts for better suggestions.
-4. **Test Suggestions**: Always test Copilot-generated code to ensure it works as expected.
+### 4. **Library Compatibility**
 
-## Additional Resources
+Ensure compatibility with the following library versions:
 
-- [Python Coding Conventions](../docs/coding-conventions.md)
-- [Environment Variables Guide](../docs/env-vars.md)
-- [Pydantic Annotated fields](../docs/llm-prompts/pydantic-annotated-fields.md)
-- [Steps to Upgrade Python](../docs/steps-to-upgrade-python.md)
+* `sqlalchemy` ≥ 2.x
+* `pydantic` ≥ 2.x
+* `fastapi` ≥ 0.100
+
+
+### 5. **Code Practices**
+
+* Use `f-string` formatting for all string interpolation except for logging message strings.
+* Use **relative imports** within the same package/module.
+* Place **all imports at the top** of the file.
+* Add comments **only when the code is not self-explanatory**.
+
+
+### 6. **JSON Serialization**
+
+* Prefer `json_dumps` / `json_loads` from `common_library.json_serialization` instead of the built-in `json.dumps` / `json.loads`.
+* When using Pydantic models, prefer methods like `model.model_dump_json()` for serialization.
+
+---
+
+## 🛠️Coding Instructions for Node.js in This Repository
+
+* Use ES6+ syntax and features.
+* Follow the `package.json` configuration for dependencies and scripts.
+* Use `eslint` for linting and `prettier` for code formatting.
+* Write modular and reusable code, adhering to the project's structure.

docs/llm-prompts/pydantic-annotated-fields.md renamed to .github/prompts/pydantic-annotated-fields.prompt.md

@@ -1,6 +1,9 @@
-# Prompt
+---
+mode: 'edit'
+description: 'Convert Pydantic model fields to use Annotated pattern'
+---
+
 
-```
 Please convert all pydantic model fields that use `Field()` with default values to use the Annotated pattern instead.
 Follow these guidelines:
 
@@ -10,7 +13,8 @@ Follow these guidelines:
 4. Add the import: `from common_library.basic_types import DEFAULT_FACTORY` if it's not already present.
 5. If `Field()` has no parameters (empty), don't use Annotated at all. Just use: `field_name: field_type = default_value`.
 6. Leave any model validations, `model_config` settings, and `field_validators` untouched.
-```
+
+
 ## Examples
 
 ### Before:
@@ -53,13 +57,11 @@ class ProjectModel(BaseModel):
     id: str = Field(default_factory=uuid.uuid4, description="Unique project identifier")
     name: str = Field(default="Untitled Project", min_length=3, max_length=50)
     created_at: datetime = Field(default_factory=datetime.now)
+    value: int = Field(..., description="Project value")
+    str_with_default: str = Field(default="foo")
+
     config: dict = Field(default={"version": "1.0", "theme": "default"})
 
-    @field_validator("name")
-    def validate_name(cls, v):
-        if v.isdigit():
-            raise ValueError("Name cannot be only digits")
-        return v
 ```
 
 ### After:
@@ -74,11 +76,9 @@ class ProjectModel(BaseModel):
     id: Annotated[str, Field(default_factory=uuid.uuid4, description="Unique project identifier")] = DEFAULT_FACTORY
     name: Annotated[str, Field(min_length=3, max_length=50)] = "Untitled Project"
     created_at: Annotated[datetime, Field(default_factory=datetime.now)] = DEFAULT_FACTORY
+    value: Annotated[int, Field(description="Project value")]
+    str_with_default: str = "foo"
+
     config: dict = {"version": "1.0", "theme": "default"}
 
-    @field_validator("name")
-    def validate_name(cls, v):
-        if v.isdigit():
-            raise ValueError("Name cannot be only digits")
-        return v
 ```

.github/workflows/ci-testing-deploy.yml

@@ -47,6 +47,7 @@ jobs:
     # Set job outputs to values from filter step
     outputs:
       aws-library: ${{ steps.filter.outputs.aws-library }}
+      celery-library: ${{ steps.filter.outputs.celery-library }}
       dask-task-models-library: ${{ steps.filter.outputs.dask-task-models-library }}
       models-library: ${{ steps.filter.outputs.models-library }}
       common-library: ${{ steps.filter.outputs.common-library }}
@@ -94,6 +95,12 @@ jobs:
               - 'services/docker-compose*'
               - 'scripts/mypy/*'
               - 'mypy.ini'
+            celery-library:
+              - 'packages/celery-library/**'
+              - 'packages/pytest-simcore/**'
+              - 'services/docker-compose*'
+              - 'scripts/mypy/*'
+              - 'mypy.ini'
             dask-task-models-library:
               - 'packages/dask-task-models-library/**'
               - 'packages/pytest-simcore/**'
@@ -1035,6 +1042,49 @@ jobs:
         with:
           flags: unittests #optional
 
+  unit-test-celery-library:
+      needs: changes
+      if: ${{ needs.changes.outputs.celery-library == 'true' || github.event_name == 'push' || github.event.inputs.force_all_builds == 'true' }}
+      timeout-minutes: 18 # if this timeout gets too small, then split the tests
+      name: "[unit] celery-library"
+      runs-on: ${{ matrix.os }}
+      strategy:
+        matrix:
+          python: ["3.11"]
+          os: [ubuntu-24.04]
+        fail-fast: false
+      steps:
+        - uses: actions/checkout@v4
+        - name: setup docker buildx
+          id: buildx
+          uses: docker/setup-buildx-action@v3
+          with:
+            driver: docker-container
+        - name: setup python environment
+          uses: actions/setup-python@v5
+          with:
+            python-version: ${{ matrix.python }}
+        - name: install uv
+          uses: astral-sh/setup-uv@v6
+          with:
+            version: "0.6.x"
+            enable-cache: false
+            cache-dependency-glob: "**/celery-library/requirements/ci.txt"
+        - name: show system version
+          run: ./ci/helpers/show_system_versions.bash
+        - name: install
+          run: ./ci/github/unit-testing/celery-library.bash install
+        - name: typecheck
+          run: ./ci/github/unit-testing/celery-library.bash typecheck
+        - name: test
+          if: ${{ !cancelled() }}
+          run: ./ci/github/unit-testing/celery-library.bash test
+        - uses: codecov/codecov-action@v5
+          if: ${{ !cancelled() }}
+          env:
+            CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
+          with:
+            flags: unittests #optional
 
   unit-test-dask-task-models-library:
     needs: changes
@@ -1824,6 +1874,7 @@ jobs:
         unit-test-clusters-keeper,
         unit-test-dask-sidecar,
         unit-test-aws-library,
+        unit-test-celery-library,
         unit-test-dask-task-models-library,
         unit-test-datcore-adapter,
         unit-test-director-v2,

.vscode/launch.template.json

@@ -44,6 +44,28 @@
       "justMyCode": false
     },
     {
+      // This test adds --external-envfile and expects a file named ".secrets" in the workspace root.
+      "name": "Python: Test w/ repo.config",
+      "type": "debugpy",
+      "request": "launch",
+      "module": "pytest",
+      "args": [
+        "--ff",
+        "--log-cli-level=INFO",
+        "--external-envfile=${workspaceFolder}/.secrets",
+        "--pdb",
+        "--setup-show",
+        "--durations=5",
+        "-s",
+        "-vv",
+        "${file}"
+      ],
+      "cwd": "${workspaceFolder}",
+      "console": "integratedTerminal",
+      "justMyCode": false
+    },
+    {
+      // This tests enables the httpx spy and dumps captures in a json. Mainly for api-server
       "name": "Python: Test-Httpx-Spy",
       "type": "debugpy",
       "request": "launch",

.vscode/settings.template.json

@@ -33,6 +33,7 @@
   "python.analysis.typeCheckingMode": "basic",
   "python.analysis.extraPaths": [
     "./packages/aws-library/src",
+    "./packages/celery-library/src",
     "./packages/common-library/src",
     "./packages/dask-task-models-library/src",
     "./packages/models-library/src",

ci/github/unit-testing/celery-library.bash

@@ -0,0 +1,43 @@
+#!/bin/bash
+# http://redsymbol.net/articles/unofficial-bash-strict-mode/
+set -o errexit  # abort on nonzero exitstatus
+set -o nounset  # abort on unbound variable
+set -o pipefail # don't hide errors within pipes
+IFS=$'\n\t'
+
+install() {
+  make devenv
+  # shellcheck source=/dev/null
+  source .venv/bin/activate
+  pushd packages/celery-library
+  make install-ci
+  popd
+  uv pip list
+}
+
+test() {
+  # shellcheck source=/dev/null
+  source .venv/bin/activate
+  pushd packages/celery-library
+  make tests-ci
+  popd
+}
+
+typecheck() {
+  # shellcheck source=/dev/null
+  source .venv/bin/activate
+  uv pip install mypy
+  pushd packages/celery-library
+  make mypy
+  popd
+}
+
+# Check if the function exists (bash specific)
+if declare -f "$1" >/dev/null; then
+  # call arguments verbatim
+  "$@"
+else
+  # Show a helpful error
+  echo "'$1' is not a known function name" >&2
+  exit 1
+fi