Merge branch 'master' of github.com:scalableminds/webknossos-libs

Author: hotzenklotz

.codespellrc

@@ -0,0 +1,6 @@
+[codespell]
+# Ref: https://github.com/codespell-project/codespell#using-a-config-file
+skip = *.lock,./webknossos/tests/cassettes,./webknossos/tests/dataset/cassettes,./webknossos/webknossos/dataset/_utils/vendor
+# some names and camelCased variables etc
+ignore-regex = \b([a-z]+[A-Z][a-zA-Z]*)\b
+ignore-words-list = nd,ND

.github/workflows/ci.yml

@@ -84,15 +84,22 @@ jobs:
           ./kind create cluster --config=tests/cluster-config.yaml
           ./kind export kubeconfig
 
+          cp ../requirements.txt .
           docker build -f tests/Dockerfile -t scalableminds/cluster-tools:latest .
           ./kind load docker-image scalableminds/cluster-tools:latest
 
       - name: Install dependencies (without docker)
-        if: ${{ matrix.executors != 'slurm' }}
+        if: ${{ matrix.executors == 'multiprocessing' }}
         run: |
           pip install -r ../requirements.txt
           poetry install
 
+      - name: Install dependencies (without docker)
+        if: ${{ matrix.executors == 'kubernetes' || matrix.executors == 'dask' }}
+        run: |
+          pip install -r ../requirements.txt
+          poetry install --all-extras
+
       - name: Check typing
         if: ${{ matrix.executors == 'multiprocessing' && matrix.python-version == '3.11' }}
         run: ./typecheck.sh
@@ -163,7 +170,7 @@ jobs:
       - name: Install dependencies
         run: |
           pip install -r ../requirements.txt
-          poetry install --extras all --with examples
+          poetry install --extras all --with examples --with dev
 
       - name: Check formatting
         if: ${{ matrix.group == 1 && matrix.python-version == '3.11' }}

.github/workflows/nightly.yml

@@ -29,7 +29,7 @@ jobs:
     - name: Install dependencies
       run: |
         pip install poetry
-        poetry install --extras all
+        poetry install --extras all --with examples --with dev
 
     - name: Check if git is dirty
       run: |

CONTRIBUTING.md

@@ -115,8 +115,8 @@ See below for specifics of the different packages. Let's have a look at the comm
   To install the dependencies for all sub-projects, run `make install`.
 
 * **Tooling** we use across the sub-projects to enforce coding styles and tests:
-    * `./format.sh`: black and isort
-    * `./lint.sh`: pylint
+    * `./format.sh`: ruff
+    * `./lint.sh`: ruff
     * `./typecheck.sh`: mypy
     * `./test.sh`: pytest and custom scripts
 
@@ -147,15 +147,6 @@ The tests also contain functionality for the WEBKNOSSOS client. There a two mode
 [`pytest-split`](https://jerry-git.github.io/pytest-split),
 which is used in the CI to split the tests for different runners.
 
-The code under `webknossos/client/_generated` is auto-generated! Please don't adapt anything in the `generated` folder manually, but re-run the code generation.
-
-The client code is generated using [openapi-python-client](https://github.com/openapi-generators/openapi-python-client) and [InducOapi](https://github.com/TheWall89/inducoapi).
-
-To re-generate the code, run
-```bash
-./generate_client.sh
-```
-
 #### `cluster_tools` package
 
 For testing the `slurm` setup a docker-compose setup is available. Please see the [respective Readme](https://github.com/scalableminds/webknossos-libs/blob/master/cluster_tools/README.md) for details.

README.md

@@ -3,7 +3,7 @@
 [![Supported Python Versions](https://img.shields.io/pypi/pyversions/webknossos.svg)](https://pypi.python.org/pypi/webknossos)
 [![Build Status](https://img.shields.io/github/actions/workflow/status/scalableminds/webknossos-libs/.github/workflows/ci.yml?branch=master)](https://github.com/scalableminds/webknossos-libs/actions?query=workflow%3A%22CI%22)
 [![Documentation](https://img.shields.io/badge/docs-passing-brightgreen.svg)](https://docs.webknossos.org/webknossos-py/index.html)
-[![Code Style](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+[![Code Style](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://docs.astral.sh/ruff/)
 
 <img align="right" src="https://static.webknossos.org/logos/webknossos-icon-only.svg" alt="WEBKNOSSOS Logo" width="100" height="100"/>
 

cluster_tools/Changelog.md

@@ -7,7 +7,7 @@ and this project adheres to [Semantic Versioning](http://semver.org/) `MAJOR.MIN
 For upgrade instructions, please check the respective *Breaking Changes* sections.
 
 ## Unreleased
-[Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.14.15...HEAD)
+[Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.14.22...HEAD)
 
 ### Breaking Changes
 
@@ -18,6 +18,44 @@ For upgrade instructions, please check the respective *Breaking Changes* section
 ### Fixed
 
 
+## [0.14.22](https://github.com/scalableminds/webknossos-libs/releases/tag/v0.14.22) - 2024-05-13
+[Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.14.21...v0.14.22)
+
+
+## [0.14.21](https://github.com/scalableminds/webknossos-libs/releases/tag/v0.14.21) - 2024-05-07
+[Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.14.20...v0.14.21)
+
+
+## [0.14.20](https://github.com/scalableminds/webknossos-libs/releases/tag/v0.14.20) - 2024-04-23
+[Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.14.19...v0.14.20)
+
+### Changed
+- Array jobs are spawned faster now, because common meta data is not serialized for each subjob. The performance improvement is especially big, when custom logging code is configured for array jobs. [#1042](https://github.com/scalableminds/webknossos-libs/pull/1042)
+- Updated ruff to v0.4.0 [1047](https://github.com/scalableminds/webknossos-libs/pull/1047)
+
+
+## [0.14.19](https://github.com/scalableminds/webknossos-libs/releases/tag/v0.14.19) - 2024-04-18
+[Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.14.18...v0.14.19)
+
+### Changed
+- Moved the dependencies for Kubernetes and Dask support into optional extras that need to be installed separately. Either with `poetry install --all-extras or pip install -e "all"`. [#1024](https://github.com/scalableminds/webknossos-libs/pull/1024)
+
+
+## [0.14.18](https://github.com/scalableminds/webknossos-libs/releases/tag/v0.14.18) - 2024-04-18
+[Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.14.17...v0.14.18)
+
+
+## [0.14.17](https://github.com/scalableminds/webknossos-libs/releases/tag/v0.14.17) - 2024-04-10
+[Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.14.16...v0.14.17)
+
+
+## [0.14.16](https://github.com/scalableminds/webknossos-libs/releases/tag/v0.14.16) - 2024-04-04
+[Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.14.15...v0.14.16)
+
+### Changed
+- Replaced black+isort formatters and pylint linter with ruff. [#1023](https://github.com/scalableminds/webknossos-libs/pull/1023)
+
+
 ## [0.14.15](https://github.com/scalableminds/webknossos-libs/releases/tag/v0.14.15) - 2024-02-07
 [Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.14.14...v0.14.15)
 

cluster_tools/README.md

@@ -1,8 +1,10 @@
 # Cluster Tools
 
 [![Build Status](https://img.shields.io/github/actions/workflow/status/scalableminds/webknossos-libs/.github/workflows/ci.yml?branch=master)](https://github.com/scalableminds/webknossos-libs/actions?query=workflow%3A%22CI%22)
+[![Code Style](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://docs.astral.sh/ruff/)
 
-This package provides python `Executor` classes for distributing tasks on a slurm cluster or via multi processing.
+
+This package provides python `Executor` classes for distributing tasks on a Slurm cluster, Kubernetes, Dask or via multi processing.
 
 ## Example
 
@@ -19,6 +21,23 @@ if __name__ == '__main__':
     assert result == [4, 9, 16]
 ```
 
+## Installation
+The `cluster_tools` package requires at least Python 3.8.
+
+You can install it from [pypi](https://pypi.org/project/cluster_tools/), e.g. via pip:
+
+```bash
+pip install cluster_tools
+```
+
+By default only the dependencies for running jobs on Slurm and via multiprocessing are installed. 
+For Kubernetes and Dask run:
+
+```bash
+pip install cluster_tools[kubernetes]
+pip install cluster_tools[dask]
+```
+
 ## Configuration
 
 ### Slurm
@@ -55,12 +74,15 @@ If you would like to configure these limits independently, you can do so by sett
 ## Dev Setup
 
 ```
+# See ./dockered-slurm/README.md for troubleshooting
 cd dockered-slurm
 docker-compose up -d
 docker exec -it slurmctld bash
 docker exec -it c1 bash
 ```
 
+Make sure to install all extra dependencies, such as Kubernetes, with `poetry install --all-extras`.
+
 Tests can be executed with `cd tests && poetry run pytest -s tests.py` after entering the container.
 Linting can be run with `./lint.sh`.
 Code formatting (black) can be run with `./format.sh`.

cluster_tools/cluster_tools/__init__.py

@@ -7,10 +7,10 @@
 from cluster_tools.executors.pickle_ import PickleExecutor
 from cluster_tools.executors.sequential import SequentialExecutor
 from cluster_tools.schedulers.cluster_executor import (
-    ClusterExecutor,
-    RemoteOutOfMemoryException,
-    RemoteResourceLimitException,
-    RemoteTimeLimitException,
+    ClusterExecutor,  # noqa:  F401 `cluster_tools.schedulers.cluster_executor.ClusterExecutor` imported but unused;
+    RemoteOutOfMemoryException,  # noqa:  F401 `cluster_tools.schedulers.cluster_executor.ClusterExecutor` imported but unused;
+    RemoteResourceLimitException,  # noqa:  F401 `cluster_tools.schedulers.cluster_executor.ClusterExecutor` imported but unused;
+    RemoteTimeLimitException,  # noqa:  F401 `cluster_tools.schedulers.cluster_executor.ClusterExecutor` imported but unused;
 )
 from cluster_tools.schedulers.kube import KubernetesExecutor
 from cluster_tools.schedulers.pbs import PBSExecutor
@@ -54,53 +54,45 @@ def _test_valid_multiprocessing() -> None:
 
 
 @overload
-def get_executor(environment: Literal["slurm"], **kwargs: Any) -> SlurmExecutor:
-    ...
+def get_executor(environment: Literal["slurm"], **kwargs: Any) -> SlurmExecutor: ...
 
 
 @overload
-def get_executor(environment: Literal["pbs"], **kwargs: Any) -> PBSExecutor:
-    ...
+def get_executor(environment: Literal["pbs"], **kwargs: Any) -> PBSExecutor: ...
 
 
 @overload
 def get_executor(
     environment: Literal["kubernetes"], **kwargs: Any
-) -> KubernetesExecutor:
-    ...
+) -> KubernetesExecutor: ...
 
 
 @overload
-def get_executor(environment: Literal["dask"], **kwargs: Any) -> DaskExecutor:
-    ...
+def get_executor(environment: Literal["dask"], **kwargs: Any) -> DaskExecutor: ...
 
 
 @overload
 def get_executor(
     environment: Literal["multiprocessing"], **kwargs: Any
-) -> MultiprocessingExecutor:
-    ...
+) -> MultiprocessingExecutor: ...
 
 
 @overload
 def get_executor(
     environment: Literal["sequential"], **kwargs: Any
-) -> SequentialExecutor:
-    ...
+) -> SequentialExecutor: ...
 
 
 @overload
 def get_executor(
     environment: Literal["debug_sequential"], **kwargs: Any
-) -> DebugSequentialExecutor:
-    ...
+) -> DebugSequentialExecutor: ...
 
 
 @overload
 def get_executor(
     environment: Literal["test_pickling"], **kwargs: Any
-) -> PickleExecutor:
-    ...
+) -> PickleExecutor: ...
 
 
 def get_executor(environment: str, **kwargs: Any) -> "Executor":

cluster_tools/cluster_tools/_utils/call.py

@@ -25,9 +25,7 @@ def call(command: str, stdin: Optional[str] = None) -> Tuple[str, str, int]:
 class CommandError(Exception):
     """Raised when a shell command exits abnormally."""
 
-    def __init__(
-        self, command: str, code: int, stderr: str
-    ):  # pylint: disable=super-init-not-called
+    def __init__(self, command: str, code: int, stderr: str):
         self.command = command
         self.code = code
         self.stderr = stderr

cluster_tools/cluster_tools/_utils/multiprocessing_logging_handler.py

@@ -52,7 +52,7 @@ def _receive(self) -> None:
                 record = self.queue.get(timeout=0.2)
                 if record is not None:
                     self.wrapped_handler.emit(record)
-            except (KeyboardInterrupt, SystemExit):  # pylint: disable=try-except-raise
+            except (KeyboardInterrupt, SystemExit):
                 raise
             # The following errors pop up quite often.
             # It seems that they can be safely ignored, though.