Cloud storage support for wklibs+zarr (#649)

* change from os.path to pathlib
* moar
* relpath
* Merge remote-tracking branch 'origin/master' into more-pathlib
* fixes
* fixes
* adds wkw backend
* stuff
* fixes
* add zarr+numcodecs
* typing
* fixes
* fixes
* fixe
* fixes
* fixes
* fixes
* fxies
* fixes
* fixes
* fixes
* fixes
* fixes
* fixes
* fixes
* fixes
* fixes
* fixes
* merged
* poetry.lock
* data_format
* adds tests for zarr
* tests for zarr
* fixes compression
* lint
* types
* deprecation warnings
* moar checks
* refactoring
* rename
* fixes
* remove shardSize from datasource-properties.json
* better resizing
* fixes
* fixes
* wkwResolutions vs resolutions
* merge
* fixes
* enum + copy_dataset
* Apply suggestions from code review

Co-authored-by: Philipp Otto <[email protected]>
* pr feedback, data_format -> array_format
* Merge branch 'more-pathlib' of github.com:scalableminds/webknossos-libs into more-pathlib
* caching zarr array
* Update webknossos/webknossos/dataset/dataset.py

Co-authored-by: Philipp Otto <[email protected]>
* pr feedback
* merged
* pr feedback
* pr feedback
* rework test_dataset.py and relative imports
* formatting
* formatting
* fixes
* tests
* rename resolutions -> mags
* merged
* pr feedback
* initial upath test
* poetry
* test fixes?
* Update Changelog.md
* merge
* Merge branch 'more-pathlib' into upath
* add stubfile for UPath
* Merge branch 'upath' of github.com:scalableminds/webknossos-libs into upath
* typing fixes
* moar upath
* fix tests?
* format
* revert test fixes
* tests?
* dirty?
* ci
* poetry.lock
* use F order
* tests
* format
* merge
* fixes
* merged
* fixes
* fix tests
* merge
* fixes
* update upath
* Merge remote-tracking branch 'origin/master' into upath
* symlink asserts
* fixes
* fixes
* test symlinks
* fixes
* fixes
* fixes
* fixes
* fixes
* fixes
* fixes
* fixes
* fix tests
* fixes
* better test
* more asserts
* changelog
* changelog
* cleanup
* Merge remote-tracking branch 'origin/master' into upath
* pr feedback
* merged
* rework tests
* pr feedback
* docs
* use Path in method signatures
* merge
* merge
* fixes bbox in Layer.downsample_mag
* tests
* tests
* tests
* Update test.sh
* tests
* tests
* test
* tests
* fixes
* pr feedback
* move minio to port 8000
* stuff
* Merge remote-tracking branch 'origin/master' into upath
* merge
* lint

Author: normanrz

cluster_tools/cluster_tools/__init__.py

@@ -1,12 +1,12 @@
 import logging
 import multiprocessing
 import os
-import shutil
 import tempfile
 from concurrent import futures
 from concurrent.futures import ProcessPoolExecutor
 from functools import partial
 from pathlib import Path
+from shutil import rmtree
 
 from . import pickling
 from .multiprocessing_logging_handler import get_multiprocessing_logging_setup_fn
@@ -137,8 +137,7 @@ def _submit_via_io(self, *args, **kwargs):
 
     @staticmethod
     def _remove_tmp_file(path, _future):
-
-        shutil.rmtree(path)
+        rmtree(path)
 
     @staticmethod
     def _setup_logging_and_execute(multiprocessing_logging_setup_fn, *args, **kwargs):

docs/poetry.lock

@@ -104,4 +104,4 @@ ENV/
 .mypy_cache/
 
 /webknossos/version.py
-testoutput
+testoutput

webknossos/.gitignore

@@ -15,6 +15,7 @@ For upgrade instructions, please check the respective *Breaking Changes* section
 ### Breaking Changes
 
 ### Added
+- Added cloud storage support for the `Dataset` classes, by using `UPath` from [universal_pathlib](https://github.com/fsspec/universal_pathlib) and [fsspec](https://github.com/fsspec/fsspec). Create remote datasets like this `Dataset(UPath("s3://bucket/path/to/dataset", key="...", secret="..."), scale=(11, 11, 24))`. Datasets on cloud storage only work with [Zarr](https://zarr.dev/)-based layers. [#649](https://github.com/scalableminds/webknossos-libs/pull/649)
 
 ### Changed
 
@@ -39,6 +40,7 @@ For upgrade instructions, please check the respective *Breaking Changes* section
 
 ### Changed
 - `MagView.compress` now skips in-place compression of already compressed mags. [#667](https://github.com/scalableminds/webknossos-libs/pull/667)
+- Replaced uses of `pathlib.Path` with `UPath` from [universal_pathlib](https://github.com/fsspec/universal_pathlib). Since `UPath` is compatible with `pathlib.Path`, changes in user code are not necessary. [#649](https://github.com/scalableminds/webknossos-libs/pull/649)
 
 ### Fixed
 - Fixed compression of downsampled mags for layers with arbitrary and potentially mag-unaligned bounding boxes. [#667](https://github.com/scalableminds/webknossos-libs/pull/667)