refactor dataset, remote_dataset, and layer (#1371)

* making the path optional in dataset, and layers

* typecheck. format. lint

* add default for version. fix structuring

* fix more type errors

* add reserve manual upload.

* implement upload on dataset object.

* update to api_version 11

* use enrich_path

* remove status from valid DatasetProperties

* add default argument

* always ask for included paths in dataset_info

* make dataset.path and layer.path optional

* moved dataset_properties

* format

* fix ruff for __init__

* fix missing import

* fix copy of attachments

* fix imports

* fix import

* move layer_categories.py

* reorder code in structuring

* reorder is None check

* improve comment

* move layer category

* move imports

* no need for conversion anymore

* fix disambiguate

* fix schema

* wk now has lower case user emails

* wk now has lower case user emails

* missing 7 cassettes.

* elaborate comment about legacy field DatasetProperties.id

* add_attachment_as_copy now supports RemoteDatasets

* don't use RemoteDataset type

* fix user names

* missing one cassette

* separate type names and container names

* do not pass includePaths

* all cassettes!

* allow add_attachment_as_copy to publish attachments to remote datasets

* update cassettes

* rename manual_upload to upload_to_path*s

* Adapt reserve routes to pass dataset_id in path; add new reserve_dataset_upload_to_paths_for_preliminary

* remove default for version

* add publish_to_preliminary_dataset

* add announce_dataset_upload.py

* update docs

* allow moving of the data when publishing

* use dataset updatePartial, update snapshots

* Allow saving RemoteDataset DatasetProperties changes back to server

* in case of symlink, directly link, no move

* rename

* add env variable to __init__.py

* move layer to link

* consolidate upload

* Revert "move layer to link"

This reverts commit 3fed75a.

* format

* use corrent variable

* refactor dataset, remote_dataset, and layer

* clean up

* refactor packages. typecheck runs through

* fixed imports

* move _array

* add layer and remote_layer to mag view

* rename attachments_type

* move download to abstract dataset

* move open_remote to abstract_dataset.py

* remove init

* make magview generic over layer

* adapt comment

* add_layer_like now accepts RemoteLayers. remove test for adding remote layers with a zarr-streaming path

* update snapshots

* increase chunk shape

* add changelog

* format and lint

* remove unnecessary asserts

* fix magview rechunk

* remove unused method

* clean up, removing code from older iterations

* removed None checks

* update changelog

* format

* restore from_remote_layer method

* use Mapping

* move download_mesh to RemoteSegmentationLayer

* move Dataset.get_remote_datasets -> RemoteDataset.list and Dataset.open_remote -> RemoteDataset.open

* use RemoteDataset.open instead of Dataset.open_remote

* fix usage of deprecated methods

* format

* add backwards compatible deprecated is_foreign property on MagView

* make ensure_layer compatible with remote layers

* update changelog

* update cassettes

* update api version to 12

* rename

* add upload_to_webknossos example to docs

* remove old link

* fix changelog

* allow new name to be passed to LayerToLink.from_remote_layer

* add tests for upload to common storage

* lint & format

* update cassettes

* move _parse_remote, _disambiguate_remote, get_remote_datasets from abstract dataset to dataset or remote dataset

* Update webknossos/Changelog.md

Co-authored-by: Norman Rzepka <[email protected]>

* add deprecation to changelog

* fix changelog

* format and fix ensure_layer

* add missing cassettes

* skip upload tests

---------

Co-authored-by: valentin-pinkau <[email protected]>
Co-authored-by: Florian M <[email protected]>
Co-authored-by: Norman Rzepka <[email protected]>

Author: 4 people

docs/mkdocs.yml

@@ -245,9 +245,9 @@ nav:
         - webknossos-py/examples/download_tiff_stack.md
         - webknossos-py/examples/remote_datasets.md
         - webknossos-py/examples/convert_4d_tiff.md
-        - webknossos-py/examples/announce_dataset_upload.md
         - webknossos-py/examples/accessing_metadata.md
         - webknossos-py/examples/explore_and_add_remote.md
+        - webknossos-py/examples/upload_to_webknossos.md
       - Annotation Examples:
         - webknossos-py/examples/apply_merger_mode.md
         - webknossos-py/examples/learned_segmenter.md

docs/src/webknossos-py/examples/announce_dataset_upload.md

@@ -0,0 +1,9 @@
+# Upload dataset to WEBKNOSSOS
+
+This example demonstrates the process of uploading a Dataset. 
+
+```python
+--8<--
+webknossos/examples/upload_to_webknossos.py
+--8<--
+```

docs/src/webknossos-py/examples/upload_to_webknossos.md

@@ -14,10 +14,22 @@ For upgrade instructions, please check the respective _Breaking Changes_ section
 
 ### Breaking Changes
 - Removed Docker image `scalableminds/webknossos-cli` build. It will not be released further. [#1376](https://github.com/scalableminds/webknossos-libs/pull/1376)
+- `Dataset.add_layer_as_ref(remote_layer.path)` does not work anymore. Please use `Dataset.add_layer_as_ref(remote_layer)` instead. [#1371](https://github.com/scalableminds/webknossos-libs/pull/1371])
+- Due to the refactoring, the imports of various classes have changed. Please update your code accordingly. [#1371](https://github.com/scalableminds/webknossos-libs/pull/1371])
+- Deprecated a number of methods [#1371](https://github.com/scalableminds/webknossos-libs/pull/1371]):
+  - `Dataset.open_remote()`, use `RemoteDataset.open` instead.
+  - `RemoteDatset.download_mesh()`, use `remote_segmentation_layer.download_mesh()` instead.
+  - `Dataset.get_remote_datasets()`, use `RemoteDataset.list()` instead.
+  - `Dataset.download()`, use `RemoteDataset.download()` instead.
+  - `mag_view.is_foreign`, use `layer.is_mag_view_foreign(mag_view)` instead.
+  - `Dataset.trigger_reload_in_datastore()`, use `RemoteDataset.trigger_reload_in_datastore()` instead.
 
 ### Added
+- Dataset.add_layer_as_ref() now accepts RemoteLayer objects as well as Layer objects. [#1371](https://github.com/scalableminds/webknossos-libs/pull/1371])
 
 ### Changed
+- Refactored the architecture, by introducing RemoteLayers, RemoteSegmentationLayers and their abstract base classes. [#1371](https://github.com/scalableminds/webknossos-libs/pull/1371])
+- Updated the api version of the webknossos-api to 12. [#1371](https://github.com/scalableminds/webknossos-libs/pull/1371])
 
 ### Fixed
 

webknossos/Changelog.md

@@ -2,7 +2,7 @@
 
 
 def main() -> None:
-    l4_sample_dataset = wk.Dataset.open_remote("l4_sample")
+    l4_sample_dataset = wk.RemoteDataset.open("l4_sample")
     # Access the metadata of the dataset
     print(l4_sample_dataset.metadata)
 

webknossos/examples/accessing_metadata.py

@@ -1,11 +1,7 @@
-from pathlib import Path
-
 import fastremap
 
 import webknossos as wk
 
-TESTDATA_DIR = Path(__file__).parent.parent / "testdata"
-
 
 def main() -> None:
     ####################################
@@ -18,15 +14,14 @@ def main() -> None:
     ).skeleton
 
     ###############################################
-    # Download and open the corresponding dataset #
+    # Open the corresponding dataset #
     ###############################################
 
-    dataset = wk.Dataset.download(
+    input_dataset = wk.RemoteDataset.open(
         "l4_sample",
-        bbox=wk.BoundingBox((3457, 3323, 1204), (40, 10, 10)),
-        path="testoutput/l4_sample",
+        organization_id="scalable_minds",
     )
-    in_layer = dataset.get_segmentation_layer("segmentation")
+    in_layer = input_dataset.get_segmentation_layer("segmentation")
     in_mag1 = in_layer.get_mag("1")
 
     ##############################
@@ -52,8 +47,10 @@ def main() -> None:
     ############################
     # Creating an output layer #
     ############################
-
-    out_layer = dataset.add_layer(
+    output_dataset = wk.Dataset(
+        "testoutput/l4_sample", voxel_size=input_dataset.voxel_size
+    )
+    out_layer = output_dataset.add_layer(
         "segmentation_remapped",
         wk.SEGMENTATION_CATEGORY,
         dtype_per_channel=in_layer.dtype_per_channel,
@@ -84,16 +81,9 @@ def apply_mapping_for_chunk(args: tuple[wk.View, wk.View, int]) -> None:
     ########################################
 
     out_layer.downsample()
-    dataset.delete_layer("segmentation")
-    dataset.upload(
+    output_dataset.upload(
         "l4_sample_remapped",
-        layers_to_link=[
-            wk.LayerToLink(
-                dataset_name="l4_sample",
-                layer_name="color",
-                organization_id="scalable_minds",
-            )
-        ],
+        layers_to_link=[input_dataset.get_layer("color")],
     )
 
 

webknossos/examples/announce_dataset_upload.py

@@ -1,7 +1,6 @@
 from pathlib import Path
 
-from webknossos import Dataset
-from webknossos.dataset import COLOR_CATEGORY
+from webknossos import COLOR_CATEGORY, Dataset
 
 INPUT_DIR = Path(__file__).parent.parent / "testdata" / "tiff"
 OUTPUT_DIR = Path("testoutput/my_tiff_dataset")

webknossos/examples/apply_merger_mode.py

@@ -9,7 +9,6 @@ def main() -> None:
     #####################
     # Opening a dataset #
     #####################
-
     dataset = wk.Dataset.open("testdata/simple_wkw_dataset")
     # Assuming that the dataset has a layer "color"
     # and the layer has the magnification 1
@@ -67,7 +66,7 @@ def main() -> None:
 
     copy_of_dataset = dataset.copy_dataset(
         "testoutput/copy_of_dataset",
-        chunk_shape=(8, 8, 8),
+        chunk_shape=(32, 32, 32),
         shard_shape=(64, 64, 64),
         compress=True,
     )

webknossos/examples/create_dataset_from_images.py

@@ -15,13 +15,14 @@ def main() -> None:
     bbox = wk.BoundingBox([50, 50, 50], [50, 50, 50])
 
     # Download the dataset
-    ds = wk.Dataset.download(
+    ds = wk.RemoteDataset.open(
         dataset_name_or_url=name,
         organization_id=organization_id,
+        webknossos_url="https://webknossos.org",
+    ).download(
         bbox=bbox,
         layers=layers,
         mags=mags,
-        webknossos_url="https://webknossos.org",
     )
 
     # Check that the layer was downloaded