RemoteDataset.used_storage_bytes and n5/neuroglancer read support (#1386)

* add RemoteDataset.used_storage_bytes

* wip

* wip

* add tests for n5 and ng

* changelog

* test for storage scan

* Update webknossos/webknossos/dataset/remote_dataset.py

Co-authored-by: Florian M <[email protected]>

* _tensorstore_data_format

---------

Co-authored-by: Florian M <[email protected]>

Author: normanrz

webknossos/Changelog.md

@@ -27,12 +27,15 @@ For upgrade instructions, please check the respective _Breaking Changes_ section
 ### Breaking Changes
 - The `endpoint_url` in `UPath` objects is now stored directly in the `storage_options` dict, instead of being stored in the `storage_options["client_kwargs"]` dict. [#1365](https://github.com/scalableminds/webknossos-libs/pull/1365)
 - Removed Docker image `scalableminds/webknossos-cli` build. It will not be released further. [#1376](https://github.com/scalableminds/webknossos-libs/pull/1376)
+- Removed inference of channels in case `numChannels` is not set in the `datasource-properties.json`. `numChannels == 1` is assumed and fails upon reading, if otherwise. [#1386](https://github.com/scalableminds/webknossos-libs/pull/1386)
 
 ### Added
 - Dataset.add_layer_as_ref() now accepts RemoteLayer objects as well as Layer objects. [#1371](https://github.com/scalableminds/webknossos-libs/pull/1371])
 - RemoteDataset.annotation_id is now exposed, when available. [#1380](https://github.com/scalableminds/webknossos-libs/pull/1380)
 - RemoteDataset.open() now accepts an annotation url as well. [#1380](https://github.com/scalableminds/webknossos-libs/pull/1380)
 - RemoteAnnotation, now has a get_agglomerate_graph method, to make working with proofreading annotations easier. [#1361](https://github.com/scalableminds/webknossos-libs/pull/1361)
+- Added `RemoteDataset.used_storage_bytes` property. [#1386](https://github.com/scalableminds/webknossos-libs/pull/1386)
+- Added read support for N5 and neuroglancer precomputed data formats. [#1386](https://github.com/scalableminds/webknossos-libs/pull/1386)
 
 ### Changed
 - Ported `test.sh` to `test.py`, run with `uv run test.py`. [#1379](https://github.com/scalableminds/webknossos-libs/pull/1379)

webknossos/testdata/simple_n5_dataset/color/1/0/0/0/0

@@ -0,0 +1 @@
+{"blockSize":[1,32,32,32],"compression":{"level":-1,"type":"gzip","useZlib":false},"dataType":"uint32","dimensions":[3,24,24,24]}

webknossos/testdata/simple_n5_dataset/color/1/1/0/0/0

@@ -0,0 +1,35 @@
+{
+  "version": 1,
+  "id": {
+    "name": "simple_n5_dataset",
+    "team": ""
+  },
+  "scale": [1.0, 1.0, 1.0],
+  "dataLayers": [
+    {
+      "name": "color",
+      "category": "color",
+      "boundingBox": {
+        "topLeft": [0, 0, 0],
+        "width": 24,
+        "height": 24,
+        "depth": 24
+      },
+      "elementClass": "uint24",
+      "dataFormat": "n5",
+      "mags": [
+        {
+          "mag": [1, 1, 1],
+          "path": "./color/1"
+        }
+      ],
+      "numChannels": 3,
+      "defaultViewConfiguration": {
+        "color": [255, 0, 0]
+      }
+    }
+  ],
+  "defaultViewConfiguration": {
+    "position": [12, 12, 12]
+  }
+}

webknossos/testdata/simple_n5_dataset/color/1/2/0/0/0

@@ -0,0 +1 @@
+{"@type":"neuroglancer_multiscale_volume","data_type":"uint8","num_channels":3,"scales":[{"chunk_sizes":[[64,64,64]],"encoding":"png","key":"1","png_level":5,"resolution":[1.0,1.0,1.0],"size":[24,24,24],"voxel_offset":[0,0,0]}],"type":"image"}

webknossos/testdata/simple_n5_dataset/color/1/attributes.json

@@ -0,0 +1,35 @@
+{
+  "version": 1,
+  "id": {
+    "name": "simple_n5_dataset",
+    "team": ""
+  },
+  "scale": [1.0, 1.0, 1.0],
+  "dataLayers": [
+    {
+      "name": "color",
+      "category": "color",
+      "boundingBox": {
+        "topLeft": [0, 0, 0],
+        "width": 24,
+        "height": 24,
+        "depth": 24
+      },
+      "elementClass": "uint24",
+      "dataFormat": "neuroglancerPrecomputed",
+      "mags": [
+        {
+          "mag": [1, 1, 1],
+          "path": "./color/1"
+        }
+      ],
+      "numChannels": 3,
+      "defaultViewConfiguration": {
+        "color": [255, 0, 0]
+      }
+    }
+  ],
+  "defaultViewConfiguration": {
+    "position": [12, 12, 12]
+  }
+}