Always store path in mags (#1290)

* wip

* fixes

* update cassettes

* json

* add test for relative paths

* moar relative paths

* upgrade pylibczirw

* changelog

* fix issue in setup_mag

* Apply suggestions from code review

Co-authored-by: Mark Bader <[email protected]>

* wip

* allow metadata-only editing of read-only layers

* Update webknossos/Changelog.md

* version 2

* static checks

* no version 2

* wip

* ./

* tests

* frombuffer

* tests

* rename methods

* changelog

* tests

* cassettes

---------

Co-authored-by: Mark Bader <[email protected]>

Author: normanrz

webknossos/Changelog.md

@@ -13,11 +13,27 @@ For upgrade instructions, please check the respective _Breaking Changes_ section
 [Commits](https://github.com/scalableminds/webknossos-libs/compare/v2.3.11...HEAD)
 
 ### Breaking Changes
+- `Dataset.shallow_copy_dataset` does not symlink layers, mags and attachments anymore. Instead mags and attachments are referenced by their path. [#1290](https://github.com/scalableminds/webknossos-libs/pull/1290)
+- Deprecated a number of methods [#1290](https://github.com/scalableminds/webknossos-libs/pull/1290):
+  - `Dataset.add_symlink_layer`, use `Dataset.add_layer_as_ref` instead.
+  - `Dataset.add_remote_layer`, use `Dataset.add_layer_as_ref` instead (renamed).
+  - `Dataset.add_copy_layer`, use `Dataset.add_layer_as_copy` instead (renamed).
+  - `Layer.add_symlink_mag`, use `Layer.add_mag_as_ref` instead.
+  - `Layer.add_remote_mag`, use `Layer.add_mag_as_ref` instead (renamed).
+  - `Layer.add_copy_mag`, use `Layer.add_mag_as_copy` instead (renamed).
+  - `Attachments.add_symlink_attachments`, use `Attachments.add_attachment_as_ref` instead (note the singular form).
+  - `Attachments.add_attachments`, use `Attachments.add_attachment_as_ref` instead (note the singular form).
+  - `Attachments.add_copy_attachments`, use `Attachments.add_attachment_as_copy` instead (note the singular form).
+- `Dataset.add_layer` and `Dataset.add_layer_like` as well as derived methods do not eagerly create a folder for the layer anymore. Folders will be created, if necessary, when mags or attachments are added.  [#1290](https://github.com/scalableminds/webknossos-libs/pull/1290)
+
+
 
 ### Added
 
 ### Changed
 - Changed the default path type to `UPath` [#1326](https://github.com/scalableminds/webknossos-libs/pull/1326)
+- All mags now store their path in the `datasource-properties.json` file. This eliminates the need for symlinks and removes the need to check multiple possible mag paths (e.g. 1-1-1 or 1). It also unifies local and remote mags. [#1290](https://github.com/scalableminds/webknossos-libs/pull/1290)
+- Dataset-local mags and attachments use the `./` prefix for their path in compliance with [IETF RFC1808](https://datatracker.ietf.org/doc/html/rfc1808). [#1290](https://github.com/scalableminds/webknossos-libs/pull/1290)
 
 ### Fixed
 
@@ -61,7 +77,7 @@ For upgrade instructions, please check the respective _Breaking Changes_ section
 [Commits](https://github.com/scalableminds/webknossos-libs/compare/v2.3.7...v2.3.8)
 
 ### Changed
-- Made the folder_id parameter of `Dataset.announce_manual_upload` optional. [#1313](https://github.com/scalableminds/webknossos-libs/pull/1313)
+- Made the `folder_id` parameter of `Dataset.announce_manual_upload` optional. [#1313](https://github.com/scalableminds/webknossos-libs/pull/1313)
 
 
 ## [2.3.7](https://github.com/scalableminds/webknossos-libs/releases/tag/v2.3.7) - 2025-05-27

webknossos/testdata/simple_zarr3_dataset/datasource-properties.json

@@ -1,4 +1,5 @@
 {
+  "version": 1,
   "id": {
     "name": "simple_zarr_dataset",
     "team": ""
@@ -18,7 +19,8 @@
       "dataFormat": "zarr3",
       "mags": [
         {
-          "mag": [1, 1, 1]
+          "mag": [1, 1, 1],
+          "path": "./color/1"
         }
       ],
       "numChannels": 3,

webknossos/testdata/tiff/datasource-properties.zarr-fixture.json

@@ -28,7 +28,8 @@
             "x": 1,
             "y": 2,
             "z": 3
-          }
+          },
+          "path": "./tiff/1"
         }
       ],
       "numChannels": 1,

webknossos/tests/cassettes/test_dataset_add_remote_mag_and_layer/test_add_mag_ref_from_local_path.yml

@@ -0,0 +1 @@
+http_interactions: []