Skip to content

Commit bbcf756

Browse files
brokkoli71normanrzmarkbader
authored
tensorstore (#1174)
* migrate changes from branch tensorstore2 * correct decorator @lru_cache usage * correct parameter of TensorStoreArray::read * debug TensorStoreArray::open * update poetry.lock * format, pytest argument -s and skip failing test * install docker-compose in nightly.yml * fix import for python<3.11 * format * refreshed snapshots * progress * re-add displayname * progress * wip * progress * progress * progress * progress * newer wk * try ubicloud * http_proxy for tensorstore * refresh snapshots * refresh snapshots * rm ZarritaArray and ZarrArray + changelog * cassettes * --add-snapshots * tests/client/test_context.py::test_user_organization * Apply suggestions from code review Co-authored-by: Mark Bader <[email protected]> * bump wk * Apply suggestions from code review Co-authored-by: Mark Bader <[email protected]> * revert test_from_images * Remove test_examples.py * Fix typing issue. * wip: pr feedback * types and fixes * rm zarrita in test_layer * update cassettes * format * update cassettes * typing_extensions * fix test_dataset_download_upload_remote.py::test_remote_dataset * fix docs/uv.lock * format * use current wk version in nightly * LazyPath * fix test_annotation * Update .github/workflows/ci.yml --------- Co-authored-by: Norman Rzepka <[email protected]> Co-authored-by: Mark Bader <[email protected]>
1 parent 3744f1f commit bbcf756

File tree

75 files changed

+7837
-77988
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

75 files changed

+7837
-77988
lines changed

‎.github/workflows/nightly.yml‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,10 @@ jobs:
3434
git diff --no-ext-diff --quiet --exit-code
3535
[[ -z $(git status -s) ]]
3636
37+
- name: Use currently deployed Webknossos version
38+
run: |
39+
sed -i "s/export DOCKER_TAG=.*$/export DOCKER_TAG=master__$(curl https://webknossos.org/api/buildinfo | jq -r .webknossos.version)/" local_wk_setup.sh
40+
3741
- name: Python tests, refreshing the network snapshots
3842
env:
3943
WK_TOKEN: ${{ secrets.WK_TOKEN }}

‎docs/mkdocs.yml‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -234,7 +234,6 @@ nav:
234234
- webknossos-py/examples/download_image_data.md
235235
- webknossos-py/examples/download_tiff_stack.md
236236
- webknossos-py/examples/remote_datasets.md
237-
- webknossos-py/examples/zarr_and_dask.md
238237
- webknossos-py/examples/convert_4d_tiff.md
239238
- webknossos-py/examples/announce_dataset_upload.md
240239
- webknossos-py/examples/accessing_metadata.md

‎docs/src/webknossos-py/examples/zarr_and_dask.md‎

Lines changed: 0 additions & 9 deletions
This file was deleted.

‎docs/uv.lock‎

Lines changed: 71 additions & 184 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎webknossos/Changelog.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,10 +13,12 @@ For upgrade instructions, please check the respective _Breaking Changes_ section
1313
[Commits](https://github.com/scalableminds/webknossos-libs/compare/v0.16.1...HEAD)
1414

1515
### Breaking Changes
16+
- `MagView.get_zarr_array` now returns a `tensorstore` array instead of a `zarr-python` array. [#1174](https://github.com/scalableminds/webknossos-libs/pull/1174)
1617

1718
### Added
1819

1920
### Changed
21+
- Using tensorstore for reading and writing zarr 2 and 3 arrays. Removed `zarrita` and `zarr` dependency. [#1174](https://github.com/scalableminds/webknossos-libs/pull/1174)
2022

2123
### Fixed
2224

‎webknossos/examples/apply_merger_mode.py‎

Lines changed: 12 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,3 @@
1-
from os import environ
21
from pathlib import Path
32
from typing import Tuple, cast
43

@@ -84,19 +83,18 @@ def apply_mapping_for_chunk(args: Tuple[wk.View, wk.View, int]) -> None:
8483
# Optionally, downsample and re-upload #
8584
########################################
8685

87-
if "PYTEST_CURRENT_TEST" not in environ:
88-
out_layer.downsample()
89-
dataset.delete_layer("segmentation")
90-
dataset.upload(
91-
"l4_sample_remapped",
92-
layers_to_link=[
93-
wk.LayerToLink(
94-
dataset_name="l4_sample",
95-
layer_name="color",
96-
organization_id="scalable_minds",
97-
)
98-
],
99-
)
86+
out_layer.downsample()
87+
dataset.delete_layer("segmentation")
88+
dataset.upload(
89+
"l4_sample_remapped",
90+
layers_to_link=[
91+
wk.LayerToLink(
92+
dataset_name="l4_sample",
93+
layer_name="color",
94+
organization_id="scalable_minds",
95+
)
96+
],
97+
)
10098

10199

102100
if __name__ == "__main__":

‎webknossos/examples/zarr_and_dask.py‎

Lines changed: 0 additions & 27 deletions
This file was deleted.

‎webknossos/local_wk_setup.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
function export_vars {
22
export WK_TOKEN=1b88db86331a38c21a0b235794b9e459856490d70408bcffb767f64ade0f83d2bdb4c4e181b9a9a30cdece7cb7c65208cc43b6c1bb5987f5ece00d348b1a905502a266f8fc64f0371cd6559393d72e031d0c2d0cabad58cccf957bb258bc86f05b5dc3d4fff3d5e3d9c0389a6027d861a21e78e3222fb6c5b7944520ef21761e
33
export WK_URL=http://localhost:9000
4-
export DOCKER_TAG=master__30776
4+
export DOCKER_TAG=master__31116
55
}
66

77
function ensure_local_test_wk {

‎webknossos/pyproject.toml‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -38,25 +38,22 @@ dependencies = [
3838
"cattrs >=22.0.0",
3939
"certifi>=2023",
4040
"cluster-tools",
41-
"fsspec ~=2024.6.0",
4241
"httpx ~=0.27.0",
4342
"loxun ~=2.0.0",
4443
"natsort ~=8.4.0",
4544
"networkx ~=3.2.1",
46-
"numcodecs ~=0.12",
4745
"numpy ~=2.0.0",
4846
"pims ~=0.7.0",
4947
"psutil ~=6.0.0",
5048
"python-dateutil ~=2.8.0",
5149
"python-dotenv ~=1.0.1",
5250
"rich ~=13.8.0",
5351
"scipy ~=1.13",
52+
"tensorstore~=0.1.69",
5453
"typer ~=0.12.5",
5554
"typing-extensions ~=4.0",
5655
"universal-pathlib ~=0.2",
5756
"wkw ==1.1.24",
58-
"zarr ~=2.18.0",
59-
"zarrita ==0.2.7",
6057
"zipp ~=3.5.0",
6158
]
6259

@@ -91,7 +88,7 @@ all = [
9188
[tool.uv]
9289
extra-index-url = ["https://pypi.scm.io/simple"]
9390
dev-dependencies = [
94-
"dask[distributed] ~=2023.9.1; python_version >='3.9'",
91+
"fsspec ~=2024.6.0",
9592
"hypothesis ~=6.35.0",
9693
"icecream ~=2.1.1",
9794
"inducoapi ~=2.0.2",
@@ -160,7 +157,7 @@ target-version = "py39"
160157
# PERF = Perflint https://docs.astral.sh/ruff/rules/#perflint-perf
161158
# FIX = Flake8 fixme https://docs.astral.sh/ruff/rules/#flake8-fixme-fix
162159
# ARG = Flake8 unused-arguments https://docs.astral.sh/ruff/rules/#flake8-unused-arguments-arg
163-
select = ["F", "E", "W", "I", "A", "PERF", "FIX", "ARG"]
160+
select = ["F", "E", "W", "I", "A", "PERF", "FIX", "ARG", "T201"]
164161

165162
# Allow fix for all enabled rules (when `--fix`) is provided.
166163
fixable = ["ALL"]
@@ -169,6 +166,9 @@ fixable = ["ALL"]
169166
# E712 true-false-comparison https://docs.astral.sh/ruff/rules/true-false-comparison/
170167
ignore = ["E501", "E712"]
171168

169+
[tool.ruff.lint.per-file-ignores]
170+
"{examples,tests,script_collection,webknossos/cli}/*" = ["T201"]
171+
172172
[tool.ruff.format]
173173
# Like Black, use double quotes for strings.
174174
quote-style = "double"

‎webknossos/test.sh‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,21 @@ if [ $# -gt 0 ] && [ "$1" = "--refresh-snapshots" ]; then
3131

3232
stop_local_test_wk
3333

34+
exit $PYTEST_EXIT_CODE
35+
elif [ $# -gt 0 ] && [ "$1" = "--add-snapshots" ]; then
36+
ensure_local_test_wk
37+
38+
# Starts a proxy server in record mode on port 3000 and sets the HTTP_PROXY env var
39+
proxay --mode record --host http://localhost:9000 --tapes-dir tests/cassettes &
40+
PROXAY_PID=$!
41+
42+
shift
43+
$PYTEST "-m" "use_proxay" "$@"
44+
PYTEST_EXIT_CODE=$?
45+
trap 'kill $PROXAY_PID' EXIT
46+
47+
stop_local_test_wk
48+
3449
exit $PYTEST_EXIT_CODE
3550
elif [ $# -gt 0 ] && [ "$1" = "--debug-cassettes" ]; then
3651
# This will start a proxay server in replay mode so that the stored cassettes can be used for debugging tests.

0 commit comments

Comments
 (0)