Skip to content

Commit 7915fd2

Browse files
normanrznormanrz
authored
CLI improvments (#1372)
* CLI fixes * docs + overwrite * changelog * pr feedback * no-downsample
1 parent f90c026 commit 7915fd2

File tree

10 files changed

+490
-140
lines changed

10 files changed

+490
-140
lines changed

‎docs/src/api/index.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
<!doctype html>
2+
<html>
3+
<head>
4+
<meta charset="utf-8">
5+
<meta http-equiv="refresh" content="0; url=./webknossos.html"/>
6+
</head>
7+
</html>

‎docs/src/api/search.js‎

Lines changed: 46 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎docs/src/cli/convert.md‎

Lines changed: 139 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -5,8 +5,8 @@ The conversion commands allow you to transform various types of images or image
55
## Available Commands
66

77
- [`convert`](#convert): Converts an image stack to a WEBKNOSSOS dataset.
8-
- [`convert-knossos`](#convert-knossos): Converts a KNOSSOS dataset to a WEBKNOSSOS dataset.
98
- [`convert-raw`](#convert-raw): Converts a RAW dataset to a WEBKNOSSOS dataset.
9+
- [`convert-knossos`](#convert-knossos): Converts a KNOSSOS dataset to a WEBKNOSSOS dataset.
1010
- [`convert-zarr`](#convert-zarr): Converts a Zarr dataset to a WEBKNOSSOS dataset.
1111

1212
---
@@ -15,7 +15,7 @@ The conversion commands allow you to transform various types of images or image
1515

1616
The `convert` command automatically detects an image stack and converts it into a WEBKNOSSOS dataset.
1717

18-
## Supported input formats
18+
### Supported input formats
1919

2020
- Standard image formats, e.g. `tiff`, `jpg`, `png`, `bmp`
2121
- Proprietary image formats, e.g. `dm3`
@@ -66,7 +66,29 @@ webknossos convert [OPTIONS] SOURCE TARGET
6666

6767
- `--compress`
6868
Enable compression of the target dataset.
69-
Default: `False`.
69+
Default: `True`.
70+
71+
- `--downsample`
72+
Downsample the target dataset.
73+
Default: `True`.
74+
75+
- `--max-mag`
76+
Create downsampled magnifications up to the magnification specified by this argument.
77+
If omitted, the coarsest magnification will be determined by using the bounding box of the layer
78+
Example: `--max-mag 2-2-1`.
79+
80+
- `--interpolation-mode`
81+
The interpolation mode that should be used.
82+
Options: `median`, `mode`, `nearest`, `bilinear`, `bicubic`.
83+
Default: `default` (= `mode` for segmentation, `median` for color).
84+
85+
- `--sampling-mode`
86+
The sampling mode to use.
87+
Options: `anisotropic`, `isotropic`, `constant_z`.
88+
Default: `anisotropic`.
89+
90+
- `--overwrite-existing`
91+
Clear target folder, if it already exists. Not enabled by default. Use with caution.
7092

7193
- `--jobs`
7294
Number of processes to be spawned for parallel execution.
@@ -77,6 +99,13 @@ webknossos convert [OPTIONS] SOURCE TARGET
7799
Options: `multiprocessing`, `slurm`, `kubernetes`, `sequential`.
78100
Default: `multiprocessing`.
79101

102+
- `--batch-size`
103+
Number of images to be processed in one batch (influences RAM consumption).
104+
When creating a WKW dataset, batch-size must be a multiple of chunk-shape's z dimension.
105+
When converting to Zarr or Zarr3, batch-size must be a multiple of the z dimension of the
106+
shard shape (chunk-shape x chunks-per-shard).
107+
Default: None (= z dimension of shard-shape).
108+
80109
### Example Commands
81110

82111
Convert an image stack locally:
@@ -93,28 +122,32 @@ webknossos convert --jobs 4 /path/to/source/images /path/to/target/dataset
93122

94123
---
95124

96-
## `convert-knossos`
125+
## `convert-raw`
97126

98-
The `convert-knossos` command converts a KNOSSOS dataset into a WEBKNOSSOS dataset.
127+
The `convert-raw` command converts a RAW dataset into a WEBKNOSSOS dataset.
99128

100129
### Usage
101130

102131
```bash
103-
webknossos convert-knossos [OPTIONS] SOURCE TARGET
132+
webknossos convert-raw [OPTIONS] SOURCE TARGET
104133
```
105134

106135
### Arguments
107136

108137
- **SOURCE**
109-
Path to the source KNOSSOS dataset.
110-
Example: `/path/to/source/knossos`.
138+
Path to the source RAW dataset.
139+
Example: `/path/to/source/raw`.
111140

112141
- **TARGET**
113142
Path to the target WEBKNOSSOS dataset.
114143
Example: `/path/to/target/dataset`.
115144

116145
### Options
117146

147+
- `--shape`
148+
Shape of the source dataset.
149+
Example: `--shape 1024,1024,512`.
150+
118151
- `--voxel-size`
119152
The size of one voxel in the source data in nanometers.
120153
Example: `--voxel-size 11.0,11.0,20.0`.
@@ -123,18 +156,36 @@ webknossos convert-knossos [OPTIONS] SOURCE TARGET
123156
The unit of the voxel size.
124157
Default: `nanometers`.
125158

159+
- `--dtype`
160+
Target datatype (e.g., `uint8`, `uint16`, `uint32`).
161+
Default: `uint8`.
162+
163+
- `--source-dtype`
164+
Source datatype (e.g., `uint8`, `uint16`, `uint32`).
165+
If omitted, it is assumed to be the same as the target datatype.
166+
167+
- `--order`
168+
Input data storage layout: `C` (row-major) or `F` (column-major).
169+
Default: `F`.
170+
126171
- `--layer-name`
127172
Name of the layer to be created.
128173
Default: `color`.
129174

130-
- `--dtype`
131-
Target datatype (e.g., `uint8`, `uint16`, `uint32`).
132-
Default: `uint8`.
175+
- `--rescale-min-max`
176+
Rescale the values of the target dataset by specifying the min and max values.
177+
Will be scaled to the range from 0 to the maximum value of the target data type or 1.0 for floats.
178+
Example: `--rescale-min-max 0.2,0.8`.
179+
180+
- `--flip-axes`
181+
The axes at which should be flipped.
182+
Input format is a comma-separated list of axis indices.
183+
For example, 1,2,3 will flip the x, y and z axes.
133184

134185
- `--data-format`
135-
Data format to store the target dataset.
136-
Options: `wkw`, `zarr`, `zarr3`
137-
Default: `zarr3`
186+
Data format to store the target dataset.
187+
Options: `wkw`, `zarr`, `zarr3`
188+
Default: `zarr3`.
138189

139190
- `--chunk-shape`
140191
Number of voxels to be stored as a chunk in the output format.
@@ -144,6 +195,32 @@ webknossos convert-knossos [OPTIONS] SOURCE TARGET
144195
Number of voxels to be stored as a shard in the output format.
145196
Example: `--shard-shape 1024,1024,1024`.
146197

198+
- `--compress`
199+
Enable compression of the target dataset.
200+
Default: `True`.
201+
202+
- `--downsample`
203+
Downsample the target dataset.
204+
Default: `True`.
205+
206+
- `--max-mag`
207+
Create downsampled magnifications up to the magnification specified by this argument.
208+
If omitted, the coarsest magnification will be determined by using the bounding box of the layer
209+
Example: `--max-mag 2-2-1`.
210+
211+
- `--interpolation-mode`
212+
The interpolation mode that should be used.
213+
Options: `median`, `mode`, `nearest`, `bilinear`, `bicubic`.
214+
Default: `default` (= `mode` for segmentation, `median` for color).
215+
216+
- `--sampling-mode`
217+
The sampling mode to use.
218+
Options: `anisotropic`, `isotropic`, `constant_z`.
219+
Default: `anisotropic`.
220+
221+
- `--overwrite-existing`
222+
Clear target folder, if it already exists. Not enabled by default. Use with caution.
223+
147224
- `--jobs`
148225
Number of processes to be spawned for parallel execution.
149226
Default: Number of CPU cores.
@@ -155,50 +232,42 @@ webknossos convert-knossos [OPTIONS] SOURCE TARGET
155232

156233
### Example Commands
157234

158-
Convert a KNOSSOS dataset locally:
235+
Convert a RAW dataset locally:
159236

160237
```bash
161-
webknossos convert-knossos /path/to/source/knossos /path/to/target/dataset
238+
webknossos convert-raw --shape 1024,1024,512 /path/to/source/raw /path/to/target/dataset
162239
```
163240

164-
Convert a KNOSSOS dataset with custom voxel size and parallel execution:
241+
Convert a RAW dataset using parallel execution:
165242

166243
```bash
167-
webknossos convert-knossos --voxel-size 11.0,11.0,20.0 --jobs 4 /path/to/source/knossos /path/to/target/dataset
244+
webknossos convert-raw --shape 1024,1024,512 --jobs 4 /path/to/source/raw /path/to/target/dataset
168245
```
169246

170247
---
171248

172-
## `convert-raw`
249+
## `convert-knossos`
173250

174-
The `convert-raw` command converts a RAW dataset into a WEBKNOSSOS dataset.
251+
The `convert-knossos` command converts a KNOSSOS dataset into a WEBKNOSSOS dataset.
175252

176253
### Usage
177254

178255
```bash
179-
webknossos convert-raw [OPTIONS] SOURCE TARGET
256+
webknossos convert-knossos [OPTIONS] SOURCE TARGET
180257
```
181258

182259
### Arguments
183260

184261
- **SOURCE**
185-
Path to the source RAW dataset.
186-
Example: `/path/to/source/raw`.
262+
Path to the source KNOSSOS dataset.
263+
Example: `/path/to/source/knossos`.
187264

188265
- **TARGET**
189266
Path to the target WEBKNOSSOS dataset.
190267
Example: `/path/to/target/dataset`.
191268

192269
### Options
193270

194-
- `--shape`
195-
Shape of the source dataset.
196-
Example: `--shape 1024,1024,512`.
197-
198-
- `--order`
199-
Input data storage layout: `C` (row-major) or `F` (column-major).
200-
Default: `F`.
201-
202271
- `--voxel-size`
203272
The size of one voxel in the source data in nanometers.
204273
Example: `--voxel-size 11.0,11.0,20.0`.
@@ -216,9 +285,9 @@ webknossos convert-raw [OPTIONS] SOURCE TARGET
216285
Default: `uint8`.
217286

218287
- `--data-format`
219-
Data format to store the target dataset.
220-
Options: `wkw`, `zarr`, `zarr3`
221-
Default: `zarr3`.
288+
Data format to store the target dataset.
289+
Options: `wkw`, `zarr`, `zarr3`
290+
Default: `zarr3`
222291

223292
- `--chunk-shape`
224293
Number of voxels to be stored as a chunk in the output format.
@@ -239,20 +308,21 @@ webknossos convert-raw [OPTIONS] SOURCE TARGET
239308

240309
### Example Commands
241310

242-
Convert a RAW dataset locally:
311+
Convert a KNOSSOS dataset locally:
243312

244313
```bash
245-
webknossos convert-raw --shape 1024,1024,512 /path/to/source/raw /path/to/target/dataset
314+
webknossos convert-knossos /path/to/source/knossos /path/to/target/dataset
246315
```
247316

248-
Convert a RAW dataset using parallel execution:
317+
Convert a KNOSSOS dataset with custom voxel size and parallel execution:
249318

250319
```bash
251-
webknossos convert-raw --shape 1024,1024,512 --jobs 4 /path/to/source/raw /path/to/target/dataset
320+
webknossos convert-knossos --voxel-size 11.0,11.0,20.0 --jobs 4 /path/to/source/knossos /path/to/target/dataset
252321
```
253322

254323
---
255324

325+
256326
## `convert-zarr`
257327

258328
The `convert-zarr` command converts a Zarr dataset into a WEBKNOSSOS dataset.
@@ -287,6 +357,10 @@ webknossos convert-zarr [OPTIONS] SOURCE TARGET
287357
Name of the layer to be created.
288358
Default: `color`.
289359

360+
- `--is-segmentation-layer`
361+
When converting one layer, signals whether layer is segmentation layer.
362+
When converting a folder, this option is ignored.
363+
290364
- `--data-format`
291365
Data format to store the target dataset.
292366
Options: `wkw`, `zarr`, `zarr3`
@@ -300,10 +374,37 @@ webknossos convert-zarr [OPTIONS] SOURCE TARGET
300374
Number of voxels to be stored as a shard in the output format.
301375
Example: `--shard-shape 1024,1024,1024`.
302376

377+
- `--flip-axes`
378+
The axes at which should be flipped.
379+
Input format is a comma-separated list of axis indices.
380+
For example, 1,2,3 will flip the x, y and z axes.
381+
303382
- `--compress`
304383
Enable compression of the target dataset.
305384
Default: `True`.
306385

386+
- `--downsample`
387+
Downsample the target dataset.
388+
Default: `True`.
389+
390+
- `--max-mag`
391+
Create downsampled magnifications up to the magnification specified by this argument.
392+
If omitted, the coarsest magnification will be determined by using the bounding box of the layer
393+
Example: `--max-mag 2-2-1`.
394+
395+
- `--interpolation-mode`
396+
The interpolation mode that should be used.
397+
Options: `median`, `mode`, `nearest`, `bilinear`, `bicubic`.
398+
Default: `default` (= `mode` for segmentation, `median` for color).
399+
400+
- `--sampling-mode`
401+
The sampling mode to use.
402+
Options: `anisotropic`, `isotropic`, `constant_z`.
403+
Default: `anisotropic`.
404+
405+
- `--overwrite-existing`
406+
Clear target folder, if it already exists. Not enabled by default. Use with caution.
407+
307408
- `--jobs`
308409
Number of processes to be spawned for parallel execution.
309410
Default: Number of CPU cores.

‎webknossos/Changelog.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,10 @@ For upgrade instructions, please check the respective _Breaking Changes_ section
1616

1717
### Added
1818
- Added context manager `VolumeLayer.edit` for creating and modifying volume annotations. [#1340](https://github.com/scalableminds/webknossos-libs/pull/1340)
19+
- Added `overwrite_existing` flag to `convert`, `convert-raw` and `convert-zarr` CLI commands. Use with caution. [#1372](https://github.com/scalableminds/webknossos-libs/pull/1372)
20+
- Added `downsample`, `max-mag`, `interpolation-mode`, `sampling-mode` args to `convert`, `convert-raw` and `convert-zarr` CLI commands, where it was missing, for consistency. [#1372](https://github.com/scalableminds/webknossos-libs/pull/1372)
21+
- Added value rescaling to `convert-raw` CLI command through `source-dtype` and `rescale-min-max` args. [#1372](https://github.com/scalableminds/webknossos-libs/pull/1372)
22+
- Added `interpolation_mode` and `compress` kwargs to `Dataset.downsample` method. [#1372](https://github.com/scalableminds/webknossos-libs/pull/1372)
1923

2024
### Changed
2125

‎webknossos/tests/test_cli.py‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,7 @@ def _tiff_cubing(out_path: Path, data_format: DataFormat) -> None:
6868
"convert",
6969
"--jobs",
7070
2,
71+
"--no-downsample",
7172
"--voxel-size",
7273
"11.24,11.24,25",
7374
"--data-format",
@@ -324,6 +325,44 @@ def test_convert_with_all_params() -> None:
324325
assert (wkw_path / PROPERTIES_FILE_NAME).exists()
325326

326327

328+
def test_convert_raw() -> None:
329+
"""Tests the functionality of convert-raw subcommand."""
330+
331+
with tmp_cwd():
332+
origin_path = Path("test.raw")
333+
origin_path.write_bytes(
334+
np.array([[0.2, 0.4], [0.6, 0.8]], dtype="float32").tobytes(order="F")
335+
)
336+
out_path = Path(f"wkw_from_{origin_path.name}")
337+
result = runner.invoke(
338+
app,
339+
[
340+
"convert-raw",
341+
"--voxel-size",
342+
"11.0,11.0,11.0",
343+
"--shape",
344+
"2,2,1",
345+
"--dtype",
346+
"uint8",
347+
"--source-dtype",
348+
"float32",
349+
"--rescale-min-max",
350+
"0.2,0.8",
351+
str(origin_path),
352+
str(out_path),
353+
],
354+
)
355+
356+
assert result.exit_code == 0, result.stdout
357+
assert (out_path / PROPERTIES_FILE_NAME).exists()
358+
359+
out_ds = Dataset.open(out_path)
360+
np.testing.assert_array_equal(
361+
out_ds.get_layer("color").get_finest_mag().read()[0],
362+
np.array([[[0], [85]], [[170], [255]]], dtype="uint8"),
363+
)
364+
365+
327366
@pytest.mark.use_proxay
328367
@pytest.mark.parametrize(
329368
"url",

0 commit comments

Comments
 (0)