Skip to content

Commit 2886661

Browse files
formanforman
authored
Merge pull request #4 from bcdev/schema-to-md
Schema to md
2 parents 0511815 + 5f6c535 commit 2886661

File tree

5 files changed

+448
-51
lines changed

5 files changed

+448
-51
lines changed

‎CONFIG.md‎

Lines changed: 163 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,163 @@
1+
Configuration for the zappend tool.
2+
3+
### `version`
4+
5+
Configuration version.
6+
It's value is `1`.
7+
8+
### `target_uri`
9+
10+
Type _string_.
11+
The URI or local path of the target Zarr dataset. Must be a directory.
12+
13+
### `target_storage_options`
14+
15+
Type _object_.
16+
Options for the filesystem given by the URI of `target_uri`.
17+
18+
### `slice_engine`
19+
20+
Type _string_.
21+
The name of the engine to be used for opening contributing datasets. Refer to the `engine` argument of the function `xarray.open_dataset()`.
22+
23+
### `slice_storage_options`
24+
25+
Type _object_.
26+
Options for the filesystem given by the protocol of the URIs of contributing datasets.
27+
28+
### `slice_polling`
29+
30+
Defines how to poll for contributing datasets.
31+
Must be one of the following:
32+
* No polling, fail immediately if dataset is not available.
33+
It's value is `false`.
34+
* Poll using default values.
35+
It's value is `true`.
36+
* Type _object_.
37+
Polling parameters.
38+
* `interval`:
39+
Type _number_.
40+
Polling interval in seconds.
41+
Defaults to `2`.
42+
43+
* `timeout`:
44+
Type _number_.
45+
Polling timeout in seconds.
46+
Defaults to `60`.
47+
48+
49+
### `temp_dir`
50+
51+
Type _string_.
52+
The URI or local path of the directory that will be used to temporarily store rollback information.
53+
54+
### `temp_storage_options`
55+
56+
Type _object_.
57+
Options for the filesystem given by the protocol of `temp_dir`.
58+
59+
### `zarr_version`
60+
61+
The Zarr version to be used.
62+
It's value is `2`.
63+
64+
### `fixed_dims`
65+
66+
Type _object_.
67+
Specifies the fixed dimensions of the target dataset. Keys are dimension names, values are dimension sizes.
68+
Object values are:
69+
70+
Type _integer_.
71+
72+
### `append_dim`
73+
74+
Type _string_.
75+
The name of the variadic append dimension.
76+
Defaults to `"time"`.
77+
78+
### `variables`
79+
80+
Type _object_.
81+
Defines dimensions, encoding, and attributes for variables in the target dataset. Object property names refer to variable names. The special name `*` refers to all variables, which is useful for defining common values.
82+
Object values are:
83+
84+
Type _object_.
85+
Variable metadata
86+
* `dims`:
87+
Type _array_.
88+
The names of the variable's dimensions in the given order. Each dimension must exist in contributing datasets.
89+
90+
* `encoding`:
91+
Type _object_.
92+
Variable storage encoding. Settings given here overwrite the encoding settings of the first contributing dataset.
93+
* `dtype`:
94+
Storage data type
95+
Must be one of `"int8", "uint8", "int16", "uint16", "int32", "uint32", "int64", "uint64", "float32", "float64"`.
96+
97+
* `chunks`:
98+
Storage chunking.
99+
Must be one of the following:
100+
* Type _array_.
101+
Chunk sizes in the order of the dimensions.
102+
* Disable chunking.
103+
It's value is `null`.
104+
105+
* `fill_value`:
106+
Storage fill value.
107+
Must be one of the following:
108+
* Type _number_.
109+
A number of type and unit of the given storage `dtype`.
110+
* Not-a-number. Can be used only if storage `dtype` is `float32` or `float64`.
111+
It's value is `"NaN"`.
112+
* No fill value.
113+
It's value is `null`.
114+
115+
* `scale_factor`:
116+
Type _number_.
117+
Scale factor for computing the in-memory value: `memory_value = scale_factor * storage_value + add_offset`.
118+
119+
* `add_offset`:
120+
Type _number_.
121+
Add offset for computing the in-memory value: `memory_value = scale_factor * storage_value + add_offset`.
122+
123+
* `units`:
124+
Type _string_.
125+
Units of the storage data type if memory data type is date/time.
126+
127+
* `calendar`:
128+
Type _string_.
129+
The calendar to be used if memory data type is date/time.
130+
131+
* `compressor`:
132+
Type _array_ | _null_.
133+
Compressor. Set to `null` to disable data compression.
134+
* `id`:
135+
Type _string_.
136+
137+
$`$id` are required.
138+
139+
* `filters`:
140+
Type _array_ | _null_.
141+
Filters. Set to `null` to not use filters.
142+
143+
144+
* `attrs`:
145+
Type _object_.
146+
Arbitrary variable metadata attributes.
147+
148+
149+
### `included_variables`
150+
151+
Type _array_.
152+
Specifies the names of variables to be included in the target dataset. Defaults to all variables found in the first contributing dataset.
153+
154+
### `excluded_variables`
155+
156+
Type _array_.
157+
Specifies the names of individual variables to be excluded from all contributing datasets.
158+
159+
### `dry_run`
160+
161+
Type _boolean_.
162+
If 'true', log only what would have been done, but don't apply any changes.
163+
Defaults to `false`.

‎tests/test_cli.py‎

Lines changed: 14 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -16,14 +16,14 @@
1616
Create or update a Zarr dataset TARGET from slice datasets SLICES.
1717
1818
Options:
19-
-c, --config CONFIG Configuration JSON or YAML file. If multiple are passed,
20-
they will be deeply merged into one.
21-
-t, --target TARGET Target Zarr dataset path or URI. Overrides the
22-
'target_uri' configuration field.
23-
--dry-run Run the tool without creating, changing, or deleting any
24-
files.
25-
--help-config Show configuration help and exit.
26-
--help Show this message and exit.
19+
-c, --config CONFIG Configuration JSON or YAML file. If multiple are
20+
passed, they will be deeply merged into one.
21+
-t, --target TARGET Target Zarr dataset path or URI. Overrides the
22+
'target_uri' configuration field.
23+
--dry-run Run the tool without creating, changing, or deleting
24+
any files.
25+
--help-config json|md Show configuration help and exit.
26+
--help Show this message and exit.
2727
"""
2828

2929
# remove indent
@@ -45,9 +45,13 @@ def test_help(self):
4545
def test_help_config(self):
4646
runner = CliRunner()
4747
# noinspection PyTypeChecker
48-
result = runner.invoke(zappend, ['--help-config'])
48+
result = runner.invoke(zappend, ['--help-config', 'json'])
4949
self.assertEqual(0, result.exit_code)
50-
self.assertIn("Configuration JSON schema:", result.output)
50+
self.assertIn('"target_uri": {', result.output)
51+
# noinspection PyTypeChecker
52+
result = runner.invoke(zappend, ['--help-config', 'md'])
53+
self.assertEqual(0, result.exit_code)
54+
self.assertIn('### `target_uri`', result.output)
5155

5256
def test_no_slices(self):
5357
runner = CliRunner()

‎tests/test_config.py‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,8 @@
1010
import yaml
1111

1212
from zappend.config import CONFIG_V1_SCHEMA
13+
from zappend.config import schema_to_json
14+
from zappend.config import schema_to_md
1315
from zappend.config import merge_configs
1416
from zappend.config import normalize_config
1517
from zappend.config import validate_config
@@ -245,3 +247,15 @@ def test_merge_config(self):
245247
self.assertEqual({"a": {"b": 3, "c": 4}},
246248
merge_configs({"a": {"b": 2, "c": 4}},
247249
{"a": {"b": 3}}))
250+
251+
def test_schema_to_json(self):
252+
# Smoke test is sufficient here
253+
text = schema_to_json()
254+
self.assertIsInstance(text, str)
255+
self.assertTrue(len(text) > 0)
256+
257+
def test_schema_to_md(self):
258+
# Smoke test is sufficient here
259+
text = schema_to_md()
260+
self.assertIsInstance(text, str)
261+
self.assertTrue(len(text) > 0)

‎zappend/cli.py‎

Lines changed: 10 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -20,18 +20,20 @@
2020
@click.option("--dry-run", is_flag=True,
2121
help="Run the tool without creating, changing,"
2222
" or deleting any files.")
23-
@click.option("--help-config", is_flag=True,
23+
@click.option("--help-config",
24+
metavar="json|md",
25+
type=click.Choice(["json", "md"]),
2426
help="Show configuration help and exit.")
2527
def zappend(slices: tuple[str, ...],
2628
config: tuple[str, ...],
2729
target: str | None,
2830
dry_run: bool,
29-
help_config: bool):
31+
help_config: str | None):
3032
"""Create or update a Zarr dataset TARGET from slice datasets SLICES.
3133
"""
3234

3335
if help_config:
34-
return _show_config_help()
36+
return _show_config_help(help_config)
3537

3638
if not slices:
3739
click.echo("No slice datasets given.")
@@ -45,12 +47,11 @@ def zappend(slices: tuple[str, ...],
4547
raise click.ClickException(f"{e}") from e
4648

4749

48-
def _show_config_help():
49-
import json
50-
from zappend.config import CONFIG_V1_SCHEMA
51-
config_schema_json = json.dumps(CONFIG_V1_SCHEMA, indent=2)
52-
print(f"Configuration JSON schema:\n")
53-
print(config_schema_json)
50+
def _show_config_help(config_help_format: str):
51+
from zappend.config import schema_to_json
52+
from zappend.config import schema_to_md
53+
to_text = schema_to_json if config_help_format == "json" else schema_to_md
54+
print(to_text() + "\n")
5455

5556

5657
if __name__ == '__main__':

0 commit comments

Comments
 (0)