Merge branch 'main' into forman-56-no_gobal_attrs

Author: forman

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,28 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: bug
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Describe how to reproduce the behavior.
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Python Environment**
+ - operating system:
+ - zappend version, output of `zappend --version`:
+ - optional: packages and their versions, output of `pip list` or `conda list`: 
+
+**Additional context**
+Add any other context about the problem here.
+
+**Traceback**
+If applicable, add a traceback of your error here.

.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,20 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: enhancement
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,10 @@
+[Description of PR]
+
+Checklist (strike out non-applicable):
+
+* [ ] Changes documented in `CHANGES.md`
+* [ ] Related issue exists and is referred to in the PR description and `CHANGES.md`
+* [ ] Added docstrings and API docs for any new/modified user-facing classes and functions
+* [ ] Changes/features documented in `docs/*`
+* [ ] Unit-tests adapted/added for changes/features 
+* [ ] Test coverage remains or increases (target 100%)

CHANGES.md

@@ -4,6 +4,15 @@
 
 * Global metadata attributes of target dataset is no longer empty. [#56]
 
+* If the target _parent_ directory did not exist, an exception was raised 
+  reporting that the lock file to be written does not exist. Changed this to
+  report that the target parent directory does not exist. [#55]
+
+### Enhancements
+
+* Added missing documentation of the `append_step` setting in the 
+  [configuration reference](https://bcdev.github.io/zappend/config/).
+
 ## Version 0.4.0 (from 2024-02-08)
 
 ### Enhancements

SECURITY.md

@@ -0,0 +1,13 @@
+# Security Policy
+
+## Supported Versions
+
+| Version | Supported          |
+| ------- | ------------------ |
+| 0.4.x   | :white_check_mark: |
+| < 0.4   | :x:                |
+
+## Reporting a Vulnerability
+
+To report a vulnerability, please post and [issue](https://github.com/bcdev/zappend/issues)
+and use prefix `[SECURITY]` for the title. Security issues will be treated with high priority.

docs/config.md

@@ -4,13 +4,13 @@
 ## `version`
 
 Configuration schema version. Allows the schema to evolve while still preserving backwards compatibility.
-Its value must be `1`.
+Its value is `1`.
 Defaults to `1`.
 
 ## `zarr_version`
 
 The Zarr version to be used.
-Its value must be `2`.
+Its value is `2`.
 Defaults to `2`.
 
 ## `fixed_dims`
@@ -25,6 +25,28 @@ Type _string_.
 The name of the variadic append dimension.
 Defaults to `"time"`.
 
+## `append_step`
+
+If set, enforces a step size in the append dimension between two slices or just enforces a direction.
+Must be one of the following:
+
+  * Arbitrary step size or not applicable.
+    Its value is `null`.
+
+  * Monotonically increasing.
+    Its value is `"+"`.
+
+  * Monotonically decreasing.
+    Its value is `"-"`.
+
+  * Type _string_.
+    A positive or negative time delta value, such as `12h`, `2D`, `-1D`.
+
+  * Type _number_.
+    A positive or negative numerical delta value.
+
+Defaults to `null`.
+
 ## `included_variables`
 
 Type _array_.
@@ -59,27 +81,29 @@ Variable metadata.
 
       * `chunks`:
         Storage chunking.
-        Must be one of the following.
+        Must be one of the following:
 
           * Type _array_.
             Chunk sizes in the order of the dimensions.
             The items of the array are of type _integer_.
 
           * Disable chunking.
-            Its value must be `null`.
+            Its value is `null`.
+        
 
       * `fill_value`:
         Storage fill value.
-        Must be one of the following.
+        Must be one of the following:
 
           * Type _number_.
             A number of type and unit of the given storage `dtype`.
 
           * Not-a-number. Can be used only if storage `dtype` is `float32` or `float64`.
-            Its value must be `"NaN"`.
+            Its value is `"NaN"`.
 
           * No fill value.
-            Its value must be `null`.
+            Its value is `null`.
+        
 
       * `scale_factor`:
         Type _number_.
@@ -122,7 +146,7 @@ Variable metadata.
 ## `target_dir`
 
 Type _string_.
-The URI or local path of the target Zarr dataset. Must be a directory.
+The URI or local path of the target Zarr dataset. Must specify a directory whose parent directory must exist.
 
 ## `target_storage_options`
 
@@ -147,13 +171,13 @@ Options for the filesystem given by the protocol of the URIs of contributing dat
 ## `slice_polling`
 
 Defines how to poll for contributing datasets.
-Must be one of the following.
+Must be one of the following:
 
   * No polling, fail immediately if dataset is not available.
-    Its value must be `false`.
+    Its value is `false`.
 
   * Poll using default values.
-    Its value must be `true`.
+    Its value is `true`.
 
   * Type _object_.
     Polling parameters.
@@ -169,6 +193,7 @@ Must be one of the following.
         Polling timeout in seconds.
         Defaults to `60`.
 
+
 ## `persist_mem_slices`
 
 Type _boolean_.
@@ -194,7 +219,7 @@ Defaults to `false`.
 ## `profiling`
 
 Profiling configuration. Allows for runtime profiling of the processing.
-Must be one of the following.
+Must be one of the following:
 
   * Type _boolean_.
     If set, profiling is enabled and output is logged using level `"INFO"`. Otherwise, profiling is disabled.
@@ -227,7 +252,7 @@ Must be one of the following.
       * `restrictions`:
         Type _array_.
         Used to limit the list down to the significant entries in the profiling report. Refer to [Stats.print_stats(*restrictions)](https://docs.python.org/3/library/profile.html#pstats.Stats.print_stats).
-        The items of the array must be one of the following.
+        The items of the array must be one of the following:
 
           * Type _integer_.
             Select a count of lines.
@@ -237,6 +262,8 @@ Must be one of the following.
 
           * Type _string_.
             Pattern-match the standard name that is printed.
+        
+
 
 ## `logging`
 
@@ -246,7 +273,7 @@ The key `version` is required.
 
   * `version`:
     Logging schema version.
-    Its value must be `1`.
+    Its value is `1`.
 
   * `formatters`:
     Type _object_.

docs/guide.md

@@ -16,7 +16,8 @@ zappend(os.listdir("inputs"), target_dir="output/mycube.zarr")
 
 Both invocations will create the Zarr dataset `output/mycube.zarr` by concatenating
 the "slice" datasets provided in the `inputs` directory along their `time` dimension. 
-Both the CLI command and the Python function can be run without any further 
+`target_dir` must specify a directory for the Zarr dataset. (Its parent directory must 
+exist.) Both the CLI command and the Python function can be run without any further 
 configuration provided the paths of the target dataset and the source slice datasets 
 are given. The target dataset path must point to a directory that will contain a Zarr 
 group to be created and updated. The slice dataset paths may be provided as Zarr as 
@@ -433,6 +434,7 @@ You can disable transaction management by specifying
 The `target_dir` setting is mandatory. If it is not specified in the configuration,
 it must be passed either as `--target` or `-t` option to the `zappend` command or as 
 `target_dir` keyword argument when using the `zappend` Python function.
+Note, the parent directory of `target_dir` must already exist.
 
 If the target path is given for another filesystem, additional storage options may be 
 passed using the optional `target_storage_options` setting. 

tests/test_api.py

@@ -85,6 +85,21 @@ def test_some_slices_local(self):
             for slice_dir in slices:
                 shutil.rmtree(slice_dir, ignore_errors=True)
 
+    def test_some_slices_local_output_to_non_existing_dir(self):
+        target_dir = "non_existent_dir/target.zarr"
+        slices = [
+            "slice-1.zarr",
+            "slice-2.zarr",
+            "slice-3.zarr",
+        ]
+        for uri in slices:
+            make_test_dataset(uri=uri)
+        with pytest.raises(
+            FileNotFoundError,
+            match="\\ATarget parent directory does not exist: .*/non_existent_dir\\Z",
+        ):
+            zappend(slices, target_dir=target_dir)
+
     def test_some_slices_with_class_slice_source(self):
         target_dir = "memory://target.zarr"
         slices = [make_test_dataset(index=3 * i) for i in range(3)]

zappend/config/markdown.py

@@ -44,9 +44,9 @@ def _schema_to_md(
     one_of = schema.get("anyOf") or schema.get("oneOf")
     if one_of:
         if sequence_name:
-            lines.append(f"The {sequence_name} must be one of the following.")
+            lines.append(f"The {sequence_name} must be one of the following:")
         else:
-            lines.append("Must be one of the following.")
+            lines.append("Must be one of the following:")
         for sub_schema in one_of:
             sub_lines = []
             _schema_to_md(sub_schema, path, sub_lines)
@@ -55,11 +55,12 @@ def _schema_to_md(
                 lines.append("  * " + sub_lines[0])
                 for sub_line in sub_lines[1:]:
                     lines.append("    " + sub_line)
+        lines.append("")
 
     const = schema.get("const", undefined)
     if const is not undefined:
         value = json.dumps(const)
-        lines.append(f"Its value must be `{value}`.")
+        lines.append(f"Its value is `{value}`.")
 
     default = schema.get("default", undefined)
     if default is not undefined:

zappend/config/schema.py

@@ -487,12 +487,15 @@
                 {"description": "Monotonically increasing.", "const": "+"},
                 {"description": "Monotonically decreasing.", "const": "-"},
                 {
-                    "description": "A time delta value.",
+                    "description": (
+                        "A positive or negative time delta value,"
+                        " such as `12h`, `2D`, `-1D`."
+                    ),
                     "type": "string",
                     "not": {"const": ""},
                 },
                 {
-                    "description": "A numerical delta value.",
+                    "description": "A positive or negative numerical delta value.",
                     "type": "number",
                     "not": {"const": 0},
                 },
@@ -520,7 +523,7 @@
         target_dir={
             "description": (
                 "The URI or local path of the target Zarr dataset."
-                " Must be a directory."
+                " Must specify a directory whose parent directory must exist."
             ),
             "type": "string",
             "minLength": 1,