Merge pull request #28 from bcdev/slice_source_in_config

Allow for passing custom slice sources via the configuration

Author: forman

CHANGES.md

@@ -1,7 +1,16 @@
 ## Version 0.2.1 (in development)
 
-* Using `sizes` instead of `dims` attribute of `xarray.Dataset` in implementation 
-  code. [#25] 
+* Allow for passing custom slice sources via the configuration.
+  The new configuration setting `slice_source` is the name of a class 
+  derived from `zappend.api.SliceSource` or a function that creates an instance 
+  of `zappend.api.SliceSource`. If `slice_source` is given, slices passed to 
+  the zappend function or CLI command will be interpreted as parameter(s) 
+  passed to the constructor of the specified class or the factory function. 
+  [#27]
+
+* Using `sizes` instead of `dims` attribute of `xarray.Dataset` in 
+  implementation code. [#25] 
+
 * Enhanced documentation including docstrings of several Python API objects.
 
 

docs/config.md

@@ -15,6 +15,11 @@ The URI or local path of the target Zarr dataset. Must be a directory.
 Type _object_.
 Options for the filesystem given by the URI of `target_dir`.
 
+### `slice_source`
+
+Type _string_.
+The fully qualified name of a class or function that provides a slice source for each slice item. If a class is given, it must be  derived from `zappend.api.SliceSource`. If a function is given, it must return an instance of  `zappend.api.SliceSource`. Refer to the user guide for more information.
+
 ### `slice_engine`
 
 Type _string_.

docs/guide.md

@@ -459,10 +459,53 @@ Or use default polling:
 
 ### Slice Sources
 
-Using the `zappend` command, slice dataset are provided as local filesystem paths 
-or by paths into other filesystems in case the slice datasets are provided by a URI.
-This section describes additional options to pass slice datasets to the `slices`
-argument of the [`zappend`](api.md) Python function.
+A _slice source_ is an object that provides a slice dataset of type `xarray.Dataset` 
+for given parameters of any type.
+
+The optional `slice_source` configuration setting is used to specify a custom 
+slice source. If not specified, `zappend` selects the slice source based on the type 
+of a given slice object. These types are described in following subsections.
+
+If given, the value of the `slice_source` setting is a class derived from
+`zappend.api.SliceSource`, or a function that creates an instance of 
+`zappend.api.SliceSource`, or the fully qualified name of the aforementioned. 
+In the case `slice_source` is given, the _slices_ argument passed to the CLI 
+command and Python function become parameters to the specified class constructor 
+or factory function.
+The individual slice items in the `SLICES` arguments of the `zappend` CLI 
+command are of type `str`, typically interpreted as file paths or URIs.
+The individual slice items passed in the `slices` argument of the
+`zappend.api.zappend()` function can be of any type, but the `tuple`, `list`, 
+and `dict` types have a special meaning:
+
+* `tuple`: a pair of the form `(args, kwargs)`, where `args` is a list
+  or tuple of positional arguments and `kwargs` is a dictionary of keyword
+  arguments;
+  * `list`: positional arguments only;
+* `dict`: keyword arguments only;
+* Any other type is interpreted as single positional argument.
+
+In addition, your class constructor or factory function specified by `slice_source` 
+may specify a positional or keyword argument named `ctx`, which will receive the 
+current processing context of type `zappend.api.Context`. 
+
+If the `slice_source` setting is _not_ specified, the slice items passed as `slices`
+argument to the [`zappend`](api.md) Python function can be one of the types described
+in the following subsections.
+
+#### `str` and `zappend.api.FileObj`
+
+A slice object of type `str` is interpreted as local file path or URI, in the case 
+the path has a protocol prefix, such as `s3://`.
+
+An alternative to providing the slice dataset as path or URI is using the `FileObj` 
+class, which combines a URI with dedicated filesystem storage options.
+
+```python
+from zappend.api import FileObj
+
+slice_obj = FileObj(slice_uri, storage_options=dict(...)) 
+```
 
 #### `xarray.Dataset`
 
@@ -504,19 +547,10 @@ at the cost of additional i/o. It therefore defaults to `false`.
 Often you want to perform some custom cleanup after a slice has been processed and
 appended to the target dataset. In this case you can write your own 
 `zappend.api.SliceSource` by implementing its `get_dataset()` and `dispose()`
-methods. Slice source instances are supposed to be created by _slice factories_,
-see below.
-
-#### `zappend.api.FileObj`
-
-An alternative to providing the slice dataset as path or URI is using the `FileObj` 
-class, which combines a URI with dedicated filesystem storage options.
+methods. 
 
-```python
-from zappend.api import FileObj
-
-slice_obj = FileObj(slice_uri, storage_options=dict(...)) 
-```
+Slice source instances are supposed to be created by _slice factories_, see 
+subsection below.
 
 #### `zappend.api.SliceFactory`
 
@@ -532,10 +566,8 @@ processed. Slice factories are created from the custom slice source and the slic
 using the utility function [to_slice_factories()][zappend.slice.factory.to_slice_factories]:
 
 ```python
-from typing import Iterable
 import numpy as np
 import xarray as xr
-from zappend.api import SliceFactory
 from zappend.api import SliceSource
 from zappend.api import to_slice_factories
 from zappend.api import zappend
@@ -578,6 +610,14 @@ zappend(to_slice_factories(MySliceSource, ["slice-1.nc", "slice-2.nc", "slice-3.
         target_dir="target.zarr")
 ```
 
+Note, the above example can be simplified by using the `slice_source` setting directly:
+
+```python
+zappend(["slice-1.nc", "slice-2.nc", "slice-3.nc"],
+        target_dir="target.zarr",
+        slice_source=MySliceSource)
+```
+
 ## Logging
 
 The `zappend` logging output is configured using the `logging` setting.

tests/test_config.py

@@ -236,6 +236,7 @@ def test_get_config_schema(self):
                 "persist_mem_slices",
                 "slice_engine",
                 "slice_polling",
+                "slice_source",
                 "slice_storage_options",
                 "target_storage_options",
                 "target_dir",

tests/test_context.py

@@ -4,11 +4,13 @@
 
 import unittest
 
+import pytest
 import xarray as xr
 from zappend.api import zappend
 from zappend.context import Context
 from zappend.fsutil.fileobj import FileObj
 from zappend.metadata import DatasetMetadata
+from zappend.slice import SliceSource
 from .helpers import clear_memory_fs
 from .helpers import make_test_dataset
 
@@ -73,3 +75,90 @@ def test_dry_run(self):
         self.assertEqual(False, ctx.dry_run)
         ctx = Context({"target_dir": "memory://target.zarr", "dry_run": True})
         self.assertEqual(True, ctx.dry_run)
+
+    def test_slice_source_as_name(self):
+        ctx = Context(
+            {
+                "target_dir": "memory://target.zarr",
+                "slice_source": "tests.test_context.new_custom_slice_source",
+            }
+        )
+        self.assertEqual(new_custom_slice_source, ctx.slice_source)
+
+        ctx = Context(
+            {
+                "target_dir": "memory://target.zarr",
+                "slice_source": "tests.test_context.CustomSliceSource",
+            }
+        )
+        self.assertEqual(CustomSliceSource, ctx.slice_source)
+
+        # staticmethod
+        ctx = Context(
+            {
+                "target_dir": "memory://target.zarr",
+                "slice_source": "tests.test_context.CustomSliceSource.new1",
+            }
+        )
+        self.assertEqual(CustomSliceSource.new1, ctx.slice_source)
+
+        # classmethod
+        ctx = Context(
+            {
+                "target_dir": "memory://target.zarr",
+                "slice_source": "tests.test_context.CustomSliceSource.new2",
+            }
+        )
+        self.assertEqual(CustomSliceSource.new2, ctx.slice_source)
+
+    def test_slice_source_as_type(self):
+        ctx = Context(
+            {
+                "target_dir": "memory://target.zarr",
+                "slice_source": new_custom_slice_source,
+            }
+        )
+        self.assertIs(new_custom_slice_source, ctx.slice_source)
+
+        ctx = Context(
+            {
+                "target_dir": "memory://target.zarr",
+                "slice_source": CustomSliceSource,
+            }
+        )
+        self.assertIs(CustomSliceSource, ctx.slice_source)
+
+        with pytest.raises(
+            TypeError,
+            match=(
+                "slice_source must a callable"
+                " or the fully qualified name of a callable"
+            ),
+        ):
+            Context(
+                {
+                    "target_dir": "memory://target.zarr",
+                    "slice_source": 11,
+                }
+            )
+
+
+def new_custom_slice_source(ctx: Context, index: int):
+    return CustomSliceSource(ctx, index)
+
+
+class CustomSliceSource(SliceSource):
+    def __init__(self, ctx: Context, index: int):
+        super().__init__(ctx)
+        self.index = index
+
+    def get_dataset(self) -> xr.Dataset:
+        return make_test_dataset(index=self.index)
+
+    @staticmethod
+    def new1(ctx: Context, index: int):
+        return CustomSliceSource(ctx, index)
+
+    @classmethod
+    def new2(cls, ctx: Context, index: int):
+        return cls(ctx, index)