Skip to content

Commit 9e69135

Browse files
committed
updated config schema and docs
1 parent 0083efa commit 9e69135

File tree

5 files changed

+55
-15
lines changed

5 files changed

+55
-15
lines changed

README.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -61,8 +61,8 @@ The `zappend` tool provides the following features:
6161
[`zappend`](cli.md) command or from Python. When used from Python using the
6262
[`zappend()`](api.md) function, slice datasets can be passed as local file
6363
paths, URIs, as datasets of type
64-
[xarray.Dataset](https://docs.xarray.dev/en/stable/generated/xarray.Dataset.html), or as custom
65-
[zappend.api.SliceSource](https://bcdev.github.io/zappend/api/#class-slicesource) objects.
64+
[xarray.Dataset](https://docs.xarray.dev/en/stable/generated/xarray.Dataset.html), or as custom
65+
[slice sources](https://bcdev.github.io/zappend/guide/#slice-sources).
6666

6767

6868
More about zappend can be found in its

docs/config.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -193,7 +193,7 @@ Options for the filesystem given by the URI of `target_dir`.
193193
## `slice_source`
194194

195195
Type _string_.
196-
The fully qualified name of a class or function that provides a slice source for each slice item. If a class is given, it must be derived from `zappend.api.SliceSource`. If a function is given, it must return an instance of `zappend.api.SliceSource`. Refer to the user guide for more information.
196+
The fully qualified name of a class or function that receives a slice item as argument(s) and provides the slice dataset. If a class is given, it must be derived from `zappend.api.SliceSource`. If the function is a context manager, it must yield an `xarray.Dataset`. If a plain function is given, it must return any valid slice item type. Refer to the user guide for more information.
197197

198198
## `slice_engine`
199199

docs/guide.md

Lines changed: 42 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -694,10 +694,14 @@ at the cost of additional i/o. It therefore defaults to `false`.
694694

695695
If you need some custom cleanup after a slice has been processed and appended to the
696696
target dataset, you can use instances of `zappend.api.SliceSource` as slice items.
697-
A `SliceSource` class requires you to implement two methods:
697+
The `SliceSource` methods with special meaning are:
698698

699-
* `get_dataset()` to return the slice dataset of type `xarray.Dataset`, and
700-
* `dispose()` to perform any resource cleanup tasks.
699+
* `get_dataset()`: a zero-argument method that returns the slice dataset of type
700+
`xarray.Dataset`. You must implement this abstract method.
701+
* `close()`: perform any resource cleanup tasks
702+
(in zappend < v0.7, the `close` methods was called `dispose`).
703+
* `__init__()`: optional constructor that receives any arguments passed to the
704+
slice source.
701705

702706
Here is the template code for your own slice source implementation:
703707

@@ -718,7 +722,7 @@ class MySliceSource(SliceSource):
718722
# You can put any processing here.
719723
return self.ds
720724

721-
def dispose(self):
725+
def close(self):
722726
# Write code here that performs cleanup.
723727
if self.ds is not None:
724728
self.ds.close()
@@ -795,7 +799,7 @@ class MySliceSource(SliceSource):
795799
self.ds = xr.open_dataset(self.slice_path)
796800
return get_mean_slice(self.ds)
797801

798-
def dispose(self):
802+
def close(self):
799803
if self.ds is not None:
800804
self.ds.close()
801805
self.ds = None
@@ -805,6 +809,39 @@ zappend(["slice-1.nc", "slice-2.nc", "slice-3.nc"],
805809
slice_source=MySliceSource)
806810
```
807811

812+
Since zappend 0.7, a slice source can also be written as a Python
813+
[context manager](https://docs.python.org/3/library/contextlib.html),
814+
which allows you implementing the `get_dataset()` and `close()`
815+
methods in one single function, instead of a class. Here is the above example
816+
written as context manager.
817+
818+
```python
819+
from contextlib import contextmanager
820+
import numpy as np
821+
import xarray as xr
822+
from zappend.api import zappend
823+
824+
# Same as above here
825+
826+
@contextmanager
827+
def get_slice_dataset(slice_path):
828+
# allocate resources here
829+
ds = xr.open_dataset(slice_path)
830+
mean_ds = get_mean_slice(ds)
831+
try:
832+
# yield (!) the slice dataset
833+
# so it can be appended
834+
yield mean_ds
835+
finally:
836+
# after slice dataset has been appended
837+
# release resources here
838+
ds.close()
839+
840+
zappend(["slice-1.nc", "slice-2.nc", "slice-3.nc"],
841+
target_dir="target.zarr",
842+
slice_source=get_slice_dataset)
843+
```
844+
808845
## Profiling
809846

810847
Runtime profiling is very important for understanding program runtime behavior

docs/index.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -39,8 +39,8 @@ The `zappend` tool provides the following features:
3939
[`zappend`](cli.md) command or from Python. When used from Python using the
4040
[`zappend()`](api.md) function, slice datasets can be passed as local file
4141
paths, URIs, as datasets of type
42-
[xarray.Dataset](https://docs.xarray.dev/en/stable/generated/xarray.Dataset.html), or as custom
43-
[zappend.api.SliceSource](https://bcdev.github.io/zappend/api/#class-slicesource) objects.
42+
[xarray.Dataset](https://docs.xarray.dev/en/stable/generated/xarray.Dataset.html), or as custom
43+
[slice sources](guide#slice-sources).
4444

4545
## How It Works
4646

zappend/config/schema.py

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -630,11 +630,14 @@
630630
},
631631
slice_source={
632632
"description": (
633-
"The fully qualified name of a class or function that provides"
634-
" a slice source for each slice item. If a class is given, it must be "
635-
" derived from `zappend.api.SliceSource`."
636-
" If a function is given, it must return an instance of "
637-
" `zappend.api.SliceSource`."
633+
"The fully qualified name of a class or function that receives a"
634+
" slice item as argument(s) and provides the slice dataset."
635+
" If a class is given,"
636+
" it must be derived from `zappend.api.SliceSource`."
637+
" If the function is a context manager,"
638+
" it must yield an `xarray.Dataset`."
639+
" If a plain function is given,"
640+
" it must return any valid slice item type."
638641
" Refer to the user guide for more information."
639642
),
640643
"type": "string",

0 commit comments

Comments
 (0)