Added logging configuration

Author: forman

CONFIG.md

@@ -1,8 +1,9 @@
-Configuration for the zappend tool.
+## Configuration for the `zappend` tool
+
 
 ### `version`
 
-Configuration version.
+Configuration schema version. Allows the schema to evolve while still preserving backwards compatibility.
 It's value is `1`.
 
 ### `target_uri`
@@ -35,11 +36,12 @@ Must be one of the following:
   It's value is `true`.
 * Type _object_.
   Polling parameters.
+  The keys `interval`, `timeout` are required.
+  
   * `interval`:
     Type _number_.
     Polling interval in seconds.
     Defaults to `2`.
-  
   * `timeout`:
     Type _number_.
     Polling timeout in seconds.
@@ -66,7 +68,6 @@ It's value is `2`.
 Type _object_.
 Specifies the fixed dimensions of the target dataset. Keys are dimension names, values are dimension sizes.
 Object values are:
-
 Type _integer_.
 
 ### `append_dim`
@@ -80,28 +81,26 @@ Defaults to `"time"`.
 Type _object_.
 Defines dimensions, encoding, and attributes for variables in the target dataset. Object property names refer to variable names. The special name `*` refers to all variables, which is useful for defining common values.
 Object values are:
-
 Type _object_.
-Variable metadata
+Variable metadata.
+
 * `dims`:
   Type _array_.
   The names of the variable's dimensions in the given order. Each dimension must exist in contributing datasets.
-
 * `encoding`:
   Type _object_.
   Variable storage encoding. Settings given here overwrite the encoding settings of the first contributing dataset.
+  
   * `dtype`:
     Storage data type
     Must be one of `"int8", "uint8", "int16", "uint16", "int32", "uint32", "int64", "uint64", "float32", "float64"`.
-  
   * `chunks`:
     Storage chunking.
     Must be one of the following:
     * Type _array_.
       Chunk sizes in the order of the dimensions.
     * Disable chunking.
       It's value is `null`.
-  
   * `fill_value`:
     Storage fill value.
     Must be one of the following:
@@ -111,36 +110,30 @@ Variable metadata
       It's value is `"NaN"`.
     * No fill value.
       It's value is `null`.
-  
   * `scale_factor`:
     Type _number_.
     Scale factor for computing the in-memory value: `memory_value = scale_factor * storage_value + add_offset`.
-  
   * `add_offset`:
     Type _number_.
     Add offset for computing the in-memory value: `memory_value = scale_factor * storage_value + add_offset`.
-  
   * `units`:
     Type _string_.
     Units of the storage data type if memory data type is date/time.
-  
   * `calendar`:
     Type _string_.
     The calendar to be used if memory data type is date/time.
-  
   * `compressor`:
     Type _array_ | _null_.
     Compressor. Set to `null` to disable data compression.
+    The key `id` is required.
+    
     * `id`:
       Type _string_.
 
-    $`$id` are required.
-  
   * `filters`:
     Type _array_ | _null_.
     Filters. Set to `null` to not use filters.
 
-
 * `attrs`:
   Type _object_.
   Arbitrary variable metadata attributes.
@@ -160,4 +153,83 @@ Specifies the names of individual variables to be excluded  from all contributin
 
 Type _boolean_.
 If 'true', log only what would have been done, but don't apply any changes.
-Defaults to `false`.
+Defaults to `false`.
+
+### `logging`
+
+Type _object_.
+Logging configuration. For details refer to the [dictionary schema](https://docs.python.org/3/library/logging.config.html#logging-config-dictschema) of the Python module `logging.config`.
+The key `version` is required.
+
+* `version`:
+  Logging schema version.
+  It's value is `1`.
+* `formatters`:
+  Type _object_.
+  Formatter definitions. Each key is a formatter id and each value is an object describing how to configure the corresponding formatter.
+  Object values are:
+  Type _object_.
+  Formatter configuration.
+  
+  * `format`:
+    Type _string_.
+    Format string in the given `style`.
+    Defaults to `"%(message)s"`.
+  * `datefmt`:
+    Type _string_.
+    Format string in the given `style` for the date/time portion.
+    Defaults to `"%Y-%m-%d %H:%M:%S,uuu"`.
+  * `style`:
+    Determines how the format string will be merged with its data.
+    Must be one of `"%", "{", "$"`.
+  
+* `filters`:
+  Type _object_.
+  Filter definitions. Each key is a filter id and each value is a dict describing how to configure the corresponding filter.
+  Object values are:
+  Type _object_.
+  Filter configuration.
+* `handlers`:
+  Type _object_.
+  Handler definitions. Each key is a handler id and each value is an object describing how to configure the corresponding handler.
+  Object values are:
+  Type _object_.
+  Handler configuration. All keys other than the following are passed through as keyword arguments to the handler's constructor. 
+  The key `class` is required.
+  
+  * `class`:
+    Type _string_.
+    The fully qualified name of the handler class. See [logging handlers](https://docs.python.org/3/library/logging.handlers.html).
+  * `level`:
+    The level of the handler.
+    Must be one of `"CRITICAL", "ERROR", "WARNING", "INFO", "DEBUG", "NOTSET"`.
+  * `formatter `:
+    Type _string_.
+    The id of the formatter for this handler.
+  * `filters`:
+    Type _array_.
+    A list of ids of the filters for this logger.
+  
+* `loggers`:
+  Type _object_.
+  Logger definitions. Each key is a logger name and each value is an object describing how to configure the corresponding logger. The tool's logger has the id `'zappend'`.
+  Object values are:
+  Type _object_.
+  Logger configuration.
+  
+  * `level`:
+    The level of the logger.
+    Must be one of `"CRITICAL", "ERROR", "WARNING", "INFO", "DEBUG", "NOTSET"`.
+  * `propagate `:
+    Type _boolean_.
+    The propagation setting of the logger.
+  * `filters`:
+    Type _array_.
+    A list of ids of the filters for this logger.
+  * `handlers`:
+    Type _array_.
+    A list of ids of the handlers for this logger.
+  
+
+
+

tests/test_cli.py

@@ -17,7 +17,8 @@
     
     Options:
       -c, --config CONFIG    Configuration JSON or YAML file. If multiple are
-                             passed, they will be deeply merged into one.
+                             passed, subsequent configurations are incremental to
+                             the previous ones.
       -t, --target TARGET    Target Zarr dataset path or URI. Overrides the
                              'target_uri' configuration field.
       --dry-run              Run the tool without creating, changing, or deleting

tests/test_config.py

@@ -213,6 +213,7 @@ def test_schema(self):
                 'excluded_variables',
                 'fixed_dims',
                 'included_variables',
+                'logging',
                 'slice_engine',
                 'slice_polling',
                 'slice_storage_options',

tests/test_log.py

@@ -0,0 +1,54 @@
+# Copyright © 2024 Norman Fomferra
+# Permissions are hereby granted under the terms of the MIT License:
+# https://opensource.org/licenses/MIT.
+import logging
+import unittest
+from click.testing import CliRunner
+
+from zappend.log import logger
+from zappend.log import configure_logging
+
+
+class LogTest(unittest.TestCase):
+
+    def test_logger(self):
+        self.assertIsInstance(logger, logging.Logger)
+        self.assertEqual(logging.NOTSET, logger.level)
+        self.assertEqual(1, len(logger.handlers))
+
+    def test_configure_logging(self):
+        logging_config = {
+            "version": 1,
+            "formatters": {
+                "normal": {
+                    "format": "%(asctime)s %(levelname)s %(message)s",
+                    "style": "%"
+                }
+            },
+            "handlers": {
+                "console": {
+                    "class": "logging.StreamHandler",
+                    "formatter": "normal"
+                }
+            },
+            "loggers": {
+                "zappend": {
+                    "level": "INFO",
+                    "handlers": ["console"]
+                }
+            }
+        }
+
+        old_level = logger.level
+        old_handlers = list(logger.handlers)
+        for h in old_handlers:
+            logger.removeHandler(h)
+
+        try:
+            configure_logging(logging_config)
+            self.assertEqual(logging.INFO, logger.level)
+            self.assertEqual(1, len(logger.handlers))
+        finally:
+            logger.setLevel(old_level)
+            for h in old_handlers:
+                logger.addHandler(h)

zappend/cli.py

@@ -11,8 +11,8 @@
               metavar="CONFIG",
               multiple=True,
               help="Configuration JSON or YAML file."
-                   " If multiple are passed,"
-                   " they will be deeply merged into one.")
+                   " If multiple are passed, subsequent configurations"
+                   " are incremental to the previous ones.")
 @click.option("--target", "-t",
               metavar="TARGET",
               help="Target Zarr dataset path or URI."