Add support for TOML config files (#660)

Author: mauvilsa

.pre-commit-config.yaml

@@ -56,6 +56,7 @@ repos:
       [
         types-PyYAML,
         types-requests,
+        types-toml,
       ]
     verbose: true
 

CHANGELOG.rst

@@ -20,6 +20,8 @@ Added
 - Support without ``pyyaml``, though only an internal refactor prior to eventual
   removal of ``pyyaml`` as a required dependency in v5.0.0 (`#652
   <https://github.com/omni-us/jsonargparse/pull/652>`__).
+- Support for ``toml`` as config file format (`#660
+  <https://github.com/omni-us/jsonargparse/pull/660>`__).
 
 Changed
 ^^^^^^^

DOCUMENTATION.rst

@@ -190,9 +190,9 @@ All tools implemented with the :func:`.auto_cli` function have the ``--config``
 option to provide settings in a config file (more details in
 :ref:`configuration-files`). This becomes very useful when the number of
 configurable parameters is large. To ease the writing of config files, there is
-also the option ``--print_config`` which prints to standard output a yaml with
-all settings that the tool supports with their default values. Users of the tool
-can be advised to follow the following steps:
+also the option ``--print_config`` which prints to standard output all settings
+that the tool supports with their default values. Users of the tool can be
+advised to follow the following steps:
 
 .. code-block:: bash
 
@@ -1246,27 +1246,30 @@ is useful for example for class instantiation.
 Configuration files
 ===================
 
-An important feature of jsonargparse is the parsing of yaml/json files. The dot
-notation hierarchy of the arguments (see :ref:`nested-namespaces`) are used for
-the expected structure in the config files.
+An important feature of jsonargparse is its ability to parse configuration
+files. The dot notation hierarchy of the arguments (see
+:ref:`nested-namespaces`) defines the expected structure in these files. By
+default, the configuration format is ``yaml``. To change the format, use the
+``parser_mode`` parameter when instantiating the parser, e.g.,
+``ArgumentParser(parser_mode="toml")``.
 
 The :py:attr:`.ArgumentParser.default_config_files` property can be set when
-creating a parser to specify patterns to search for configuration files. For
-example if a parser is created as
+creating a parser to specify patterns for searching configuration files. For
+example, if a parser is created as
 ``ArgumentParser(default_config_files=['~/.myapp.yaml', '/etc/myapp.yaml'])``,
-when parsing if any of those two config files exist it will be parsed and used
-to override the defaults. All matched config files are parsed and applied in the
-given order. The default config files are always parsed first, this means that
-any command line argument will override its values.
-
-It is also possible to add an argument to explicitly provide a configuration
-file path. Providing a config file as an argument does not disable the parsing
-of ``default_config_files``. The config argument would be parsed in the specific
-position among the command line arguments. Therefore the arguments found after
-would override the values from that config file. The config argument can be
-given multiple times, each overriding the values of the previous. Using the
-example parser from the :ref:`nested-namespaces` section above, we could have
-the following config file in yaml format:
+it will search for and parse any of these files if they exist, using them to
+override the defaults. All matched configuration files are parsed and applied in
+the given order. The default configuration files are always parsed first,
+meaning any command line argument will override their values.
+
+You can also add an argument to explicitly provide a configuration file path.
+Providing a configuration file as an argument does not disable the parsing of
+``default_config_files``. The configuration argument will be parsed in the
+specific position among the command line arguments, so arguments found afterward
+will override the values from that configuration file. The configuration
+argument can be given multiple times, each instance overriding the values of the
+previous one. Using the example parser from the :ref:`nested-namespaces` section
+above, we could have the following configuration file in yaml format:
 
 .. code-block:: yaml
 
@@ -1336,10 +1339,10 @@ yaml comments by using ``--print_config=comments``. Another option is
 
 From within python it is also possible to serialize a config object by using
 either the :py:meth:`.ArgumentParser.dump` or :py:meth:`.ArgumentParser.save`
-methods. Three formats with a particular style are supported: ``yaml``, ``json``
-and ``json_indented``. It is possible to add more dumping formats by using the
-:func:`.set_dumper` function. For example to allow dumping using PyYAML's
-``default_flow_style`` do the following:
+methods. Several formats with a particular style are supported: ``yaml``,
+``toml``, ``json_compact`` and ``json_indented``. It is possible to add more
+dumping formats by using the :func:`.set_dumper` function. For example to allow
+dumping using PyYAML's ``default_flow_style`` do the following:
 
 .. testcode::
 

README.rst

@@ -80,8 +80,8 @@ Other notable features include:
   hierarchies.
 
 - **Config file formats:** `json <https://www.json.org/>`__, `yaml
-  <https://yaml.org/>`__, `jsonnet <https://jsonnet.org/>`__ and extendable to
-  more formats.
+  <https://yaml.org/>`__, `toml <https://toml.io/>`__, `jsonnet
+  <https://jsonnet.org/>`__ and extendable to more formats.
 
 - **Relative paths:** within config files and parsing of config paths referenced
   inside other configs.
@@ -134,10 +134,10 @@ You can install using `pip <https://pypi.org/project/jsonargparse/>`__ as:
 By default the only dependency that jsonargparse installs is `PyYAML
 <https://pypi.org/project/PyYAML/>`__. However, several optional features can be
 enabled by specifying any of the following extras requires: ``signatures``,
-``jsonschema``, ``jsonnet``, ``urls``, ``fsspec``, ``ruyaml``, ``omegaconf``,
-``shtab`` and ``argcomplete``. There is also the ``all`` extras require to
-enable all optional features (excluding tab completion ones). Installing
-jsonargparse with extras require is as follows:
+``jsonschema``, ``jsonnet``, ``urls``, ``fsspec``, ``toml``, ``ruyaml``,
+``omegaconf``, ``shtab`` and ``argcomplete``. There is also the ``all`` extras
+require to enable all optional features (excluding tab completion ones).
+Installing jsonargparse with extras require is as follows:
 
 .. code-block:: bash
 

jsonargparse/_core.py

@@ -682,8 +682,6 @@ def _load_config_parser_mode(
             cfg_dict = load_value(cfg_str, path=cfg_path, ext_vars=ext_vars)
         except get_loader_exceptions() as ex:
             raise TypeError(f"Problems parsing config: {ex}") from ex
-        if cfg_dict is None:
-            return Namespace()
         if key and isinstance(cfg_dict, dict):
             cfg_dict = cfg_dict.get(key, {})
         if not isinstance(cfg_dict, dict):
@@ -994,6 +992,9 @@ def get_defaults(self, skip_validation: bool = False, **kwargs) -> Namespace:
 
         default_config_files = self._get_default_config_files()
         for key, default_config_file in default_config_files:
+            default_config_file_content = default_config_file.get_content()
+            if not default_config_file_content.strip():
+                continue
             with change_to_path_dir(default_config_file), parser_context(parent_parser=self):
                 cfg_file = self._load_config_parser_mode(default_config_file.get_content(), key=key)
                 cfg = self.merge_config(cfg_file, cfg)

jsonargparse/_loaders_dumpers.py

@@ -2,10 +2,11 @@
 
 import inspect
 import re
-from typing import Any, Callable, Dict, Optional, Tuple, Type
+from contextlib import suppress
+from typing import Any, Callable, Dict, Optional, Set, Tuple, Type
 
 from ._common import load_value_mode, parent_parser
-from ._optionals import import_jsonnet, omegaconf_support, pyyaml_available
+from ._optionals import import_jsonnet, import_toml_dumps, import_toml_loads, omegaconf_support, pyyaml_available
 from ._type_checking import ArgumentParser
 
 __all__ = [
@@ -15,9 +16,25 @@
 ]
 
 
+not_loaded = object()
 yaml_default_loader = None
 
 
+def load_basic(value):
+    value = value.strip()
+    if value == "true":
+        return True
+    if value == "false":
+        return False
+    if value == "null":
+        return None
+    if value.isdigit() or (value.startswith("-") and value[1:].isdigit()):
+        return int(value)
+    if value.replace(".", "", 1).replace("e", "", 1).replace("-", "", 2).isdigit() and ("e" in value or "." in value):
+        return float(value)
+    return not_loaded
+
+
 def get_yaml_default_loader():
     global yaml_default_loader
     if yaml_default_loader:
@@ -57,7 +74,7 @@ def remove_implicit_resolver(cls, tag_to_remove):
     )
 
     yaml_default_loader = DefaultLoader
-    return DefaultLoader
+    return yaml_default_loader
 
 
 def yaml_load(stream):
@@ -74,13 +91,15 @@ def yaml_load(stream):
     return value
 
 
-def json_load(stream):
+def json_load(value):
     import json
 
-    try:
-        return json.loads(stream)
-    except json.JSONDecodeError:
-        return stream
+    return json.loads(value)
+
+
+def toml_load(value):
+    toml_loads, _ = import_toml_loads("toml_load")
+    return toml_loads(value)
 
 
 def jsonnet_load(stream, path="", ext_vars=None):
@@ -101,10 +120,15 @@ def jsonnet_load(stream, path="", ext_vars=None):
 loaders: Dict[str, Callable] = {
     "yaml": yaml_load,
     "json": json_load,
-    "jsonnet": jsonnet_load,
+    "toml": toml_load,
 }
-
 loader_exceptions: Dict[str, Tuple[Type[Exception], ...]] = {}
+loader_json_superset: Dict[str, bool] = {
+    "yaml": True,
+    "json": True,
+    "toml": False,
+}
+loader_params: Dict[str, Set[str]] = {}
 
 
 def get_load_value_mode() -> str:
@@ -123,28 +147,56 @@ def get_loader_exceptions(mode: Optional[str] = None) -> Tuple[Type[Exception],
         if mode == "yaml":
             loader_exceptions[mode] = (__import__("yaml").YAMLError,)
         elif mode == "json":
-            loader_exceptions[mode] = tuple()
+            loader_exceptions[mode] = (__import__("json").JSONDecodeError,)
+        elif mode == "toml":
+            loader_exceptions[mode] = (import_toml_loads("get_loader_exceptions")[1],)
         elif mode == "jsonnet":
-            return get_loader_exceptions("yaml" if pyyaml_available else "json")
+            return get_loader_exceptions("yaml" if pyyaml_available else "json") + (ValueError,)
     return loader_exceptions[mode]
 
 
-json_or_yaml_load = yaml_load if pyyaml_available else json_load
+def json_or_yaml_load(value):
+    if pyyaml_available:
+        if isinstance(value, str) and value.strip() == "":
+            return value
+        return yaml_load(value)
+    return json_load(value)
+
+
 json_or_yaml_loader_exceptions = get_loader_exceptions("yaml" if pyyaml_available else "json")
 
 
+def load_list_or_dict(value: str):
+    strip = value.strip()
+    if (strip.startswith("[") and strip.endswith("]")) or (strip.startswith("{") and strip.endswith("}")):
+        import json
+
+        with suppress(json.JSONDecodeError):
+            return json.loads(strip)
+    return not_loaded
+
+
 def load_value(value: str, simple_types: bool = False, **kwargs):
-    if not value:
-        return None
-    elif value.strip() == "-":
+    if value.strip() == "-":
         return value
-    loader = loaders[get_load_value_mode()]
-    if kwargs:
-        params = set(list(inspect.signature(loader).parameters)[1:])
-        kwargs = {k: v for k, v in kwargs.items() if k in params}
-    loaded_value = loader(value, **kwargs)
+
+    loaded_value = load_basic(value)
+
+    mode = get_load_value_mode()
+    if loaded_value is not_loaded and not loader_json_superset[mode]:
+        loaded_value = load_list_or_dict(value)
+
+    if loaded_value is not_loaded:
+        loader = loaders[mode]
+        load_kwargs = {}
+        if kwargs and mode in loader_params:
+            params = loader_params[mode]
+            load_kwargs = {k: v for k, v in kwargs.items() if k in params}
+        loaded_value = loader(value, **load_kwargs)
+
     if not simple_types and isinstance(loaded_value, (int, float, bool, str)):
         loaded_value = value
+
     return loaded_value
 
 
@@ -172,7 +224,7 @@ def yaml_comments_dump(data, parser):
     return formatter.add_yaml_comments(dump)
 
 
-def json_dump(data):
+def json_compact_dump(data):
     import json
 
     return json.dumps(data, separators=(",", ":"), **dump_json_kwargs)
@@ -184,18 +236,26 @@ def json_indented_dump(data):
     return json.dumps(data, indent=2, **dump_json_kwargs) + "\n"
 
 
+def toml_dump(data):
+    toml_dumps = import_toml_dumps("toml_dump")
+    return toml_dumps(data)
+
+
 dumpers: Dict[str, Callable] = {
     "yaml": yaml_dump,
     "yaml_comments": yaml_comments_dump,
-    "json": json_dump,
+    "json": json_compact_dump,
+    "json_compact": json_compact_dump,
     "json_indented": json_indented_dump,
+    "toml": toml_dump,
     "jsonnet": json_indented_dump,
 }
 
 comment_prefix: Dict[str, str] = {
     "yaml": "# ",
     "yaml_comments": "# ",
     "jsonnet": "// ",
+    "toml": "# ",
 }
 
 
@@ -220,6 +280,7 @@ def set_loader(
     mode: str,
     loader_fn: Callable[[str], Any],
     exceptions: Tuple[Type[Exception], ...] = tuple(),
+    json_superset: bool = True,
 ):
     """Sets the value loader function to be used when parsing with a certain mode.
 
@@ -234,9 +295,14 @@ def set_loader(
         loader_fn: The loader function to set. Example: ``yaml.safe_load``.
         exceptions: Exceptions that the loader can raise when load fails.
             Example: (yaml.YAMLError,).
+        json_superset: Whether the loader can load JSON data.
     """
     loaders[mode] = loader_fn
     loader_exceptions[mode] = exceptions
+    loader_json_superset[mode] = json_superset
+    params = set(list(inspect.signature(loader_fn).parameters)[1:])
+    if params:
+        loader_params[mode] = params
 
 
 def get_loader(mode: str):
@@ -259,3 +325,6 @@ def set_omegaconf_loader():
         from ._optionals import get_omegaconf_loader
 
         set_loader("omegaconf", get_omegaconf_loader())
+
+
+set_loader("jsonnet", jsonnet_load, get_loader_exceptions("jsonnet"))

jsonargparse/_optionals.py

@@ -15,6 +15,8 @@
 
 
 pyyaml_available = bool(find_spec("yaml"))
+toml_load_available = bool(find_spec("toml") or find_spec("tomllib"))
+toml_dump_available = bool(find_spec("toml"))
 typing_extensions_support = find_spec("typing_extensions") is not None
 typeshed_client_support = find_spec("typeshed_client") is not None
 jsonschema_support = find_spec("jsonschema") is not None
@@ -103,6 +105,24 @@ def missing_package_raise(package, importer):
         raise ImportError(f"{package} package is required by {importer} :: {ex}") from ex
 
 
+def import_toml_loads(importer):
+    if find_spec("tomllib"):
+        import tomllib
+
+        return tomllib.loads, tomllib.TOMLDecodeError
+    else:
+        with missing_package_raise("toml", importer):
+            import toml
+
+        return toml.loads, toml.TomlDecodeError
+
+
+def import_toml_dumps(importer):
+    with missing_package_raise("toml", importer):
+        import toml
+    return toml.dumps
+
+
 def import_jsonschema(importer):
     with missing_package_raise("jsonschema", importer):
         import jsonschema