feature: support JSON config as well as YAML (#1490)

Author: jmartin-tech

docs/source/_config.rst

@@ -15,8 +15,8 @@ Config values are loaded in the following priority (lowest-first):
 
 * Plugin defaults in the code
 * Core config: from ``garak/resources/garak.core.yaml``; not to be overridden
-* Site config: from ``$HOME/.config/garak/garak.site.yaml``
-* Runtime config: from an optional config file specified manually, via e.g. CLI parameter
+* Site config: from ``$HOME/.config/garak/garak.site.yaml`` or ``garak.site.json``
+* Runtime config: from an optional config file (YAML or JSON) specified manually, via e.g. CLI parameter
 * Command-line options
 
 

docs/source/configurable.rst

@@ -14,10 +14,10 @@ Specifying Custom Configuration
 garak can be configured in multiple ways:
 
 * Via command-line parameters
-* Using YAML configs
+* Using YAML or JSON config files
 * Through specifying JSON on the command line
 
-The easiest way is often to use a YAML config, and how to do that is
+The easiest way is often to use a config file (YAML or JSON), and how to do that is
 described below.
 
 Garak Config Hierarchy
@@ -28,13 +28,13 @@ Configuration values can come from multiple places. At garak load, the
 the priority of which values go where. The hierarchy is as follows:
 
 1. Values given at the command line
-2. Config values given in a YAML file passed via ``--config``
-3. Values in a YAML site config, ``garak.site.yaml``, placed in the config directory (``XDG_CONFIG_DIR``, which is ``~/.config/garak/`` on Linux; see XDG spec for details)
+2. Config values given in a YAML or JSON file passed via ``--config``
+3. Values in a YAML or JSON site config, ``garak.site.yaml``, ``garak.site.yml``, or ``garak.site.json``, placed in the config directory (``XDG_CONFIG_DIR``, which is ``~/.config/garak/`` on Linux; see XDG spec for details)
 4. Fixed values kept in the garak core config - don't edit this. Package updates will overwrite it, and you might break your garak install. It's in ``garak/resources`` if you want to take a look.
 5. Default values specified in plugin code
 
-Config YAML
-^^^^^^^^^^^
+Config Files (YAML and JSON)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Let's take a look at the core config.
 
@@ -169,7 +169,12 @@ Bundled Quick Configs
 ^^^^^^^^^^^^^^^^^^^^^
 
 Garak comes bundled with some quick configs that can be loaded directly using ``--config``.
-These don't need the ``.yml`` extension when being requested. They include:
+
+**Note on extensions:** JSON configs can be loaded without the ``.json`` extension (e.g., ``--config fast``).
+YAML configs require the explicit ``.yaml`` or ``.yml`` extension (e.g., ``--config fast.yaml`` or ``--config fast.yml``).
+Extensions are case-insensitive, so ``.JSON``, ``.YAML``, and ``.YML`` are also accepted.
+
+Bundled configs include:
 
 * ``broad`` - Run all active probes, just once each, for a rapid broad test
 * ``fast`` - Go through a selection of light probes; skip extended detectors
@@ -178,17 +183,19 @@ These don't need the ``.yml`` extension when being requested. They include:
 * ``notox`` - Scan without any toxicity-inducing probes
 * ``tox_and_buffs`` - Go through toxicity & slur probes, using only relevant payloads, and a fast paraphraser
 
-These are great places to look at to get an idea of how garak YAML configs can look.
+These are great places to look at to get an idea of how garak configs can look.
 Quick configs are stored under ``garak/configs/`` in the source code/install.
 
 
 Using a Custom Config
 ^^^^^^^^^^^^^^^^^^^^^
 
-To override values in this we can create a new YAML file and point to it from the
+To override values in this we can create a new config file (YAML or JSON) and point to it from the
 command line using ``--config``. For example, to select just ``latentinjection``
 probes and run each prompt just once:
 
+**YAML format:**
+
 .. code-block:: yaml
 
     ---
@@ -199,6 +206,23 @@ probes and run each prompt just once:
         probe_spec: latentinjection
 
 If we save this as ``latent1.yaml`` somewhere, then we can use it with ``garak --config latent1.yaml``.
+Note: YAML configs require the explicit ``.yaml`` or ``.yml`` extension (case-insensitive).
+
+**JSON format:**
+
+.. code-block:: json
+
+    {
+      "run": {
+        "generations": 1
+      },
+      "plugins": {
+        "probe_spec": "latentinjection"
+      }
+    }
+
+If we save this as ``latent1.json`` somewhere, then we can use it with ``garak --config latent1.json``
+or without the extension: ``garak --config latent1``.
 
 Using a Custom JSON Config
 ^^^^^^^^^^^^^^^^^^^^^^^^^^

garak/_config.py

@@ -11,6 +11,7 @@
 from collections import defaultdict
 from dataclasses import dataclass
 import importlib
+import json
 import logging
 import os
 import stat
@@ -155,13 +156,22 @@ def _combine_into(d: dict, combined: dict) -> dict:
     return combined
 
 
-def _load_yaml_config(settings_filenames) -> dict:
+def _load_config_files(settings_filenames) -> dict:
     global config_files
     config_files += settings_filenames
     config = nested_dict()
     for settings_filename in settings_filenames:
         with open(settings_filename, encoding="utf-8") as settings_file:
-            settings = yaml.safe_load(settings_file)
+            try:
+                settings = json.load(settings_file)
+            except json.JSONDecodeError:
+                settings_file.seek(0)
+                try:
+                    settings = yaml.safe_load(settings_file)
+                except yaml.YAMLError as e:
+                    message = f"Failed to parse config file {settings_filename} as JSON or YAML: {e}"
+                    logging.error(message)
+                    raise ValueError(message) from e
             if settings is not None:
                 if _key_exists(settings, "api_key"):
                     if platform.system() == "Windows":
@@ -216,7 +226,7 @@ def _load_yaml_config(settings_filenames) -> dict:
 
 def _store_config(settings_files) -> None:
     global system, run, plugins, reporting, version
-    settings = _load_yaml_config(settings_files)
+    settings = _load_config_files(settings_files)
     system = _set_settings(system, settings["system"])
     run = _set_settings(run, settings["run"])
     run.user_agent = run.user_agent.replace("{version}", version)
@@ -292,27 +302,99 @@ def load_config(
 
     settings_files = [str(transient.package_dir / "resources" / "garak.core.yaml")]
 
-    fq_site_config_filename = str(transient.config_dir / site_config_filename)
-    if os.path.isfile(fq_site_config_filename):
-        settings_files.append(fq_site_config_filename)
+    if site_config_filename == "garak.site.yaml":
+        site_config_json = str(transient.config_dir / "garak.site.json")
+        site_config_yaml = str(transient.config_dir / "garak.site.yaml")
+        site_config_yml = str(transient.config_dir / "garak.site.yml")
+
+        has_json = os.path.isfile(site_config_json)
+        has_yaml = os.path.isfile(site_config_yaml)
+        has_yml = os.path.isfile(site_config_yml)
+
+        if sum([has_json, has_yaml, has_yml]) > 1:
+            message = "Multiple site config files found (garak.site.json, garak.site.yaml, garak.site.yml). Please use only one site config format."
+            logging.error(message)
+            raise ValueError(message)
+        elif has_json:
+            settings_files.append(site_config_json)
+        elif has_yaml:
+            settings_files.append(site_config_yaml)
+        elif has_yml:
+            settings_files.append(site_config_yml)
+        else:
+            logging.debug(
+                "no site config found at: %s, %s, or %s",
+                site_config_json,
+                site_config_yaml,
+                site_config_yml,
+            )
     else:
-        # warning, not error, because this one has a default value
-        logging.debug("no site config found at: %s", fq_site_config_filename)
+        fq_site_config_filename = str(transient.config_dir / site_config_filename)
+        if os.path.isfile(fq_site_config_filename):
+            settings_files.append(fq_site_config_filename)
+        else:
+            logging.debug("no site config found at: %s", fq_site_config_filename)
 
     if run_config_filename is not None:
-        # take config file path as provided
+        # If file exists as-is, use it
         if os.path.isfile(run_config_filename):
             settings_files.append(run_config_filename)
-        elif os.path.isfile(
-            str(transient.package_dir / "configs" / (run_config_filename + ".yaml"))
-        ):
-            settings_files.append(
-                str(transient.package_dir / "configs" / (run_config_filename + ".yaml"))
-            )
+        # If explicit extension, check bundled
+        elif run_config_filename.lower().endswith((".json", ".yaml", ".yml")):
+            bundled = str(transient.package_dir / "configs" / run_config_filename)
+            if os.path.isfile(bundled):
+                settings_files.append(bundled)
+            else:
+                message = f"run config not found: {run_config_filename}"
+                logging.error(message)
+                raise FileNotFoundError(message)
+        # Extension-less: JSON-only, YAML needs explicit .yaml/.yml
         else:
-            message = f"run config not found: {run_config_filename}"
-            logging.error(message)
-            raise FileNotFoundError(message)
+            json_path = run_config_filename + ".json"
+            yaml_path = run_config_filename + ".yaml"
+            yml_path = run_config_filename + ".yml"
+            json_bundled = str(
+                transient.package_dir / "configs" / (run_config_filename + ".json")
+            )
+            yaml_bundled = str(
+                transient.package_dir / "configs" / (run_config_filename + ".yaml")
+            )
+            yml_bundled = str(
+                transient.package_dir / "configs" / (run_config_filename + ".yml")
+            )
+
+            has_json = os.path.isfile(json_path)
+            has_yaml = os.path.isfile(yaml_path)
+            has_yml = os.path.isfile(yml_path)
+            has_json_bundled = os.path.isfile(json_bundled)
+            has_yaml_bundled = os.path.isfile(yaml_bundled)
+            has_yml_bundled = os.path.isfile(yml_bundled)
+
+            # Direct path: JSON-only, warn if both exist
+            if has_json or has_yaml or has_yml:
+                if has_json and (has_yaml or has_yml):
+                    yaml_ext = ".yaml" if has_yaml else ".yml"
+                    logging.warning(
+                        f"Both {run_config_filename}.json and {yaml_ext} found. Using .json"
+                    )
+                if has_json:
+                    settings_files.append(json_path)
+                else:
+                    yaml_ext = ".yaml" if has_yaml else ".yml"
+                    message = f"Found {run_config_filename}{yaml_ext} but YAML needs explicit .yaml/.yml extension"
+                    logging.error(message)
+                    raise FileNotFoundError(message)
+            elif has_json_bundled:
+                settings_files.append(json_bundled)
+            elif has_yaml_bundled or has_yml_bundled:
+                yaml_ext = ".yaml" if has_yaml_bundled else ".yml"
+                message = f"Found {run_config_filename}{yaml_ext} but YAML needs explicit .yaml/.yml extension"
+                logging.error(message)
+                raise FileNotFoundError(message)
+            else:
+                message = f"run config not found: {run_config_filename}"
+                logging.error(message)
+                raise FileNotFoundError(message)
 
     logging.debug("Loading configs from: %s", ",".join(settings_files))
     _store_config(settings_files=settings_files)

garak/cli.py

@@ -135,7 +135,7 @@ def main(arguments=None) -> None:
         help="number of generations per prompt",
     )
     parser.add_argument(
-        "--config", type=str, default=None, help="YAML config file for this run"
+        "--config", type=str, default=None, help="YAML or JSON config file for this run"
     )
 
     ## PLUGINS
@@ -298,7 +298,12 @@ def main(arguments=None) -> None:
     # load site config before loading CLI config
     _cli_config_supplied = args.config is not None
     prior_user_agents = _config.get_http_lib_agents()
-    _config.load_config(run_config_filename=args.config)
+    try:
+        _config.load_config(run_config_filename=args.config)
+    except FileNotFoundError as e:
+        logging.exception(e)
+        print(f"❌{e}")
+        exit(1)
 
     # extract what was actually passed on CLI; use a masking argparser
     aux_parser = argparse.ArgumentParser(argument_default=argparse.SUPPRESS)

tests/buffs/test_buff_config.py

@@ -26,7 +26,7 @@
 
 def test_include_original_prompt():
     # https://github.com/python/cpython/pull/97015 to ensure Windows compatibility
-    with tempfile.NamedTemporaryFile(buffering=0, delete=False) as tmp:
+    with tempfile.NamedTemporaryFile(buffering=0, delete=False, suffix=".yaml") as tmp:
         tmp.write(
             """---
 plugins:
@@ -66,7 +66,7 @@ def test_include_original_prompt():
 
 
 def test_exclude_original_prompt():
-    with tempfile.NamedTemporaryFile(buffering=0, delete=False) as tmp:
+    with tempfile.NamedTemporaryFile(buffering=0, delete=False, suffix=".yaml") as tmp:
         tmp.write(
             """---
 plugins: