Support List/Tuple/Dict of dataclasses via CLI, config files, and encode/decode

Add support for container-of-dataclass fields (List[DC], Tuple[DC, ...],
Dict[str, DC]) as first-class citizens. These can now be set from the
command line as YAML/JSON strings (e.g. --points '[{x: 1.0, y: 2.0}]'),
from config files, and via encode/decode. Deeply nested structures are
also supported.

- Add is_dict_of_dataclasses utility and wire it into DataclassWrapper
- Replace NotImplementedError with working FieldWrapper registration
- Add 78 tests covering encode/decode, config file, CLI, and deep nesting
- Document the feature in README, docs/step_by_step, and docs/api
- Bump version to 0.3.2
- Drop Python 3.6 support (bump to >=3.7, remove dataclasses backport)
- Update CI to Python 3.7-3.12, actions v4/v5
- Fix mutable default warnings in test fixtures

Author: yairf11

.github/workflows/pytest.yml

@@ -7,15 +7,12 @@ jobs:
     strategy:
       matrix:
         os: [ ubuntu-latest, macos-latest ]
-        python-version: [ '3.6', '3.7', '3.8', '3.9', '3.10' ]
-        exclude:
-          - os: macos-latest
-            python-version: '3.6'
+        python-version: [ '3.7', '3.8', '3.9', '3.10', '3.11', '3.12' ]
     runs-on: ${{ matrix.os }}
     steps:
-    - uses: actions/checkout@v2
+    - uses: actions/checkout@v4
     - name: Set up Python ${{ matrix.python-version }} on ${{ matrix.os }}
-      uses: actions/setup-python@v1
+      uses: actions/setup-python@v5
       with:
         python-version: ${{ matrix.python-version }}
     - name: Install dependencies

README.md

@@ -243,6 +243,65 @@ Training my_third_exp...
     Saving to /share/experiments/my_third_exp
 ```
 
+#### Lists and Dicts of Dataclasses 🐲
+`pyrallis` also supports `List[Dataclass]`, `Tuple[Dataclass, ...]`, and `Dict[str, Dataclass]` fields. These can be set from a config file in the natural YAML structure, **or directly from the command line as a YAML/JSON string**:
+
+```python
+from dataclasses import dataclass, field
+from typing import List, Dict
+import pyrallis
+
+@dataclass
+class DatasetConfig:
+    path: str = ''
+    split: str = 'train'
+
+@dataclass
+class TrainConfig:
+    # A list of datasets to train on
+    datasets: List[DatasetConfig] = field(default_factory=list)
+    # Named dataset overrides per evaluation task
+    eval_sets: Dict[str, DatasetConfig] = field(default_factory=dict)
+
+@pyrallis.wrap()
+def main(cfg: TrainConfig):
+    for ds in cfg.datasets:
+        print(f'Training on: {ds.path} (split={ds.split})')
+    for name, ds in cfg.eval_sets.items():
+        print(f'Eval {name}: {ds.path} (split={ds.split})')
+```
+
+Pass a YAML flow sequence or mapping from the command line:
+```console
+$ python train_model.py \
+    --datasets '[{path: /data/imagenet, split: train}, {path: /data/coco, split: train}]' \
+    --eval_sets '{imagenet_val: {path: /data/imagenet, split: val}, coco_val: {path: /data/coco, split: val}}'
+Training on: /data/imagenet (split=train)
+Training on: /data/coco (split=train)
+Eval imagenet_val: /data/imagenet (split=val)
+Eval coco_val: /data/coco (split=val)
+```
+
+The equivalent YAML config file uses the standard nested structure:
+```yaml
+datasets:
+  - path: /data/imagenet
+    split: train
+  - path: /data/coco
+    split: train
+eval_sets:
+  imagenet_val:
+    path: /data/imagenet
+    split: val
+  coco_val:
+    path: /data/coco
+    split: val
+```
+
+This also works with deeply nested structures -- for example a `List[Dataclass]` where each dataclass itself contains a `Dict[str, Dataclass]` field, and so on.
+
+> CLI args always take priority over the config file, so you can mix and match: load a base config from a file and override specific container fields on the command line.
+
 ### 🐲 5/5 Easy Serialization with `pyrallis.dump` 🐲
 As your config get longer you will probably want to start working with configuration files. Pyrallis supports encoding a dataclass configuration into a `yaml` file 💾
 

docs/api.md

@@ -353,6 +353,15 @@ Reading sequences and dictionaries is done using the `yaml` syntax. Notice that
 $ python train_model.py --worker_inds=[2,18,42] --worker_names="{2: 'George', 18: 'Ben'}"
 ```
 
+Collections can also contain dataclass types. `#!python typing.List[SomeDataclass]`, `#!python typing.Tuple[SomeDataclass, ...]`, and `#!python typing.Dict[str, SomeDataclass]` are all supported. From the command line these are passed as YAML/JSON strings:
+
+```console
+$ python train_model.py --points '[{x: 1.0, y: 2.0}, {x: 3.0, y: 4.0}]'
+$ python train_model.py --mapping '{origin: {x: 0.0, y: 0.0}, target: {x: 1.0, y: 2.0}}'
+```
+
+Deeply nested structures (e.g. a `List[Dataclass]` where each dataclass contains a `Dict[str, Dataclass]` field) are also supported.
+
 #### Typing Types
 
 `typing.Any`

docs/step_by_step.md

@@ -172,6 +172,65 @@ Training my_third_exp...
     Saving to /share/experiments/my_third_exp
 ```
 
+## Lists and Dicts of Dataclasses
+`pyrallis` also supports `List[Dataclass]`, `Tuple[Dataclass, ...]`, and `Dict[str, Dataclass]` fields. These can be set from a config file in the natural YAML structure, **or directly from the command line as a YAML/JSON string**:
+
+```python
+from dataclasses import dataclass, field
+from typing import List, Dict
+import pyrallis
+
+@dataclass
+class DatasetConfig:
+    path: str = ''
+    split: str = 'train'
+
+@dataclass
+class TrainConfig:
+    # A list of datasets to train on
+    datasets: List[DatasetConfig] = field(default_factory=list)
+    # Named dataset overrides per evaluation task
+    eval_sets: Dict[str, DatasetConfig] = field(default_factory=dict)
+
+@pyrallis.wrap()
+def main(cfg: TrainConfig):
+    for ds in cfg.datasets:
+        print(f'Training on: {ds.path} (split={ds.split})')
+    for name, ds in cfg.eval_sets.items():
+        print(f'Eval {name}: {ds.path} (split={ds.split})')
+```
+
+Pass a YAML flow sequence or mapping from the command line:
+```console
+$ python train_model.py \
+    --datasets '[{path: /data/imagenet, split: train}, {path: /data/coco, split: train}]' \
+    --eval_sets '{imagenet_val: {path: /data/imagenet, split: val}, coco_val: {path: /data/coco, split: val}}'
+Training on: /data/imagenet (split=train)
+Training on: /data/coco (split=train)
+Eval imagenet_val: /data/imagenet (split=val)
+Eval coco_val: /data/coco (split=val)
+```
+
+The equivalent YAML config file uses the standard nested structure:
+```yaml
+datasets:
+  - path: /data/imagenet
+    split: train
+  - path: /data/coco
+    split: train
+eval_sets:
+  imagenet_val:
+    path: /data/imagenet
+    split: val
+  coco_val:
+    path: /data/coco
+    split: val
+```
+
+This also works with deeply nested structures -- for example a `List[Dataclass]` where each dataclass itself contains a `Dict[str, Dataclass]` field, and so on.
+
+> CLI args always take priority over the config file, so you can mix and match: load a base config from a file and override specific container fields on the command line.
+
 ## Easy Serialization
 As your config get longer you will probably want to start working with configuration files. Pyrallis supports encoding a dataclass configuration into a `yaml` file 💾
 

pyrallis/__init__.py

@@ -1,4 +1,4 @@
-__version__ = "0.3.1"
+__version__ = "0.3.2"
 
 from . import wrappers, utils
 from .argparsing import wrap, parse

pyrallis/utils.py

@@ -173,6 +173,15 @@ def is_tuple_or_list_of_dataclasses(t: Type) -> bool:
     return is_tuple_or_list(t) and is_dataclass_type(get_item_type(t))
 
 
+def is_dict_of_dataclasses(t: Type) -> bool:
+    if not is_dict(t):
+        return False
+    args = get_type_arguments(t)
+    if args and len(args) == 2:
+        return is_dataclass_type(args[1])
+    return False
+
+
 def contains_dataclass_type_arg(t: Type) -> bool:
     if is_dataclass_type(t):
         return True

pyrallis/wrappers/dataclass_wrapper.py

@@ -51,11 +51,15 @@ def __init__(
             if not field.init:
                 continue
 
-            elif utils.is_tuple_or_list_of_dataclasses(field.type):
-                raise NotImplementedError(
-                    f"Field {field.name} is of type {field.type}, which isn't "
-                    f"supported yet. (container of a dataclass type)"
+            elif utils.is_tuple_or_list_of_dataclasses(field.type) or utils.is_dict_of_dataclasses(field.type):
+                # Container of dataclasses (List/Tuple/Dict) - treat as a regular field.
+                # Settable via config files, encode/decode, and via CLI as a YAML/JSON string
+                # (e.g. --points '[{x: 1.0, y: 2.0}]' or --mapping '{key: {x: 1.0}}').
+                field_wrapper = field_wrapper_class(field, parent=self, prefix=self.prefix)
+                logger.debug(
+                    f"wrapped container-of-dataclass field at {field_wrapper.dest} has a default value of {field_wrapper.default}"
                 )
+                self.fields.append(field_wrapper)
 
             elif dataclasses.is_dataclass(field.type):
                 # handle a nested dataclass attribute

setup.py

@@ -33,10 +33,9 @@ def find_version(*file_paths: str) -> str:
         "License :: OSI Approved :: MIT License",
         "Operating System :: OS Independent",
     ],
-    python_requires=">=3.6",
+    python_requires=">=3.7",
     install_requires=[
         "typing_inspect",
-        "dataclasses;python_version<'3.7'",
         'pyyaml'
     ],
     setup_requires=["pre-commit"],

tests/conftest.py

@@ -181,7 +181,7 @@ class HyperParameters(TestSetup):
         use_custom_likes: bool = True
 
         # Gender model settings
-        gender: TaskHyperParameters = TaskHyperParameters(
+        gender: TaskHyperParameters = field(default_factory=lambda: TaskHyperParameters(
             "gender",
             num_layers=1,
             num_units=32,
@@ -190,10 +190,10 @@ class HyperParameters(TestSetup):
             dropout_rate=0.1,
             use_image_features=True,
             use_likes=True,
-        )
+        ))
 
         # Age Group Model settings
-        age_group: TaskHyperParameters = TaskHyperParameters(
+        age_group: TaskHyperParameters = field(default_factory=lambda: TaskHyperParameters(
             "age_group",
             num_layers=2,
             num_units=64,
@@ -202,10 +202,10 @@ class HyperParameters(TestSetup):
             dropout_rate=0.1,
             use_image_features=True,
             use_likes=True,
-        )
+        ))
 
         # Personality Model(s) settings:
-        personality: TaskHyperParameters = TaskHyperParameters(
+        personality: TaskHyperParameters = field(default_factory=lambda: TaskHyperParameters(
             "personality",
             num_layers=1,
             num_units=8,
@@ -214,6 +214,6 @@ class HyperParameters(TestSetup):
             dropout_rate=0.1,
             use_image_features=False,
             use_likes=False,
-        )
+        ))
 
     return HyperParameters

tests/test_inheritance.py

@@ -22,8 +22,8 @@ class ExtendedC(Base, TestSetup):
 
 @dataclass
 class Inheritance(TestSetup):
-    ext_b: ExtendedB = ExtendedB()
-    ext_c: ExtendedC = ExtendedC()
+    ext_b: ExtendedB = field(default_factory=ExtendedB)
+    ext_c: ExtendedC = field(default_factory=ExtendedC)
 
 
 def test_simple_subclassing_no_args():