Running LightningCLI from Python cannot pass a config.yaml file

[scy-helbling]

Bug description

I want to import my LightningCLI (defined in main.py) from a second file run_multiple_seeds.py to execute multiple fit() runs with each slightly different configurations. As a baseline configuration, I want to have a config.yaml file and adjust this in a loop.

Calling the cli_main() from Python should be supported as stated here in the docs. Moreover, the docs state there that:

All the features that are supported from the command line can be used when giving args as a list of strings. It is also possible to provide a dict or jsonargparse.Namespace.

I interpret this that it also supports parsing the config.yaml including any fancy ${other.param} interpolation.

Expected behavior:

1. Calling main.py with command line arguments -c config.yaml fit successfully fits the model.
2. Calling run_multiple_seeds.py without any command line arguments successfully fits the model three times.

Actual behavior:
(1) works.

(2): With failure_mode = "PASSING_CONFIG_AS_FILE_REFERENCE":
This throws a lightning fabric exception lightning.fabric.utilities.exceptions.MisconfigurationException: No configure_optimizers() method defined, indicating that the entire config.yaml wasn't parsed at all (since this would contain the necessary optimizer configuration).

(2): With failure_mode = "PASSING_CONFIG_AS_DICT":
This throws an argparse ArgumentError Expected a <class 'int'>. Got value: ${trainer.max_epochs}, indicating that the omegaconf variable interpolation didn't work on the passed dict.
Note: Replacing ${trainer.max_epochs} with 5 fixes the issue.

What version are you seeing the problem on?

v2.4 & 2.5.1

How to reproduce the bug

File main.py:

import torch
from lightning import LightningModule
from lightning.pytorch.cli import ArgsType, LightningCLI
from lightning.pytorch.demos.boring_classes import BoringDataModule


class BoringModel(LightningModule):
    def __init__(self, out_dim: int = 10):
        super().__init__()
        self.l1 = torch.nn.Linear(32, out_dim)

    def forward(self, x):
        return torch.relu(self.l1(x.view(x.size(0), -1)))

    def training_step(self, batch, batch_nb):
        x = batch
        x = self(x)
        return x.sum()


class MyLightningCLI(LightningCLI):
    pass


def cli_main(args: ArgsType = None):
    # noinspection PyUnusedLocal
    cli = MyLightningCLI(
        BoringModel,
        BoringDataModule,
        args=args,
        parser_kwargs={"parser_mode": "omegaconf"},
        run=args is None,
    )
    return cli


if __name__ == "__main__":
    cli_main()

File run_multiple_seeds.py:

import yaml

from main import cli_main

if __name__ == '__main__':
    failure_mode = "PASSING_CONFIG_AS_FILE_REFERENCE"

    if failure_mode == "PASSING_CONFIG_AS_DICT":
        with open("config.yaml", "r") as f:
            config = yaml.safe_load(f)

    elif failure_mode == "PASSING_CONFIG_AS_FILE_REFERENCE":
        config = {"config": "config.yaml"}

    for seed in [42, 43, 44]:
        # Modify the seed value in the configuration
        config["seed_everything"] = seed
        cli = cli_main(args=config)
        cli.trainer.fit(cli.model, cli.datamodule)

File config.yaml:

# lightning.pytorch==2.4.0.post0
trainer:
  max_epochs: 5
optimizer:
  class_path: torch.optim.Adam
  init_args:
    lr: 0.003
lr_scheduler:
  class_path: torch.optim.lr_scheduler.CosineAnnealingLR
  init_args:
    T_max: ${trainer.max_epochs}

Error messages and logs

Error with failure_mode = "PASSING_CONFIG_AS_DICT":

2025-04-16 17:54:57,872 - LightningArgumentParser - DEBUG - Loaded parser defaults: Namespace(config=None, seed_everything=True, trainer=Namespace(accelerator='auto', strategy='auto', devices='auto', num_nodes=1, precision=None, logger=None, callbacks=None, fast_dev_run=False, max_epochs=None, min_epochs=None, max_steps=-1, min_steps=None, max_time=None, limit_train_batches=None, limit_val_batches=None, limit_test_batches=None, limit_predict_batches=None, overfit_batches=0.0, val_check_interval=None, check_val_every_n_epoch=1, num_sanity_val_steps=None, log_every_n_steps=None, enable_checkpointing=None, enable_progress_bar=None, enable_model_summary=None, accumulate_grad_batches=1, gradient_clip_val=None, gradient_clip_algorithm=None, deterministic=None, benchmark=None, inference_mode=True, use_distributed_sampler=True, profiler=None, detect_anomaly=False, barebones=False, plugins=None, sync_batchnorm=False, reload_dataloaders_every_n_epochs=0, default_root_dir=None), model=Namespace(out_dim=10), optimizer=None, lr_scheduler=None)
2025-04-16 17:54:57,878 - LightningArgumentParser - DEBUG - Skipping parameter "params" from "torch.optim.Adam.__init__" because of: Parameter requested to be skipped.
2025-04-16 17:54:57,880 - LightningArgumentParser - DEBUG - Parsed object: Namespace()
2025-04-16 17:54:57,897 - LightningArgumentParser - DEBUG - Skipping parameter "optimizer" from "torch.optim.lr_scheduler.CosineAnnealingLR.__init__" because of: Parameter requested to be skipped.
2025-04-16 17:54:57,898 - LightningArgumentParser - ERROR - Parser key "T_max":
  Expected a <class 'int'>. Got value: ${trainer.max_epochs}
2025-04-16 17:54:57,898 - LightningArgumentParser - ERROR - Parser key "lr_scheduler":
  Does not validate against any of the Union subtypes
  Subtypes: [<class 'NoneType'>, <class 'torch.optim.lr_scheduler.LRScheduler'>, <class 'lightning.pytorch.cli.ReduceLROnPlateau'>]
  Errors:
    - Expected a <class 'NoneType'>
    - Problem with given class_path 'torch.optim.lr_scheduler.CosineAnnealingLR':
        Parser key "T_max":
          Expected a <class 'int'>. Got value: ${trainer.max_epochs}
    - Import path torch.optim.lr_scheduler.CosineAnnealingLR does not correspond to a subclass of ReduceLROnPlateau
  Given value type: <class 'dict'>
  Given value: {'class_path': 'torch.optim.lr_scheduler.CosineAnnealingLR', 'init_args': {'T_max': '${trainer.max_epochs}'}}
2025-04-16 17:54:57,898 - LightningArgumentParser - DEBUG - Debug enabled, thus raising exception instead of exit.
Traceback (most recent call last):
  File "C:\my_project\.venv\Lib\site-packages\jsonargparse\_typehints.py", line 838, in adapt_typehints
    vals.append(adapt_typehints(val, subtype, **adapt_kwargs))
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\my_project\.venv\Lib\site-packages\jsonargparse\_typehints.py", line 787, in adapt_typehints
    raise_unexpected_value(f"Expected a {typehint}", val)
  File "C:\my_project\.venv\Lib\site-packages\jsonargparse\_typehints.py", line 718, in raise_unexpected_value
    raise ValueError(message) from exception
ValueError: Expected a <class 'NoneType'>. Got value: {'class_path': 'torch.optim.lr_scheduler.CosineAnnealingLR', 'init_args': {'T_max': '${trainer.max_epochs}'}}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\my_project\.venv\Lib\site-packages\jsonargparse\_typehints.py", line 597, in _check_type
    raise ex
  File "C:\my_project\.venv\Lib\site-packages\jsonargparse\_typehints.py", line 582, in _check_type
    val = adapt_typehints(val, self._typehint, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\my_project\.venv\Lib\site-packages\jsonargparse\_typehints.py", line 846, in adapt_typehints
    raise_union_unexpected_value(sorted_subtypes, val, vals)
  File "C:\my_project\.venv\Lib\site-packages\jsonargparse\_typehints.py", line 725, in raise_union_unexpected_value
    raise ValueError(
ValueError: Does not validate against any of the Union subtypes
Subtypes: [<class 'NoneType'>, <class 'torch.optim.lr_scheduler.LRScheduler'>, <class 'lightning.pytorch.cli.ReduceLROnPlateau'>]
Errors:
  - Expected a <class 'NoneType'>
  - Problem with given class_path 'torch.optim.lr_scheduler.CosineAnnealingLR':
      Parser key "T_max":
        Expected a <class 'int'>. Got value: ${trainer.max_epochs}
  - Import path torch.optim.lr_scheduler.CosineAnnealingLR does not correspond to a subclass of ReduceLROnPlateau
Given value type: <class 'dict'>
Given value: {'class_path': 'torch.optim.lr_scheduler.CosineAnnealingLR', 'init_args': {'T_max': '${trainer.max_epochs}'}}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\my_project\.venv\Lib\site-packages\jsonargparse\_core.py", line 505, in parse_object
    cfg_apply = self._apply_actions(cfg_obj, prev_cfg=cfg)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\my_project\.venv\Lib\site-packages\jsonargparse\_core.py", line 1372, in _apply_actions
    value = self._check_value_key(action, value, action_dest, prev_cfg)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\my_project\.venv\Lib\site-packages\jsonargparse\_core.py", line 1424, in _check_value_key
    value = action._check_type_(value, cfg=cfg)  # type: ignore[attr-defined]
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\my_project\.venv\Lib\site-packages\jsonargparse\_common.py", line 333, in _check_type_
    return self._check_type(value, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\my_project\.venv\Lib\site-packages\jsonargparse\_typehints.py", line 610, in _check_type
    raise TypeError(f'Parser key "{self.dest}"{elem}:\n{error}') from ex
TypeError: Parser key "lr_scheduler":
  Does not validate against any of the Union subtypes
  Subtypes: [<class 'NoneType'>, <class 'torch.optim.lr_scheduler.LRScheduler'>, <class 'lightning.pytorch.cli.ReduceLROnPlateau'>]
  Errors:
    - Expected a <class 'NoneType'>
    - Problem with given class_path 'torch.optim.lr_scheduler.CosineAnnealingLR':
        Parser key "T_max":
          Expected a <class 'int'>. Got value: ${trainer.max_epochs}
    - Import path torch.optim.lr_scheduler.CosineAnnealingLR does not correspond to a subclass of ReduceLROnPlateau
  Given value type: <class 'dict'>
  Given value: {'class_path': 'torch.optim.lr_scheduler.CosineAnnealingLR', 'init_args': {'T_max': '${trainer.max_epochs}'}}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "C:\my_project\run_multiple_seeds.py", line 18, in <module>
    cli = cli_main(args=config)
          ^^^^^^^^^^^^^^^^^^^^^
  File "C:\my_project\main.py", line 27, in cli_main
    cli = MyLightningCLI(
          ^^^^^^^^^^^^^^^
  File "C:\my_project\.venv\Lib\site-packages\lightning\pytorch\cli.py", line 383, in __init__
    self.parse_arguments(self.parser, args)
  File "C:\my_project\.venv\Lib\site-packages\lightning\pytorch\cli.py", line 532, in parse_arguments
    self.config = parser.parse_object(args)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\my_project\.venv\Lib\site-packages\jsonargparse\_deprecated.py", line 123, in patched_parse
    cfg = parse_method(*args, _skip_validation=_skip_validation, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\my_project\.venv\Lib\site-packages\jsonargparse\_core.py", line 518, in parse_object
    self.error(str(ex), ex)
  File "C:\my_project\.venv\Lib\site-packages\jsonargparse\_core.py", line 1065, in error
    raise argument_error(message) from ex
argparse.ArgumentError: Parser key "lr_scheduler":
  Does not validate against any of the Union subtypes
  Subtypes: [<class 'NoneType'>, <class 'torch.optim.lr_scheduler.LRScheduler'>, <class 'lightning.pytorch.cli.ReduceLROnPlateau'>]
  Errors:
    - Expected a <class 'NoneType'>
    - Problem with given class_path 'torch.optim.lr_scheduler.CosineAnnealingLR':
        Parser key "T_max":
          Expected a <class 'int'>. Got value: ${trainer.max_epochs}
    - Import path torch.optim.lr_scheduler.CosineAnnealingLR does not correspond to a subclass of ReduceLROnPlateau
  Given value type: <class 'dict'>
  Given value: {'class_path': 'torch.optim.lr_scheduler.CosineAnnealingLR', 'init_args': {'T_max': '${trainer.max_epochs}'}}

---

Error with failure_mode = "PASSING_CONFIG_AS_FILE_REFERENCE":

2025-04-16 18:01:33,959 - LightningArgumentParser - DEBUG - Loaded parser defaults: Namespace(config=None, seed_everything=True, trainer=Namespace(accelerator='auto', strategy='auto', devices='auto', num_nodes=1, precision=None, logger=None, callbacks=None, fast_dev_run=False, max_epochs=None, min_epochs=None, max_steps=-1, min_steps=None, max_time=None, limit_train_batches=None, limit_val_batches=None, limit_test_batches=None, limit_predict_batches=None, overfit_batches=0.0, val_check_interval=None, check_val_every_n_epoch=1, num_sanity_val_steps=None, log_every_n_steps=None, enable_checkpointing=None, enable_progress_bar=None, enable_model_summary=None, accumulate_grad_batches=1, gradient_clip_val=None, gradient_clip_algorithm=None, deterministic=None, benchmark=None, inference_mode=True, use_distributed_sampler=True, profiler=None, detect_anomaly=False, barebones=False, plugins=None, sync_batchnorm=False, reload_dataloaders_every_n_epochs=0, default_root_dir=None), model=Namespace(out_dim=10), optimizer=None, lr_scheduler=None)
2025-04-16 18:01:33,965 - LightningArgumentParser - DEBUG - Parsed object: {'config': 'config.yaml', 'seed_everything': 42}
Seed set to 42
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
HPU available: False, using: 0 HPUs
C:\my_project\.venv\Lib\site-packages\lightning\pytorch\loops\utilities.py:72: `max_epochs` was not set. Setting it to 1000 epochs. To train without an epoch limit, set `max_epochs=-1`.
Traceback (most recent call last):
  File "C:\my_project\run_multiple_seeds.py", line 23, in <module>
    cli.trainer.fit(cli.model, cli.datamodule)
  File "C:\my_project\.venv\Lib\site-packages\lightning\pytorch\trainer\trainer.py", line 538, in fit
    call._call_and_handle_interrupt(
  File "C:\my_project\.venv\Lib\site-packages\lightning\pytorch\trainer\call.py", line 47, in _call_and_handle_interrupt
    return trainer_fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\my_project\.venv\Lib\site-packages\lightning\pytorch\trainer\trainer.py", line 574, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "C:\my_project\.venv\Lib\site-packages\lightning\pytorch\trainer\trainer.py", line 931, in _run
    _verify_loop_configurations(self)
  File "C:\my_project\.venv\Lib\site-packages\lightning\pytorch\trainer\configuration_validator.py", line 36, in _verify_loop_configurations
    __verify_train_val_loop_configuration(trainer, model)
  File "C:\my_project\.venv\Lib\site-packages\lightning\pytorch\trainer\configuration_validator.py", line 59, in __verify_train_val_loop_configuration
    raise MisconfigurationException(
lightning.fabric.utilities.exceptions.MisconfigurationException: No `configure_optimizers()` method defined. Lightning `Trainer` expects as minimum a `training_step()`, `train_dataloader()` and `configure_optimizers()` to be defined.

Environment

Current environment

• CUDA:
  • GPU:
    • NVIDIA GeForce RTX 3060
  • available: True
  • version: 12.1
• Lightning:
  • lightning: 2.5.1
  • lightning-utilities: 0.11.8
  • pytorch-lightning: 2.4.0
  • pytorch-model-summary: 0.1.2
  • torch: 2.5.1+cu121
  • torchinfo: 1.8.0
  • torchmetrics: 1.5.1
  • torchvision: 0.20.1
• Packages:
  • absl-py: 2.1.0
  • accelerate: 1.1.0
  • aiohappyeyeballs: 2.4.3
  • aiohttp: 3.10.10
  • aiosignal: 1.3.1
  • antlr4-python3-runtime: 4.9.3
  • anyio: 4.6.2.post1
  • argon2-cffi: 23.1.0
  • argon2-cffi-bindings: 21.2.0
  • arrow: 1.3.0
  • asttokens: 3.0.0
  • astunparse: 1.6.3
  • async-lru: 2.0.4
  • attrs: 25.3.0
  • autocommand: 2.2.2
  • babel: 2.16.0
  • backports.tarfile: 1.2.0
  • beautifulsoup4: 4.12.3
  • black: 25.1.0
  • bleach: 6.1.0
  • blinker: 1.9.0
  • boxcutterdata: 3.1.0
  • build: 1.2.2.post1
  • certifi: 2025.1.31
  • cffi: 1.17.1
  • charset-normalizer: 3.4.1
  • click: 8.1.8
  • colorama: 0.4.6
  • comm: 0.2.2
  • configargparse: 1.7
  • contourpy: 1.3.0
  • cycler: 0.12.1
  • dash: 3.0.0
  • dash-core-components: 2.0.0
  • dash-html-components: 2.0.0
  • dash-table: 5.0.0
  • debugpy: 1.8.7
  • decorator: 5.2.1
  • defusedxml: 0.7.1
  • dm-tree: 0.1.8
  • docstring-parser: 0.16
  • etils: 1.10.0
  • executing: 2.2.0
  • fastjsonschema: 2.21.1
  • filelock: 3.16.1
  • flask: 3.0.3
  • flatbuffers: 24.3.25
  • fonttools: 4.54.1
  • fqdn: 1.5.1
  • frozenlist: 1.5.0
  • fsspec: 2024.10.0
  • gast: 0.6.0
  • google-pasta: 0.2.0
  • googleapis-common-protos: 1.66.0
  • grad-cam: 1.5.4
  • grpcio: 1.67.1
  • h11: 0.14.0
  • h5py: 3.12.1
  • httpcore: 1.0.6
  • httpx: 0.27.2
  • huggingface-hub: 0.26.2
  • idna: 3.10
  • imageio: 2.36.1
  • immutabledict: 4.2.1
  • importlib-metadata: 8.6.1
  • importlib-resources: 6.4.5
  • inflect: 7.3.1
  • ipykernel: 6.29.5
  • ipympl: 0.9.4
  • ipynbname: 2024.1.0.0
  • ipython: 9.0.2
  • ipython-genutils: 0.2.0
  • ipython-pygments-lexers: 1.1.1
  • ipywidgets: 8.1.5
  • isoduration: 20.11.0
  • isort: 6.0.1
  • itsdangerous: 2.2.0
  • jaraco.collections: 5.1.0
  • jaraco.context: 5.3.0
  • jaraco.functools: 4.0.1
  • jaraco.text: 3.12.1
  • jedi: 0.19.2
  • jinja2: 3.1.6
  • joblib: 1.4.2
  • json5: 0.9.25
  • jsonargparse: 4.38.0
  • jsonpointer: 3.0.0
  • jsonschema: 4.23.0
  • jsonschema-specifications: 2024.10.1
  • jupyter-client: 8.6.3
  • jupyter-core: 5.7.2
  • jupyter-events: 0.10.0
  • jupyter-lsp: 2.2.5
  • jupyter-server: 2.14.2
  • jupyter-server-terminals: 0.5.3
  • jupyterlab: 4.2.5
  • jupyterlab-pygments: 0.3.0
  • jupyterlab-server: 2.27.3
  • jupyterlab-widgets: 3.0.13
  • jupytext: 1.16.4
  • kagglehub: 0.3.4
  • keras: 3.6.0
  • keras-core: 0.1.7
  • kiwisolver: 1.4.7
  • lazy-loader: 0.4
  • libclang: 18.1.1
  • lightning: 2.5.1
  • lightning-utilities: 0.11.8
  • llvmlite: 0.43.0
  • markdown: 3.7
  • markdown-it-py: 3.0.0
  • markupsafe: 3.0.2
  • matplotlib: 3.9.2
  • matplotlib-inline: 0.1.7
  • mdit-py-plugins: 0.4.2
  • mdurl: 0.1.2
  • mistune: 3.0.2
  • ml-dtypes: 0.4.1
  • more-itertools: 10.3.0
  • mpmath: 1.3.0
  • multidict: 6.1.0
  • mypy-extensions: 1.0.0
  • namex: 0.0.8
  • narwhals: 1.31.0
  • nbclient: 0.10.0
  • nbconvert: 7.16.4
  • nbformat: 5.10.4
  • nest-asyncio: 1.6.0
  • networkx: 3.4.2
  • notebook-shim: 0.2.4
  • numba: 0.60.0
  • numpy: 1.26.4
  • omegaconf: 2.3.0
  • open3d: 0.19.0
  • opencv-python: 4.10.0.84
  • opt-einsum: 3.4.0
  • optree: 0.13.1
  • overrides: 7.7.0
  • packaging: 24.2
  • pandas: 2.2.3
  • pandocfilters: 1.5.1
  • parso: 0.8.4
  • pathspec: 0.12.1
  • pillow: 11.1.0
  • pip: 24.0
  • pip-tools: 7.4.1
  • platformdirs: 4.3.7
  • plotly: 6.0.1
  • prometheus-client: 0.21.0
  • promise: 2.3
  • prompt-toolkit: 3.0.50
  • propcache: 0.2.0
  • protobuf: 5.28.3
  • psutil: 6.1.0
  • pure-eval: 0.2.3
  • pyarrow: 18.0.0
  • pycparser: 2.22
  • pygments: 2.19.1
  • pyparsing: 3.2.0
  • pyproject-hooks: 1.2.0
  • pyqt6: 6.8.1
  • pyqt6-qt6: 6.8.2
  • pyqt6-sip: 13.10.0
  • pyside6-addons: 6.8.0.2
  • pyside6-essentials: 6.8.0.2
  • python-dateutil: 2.9.0.post0
  • python-json-logger: 2.0.7
  • pytorch-lightning: 2.4.0
  • pytorch-model-summary: 0.1.2
  • pytz: 2025.1
  • pywin32: 310
  • pywinpty: 2.0.14
  • pyyaml: 6.0.2
  • pyzmq: 26.2.0
  • referencing: 0.36.2
  • regex: 2024.9.11
  • requests: 2.32.3
  • retrying: 1.3.4
  • rfc3339-validator: 0.1.4
  • rfc3986-validator: 0.1.1
  • rich: 13.9.4
  • rpds-py: 0.23.1
  • safetensors: 0.4.5
  • scikit-image: 0.25.0
  • scikit-learn: 1.5.2
  • scipy: 1.14.1
  • seaborn: 0.13.2
  • send2trash: 1.8.3
  • setuptools: 77.0.3
  • shiboken6: 6.8.0.2
  • simple-parsing: 0.1.6
  • six: 1.17.0
  • sknw: 0.15
  • sniffio: 1.3.1
  • soupsieve: 2.6
  • stack-data: 0.6.3
  • stringcase: 1.2.0
  • sympy: 1.13.1
  • tenacity: 9.0.0
  • tensorboard: 2.18.0
  • tensorboard-data-server: 0.7.2
  • tensorflow: 2.18.0
  • tensorflow-datasets: 4.9.7
  • tensorflow-intel: 2.18.0
  • tensorflow-io-gcs-filesystem: 0.31.0
  • tensorflow-metadata: 1.16.1
  • termcolor: 2.5.0
  • terminado: 0.18.1
  • threadpoolctl: 3.5.0
  • tifffile: 2025.1.10
  • timm: 1.0.11
  • tinycss2: 1.4.0
  • tokenizers: 0.20.1
  • toml: 0.10.2
  • tomli: 2.0.1
  • torch: 2.5.1+cu121
  • torchinfo: 1.8.0
  • torchmetrics: 1.5.1
  • torchvision: 0.20.1
  • tornado: 6.4.1
  • tqdm: 4.66.6
  • traitlets: 5.14.3
  • transformers: 4.46.1
  • ttach: 0.0.3
  • typeguard: 4.3.0
  • types-python-dateutil: 2.9.0.20241003
  • typeshed-client: 2.7.0
  • typing-extensions: 4.12.2
  • tzdata: 2025.1
  • uri-template: 1.3.0
  • urllib3: 2.3.0
  • wcwidth: 0.2.13
  • webcolors: 24.8.0
  • webencodings: 0.5.1
  • websocket-client: 1.8.0
  • werkzeug: 3.0.6
  • wheel: 0.44.0
  • widgetsnbextension: 4.0.13
  • wrapt: 1.16.0
  • yarl: 1.17.1
  • zipp: 3.21.0
  • zivid: 2.13.1.2.13.1
  • zividexploratory: 0.0.1
• System:
  • OS: Windows
  • architecture:
    • 64bit
    • WindowsPE
  • processor: Intel64 Family 6 Model 158 Stepping 10, GenuineIntel
  • python: 3.11.9
  • release: 10
  • version: 10.0.22631

[scy-helbling]

Note, I just now also verified the bug reproducibility with Lightning v2.5.1 but I don't have the right to add that label as well.

[mauvilsa]

As noted here, variable interpolation is only supported on a single yaml. It needs to be OmegaConf reading this yaml, so interpolation will not work if the yaml is loaded manually. If you want interpolation support when parsing a dict, please create a feature request in jsonargparse. Though, I don't know right now if that can be easily implemented.

But anyway, the case explained above it is easy to make it work. Just do:

cli = cli_main(args=["--config=config.yaml", f"--seed_everything={seed}"])

[scy-helbling]

Thanks for clearing this up. I kind of worked around this by manually parsing the config using OmegaConf now. This is way more flexible since I sometimes also want to sweep different parameters than just the seed.

For reference for others, in run_multiple_seeds.py, I now do:

baseline_config = OmegaConf.to_container(OmegaConf.load("config.yaml"), resolve=True)
# Modify baseline_config
cli = cli_main(args=baseline_config)

[mauvilsa]

@scy-helbling I managed to implement an improved omegaconf's variable interpolation in jsonargparse. It supports interpolation across configs and command line arguments. I would be interested in knowing if the sweeping that you do works with this without the need of that workaround. Please have a look and and tryout omni-us/jsonargparse#765

[scy-helbling]

@mauvilsa I tried out your feature branch by:

1. Using the replication code above with the change

- parser_kwargs={"parser_mode": "omegaconf"},
+ parser_kwargs={"parser_mode": "omegaconf+"},

2. Installing an updated version of jsonargparse via pip install -U --force-reinstall git+https://github.com/omni-us/jsonargparse.git@issue-461-omegaconf-global#egg=jsonargparse[signatures]

The calling method (2) is still failing unimproved in both error modes :(

[mauvilsa]

@scy-helbling thank you for testing. There was a separate bug in the parsing of {"config": "config.yaml"}. I have fixed it in the same branch. You can try again.

[scy-helbling]

@mauvilsa This fixes both issues:

1. PASSING_CONFIG_AS_FILE_REFERENCE: args={"config": "config.yaml"} is successful (no code changes necessary) ✅
2. PASSING_CONFIG_AS_DICT: args=yaml.safe_load(<config_file_ref>) is successful with {"parser_mode": "omegaconf+"} ✅

Thank you!