SaveConfigCallback does not work as expected and raises an error under specific settings (DDPStrategy start_method="spawn")

[lengmo1996]

Bug description

When using LightningCLI with

save_config_callback = SaveConfigCallback

and configuring the distributed strategy as

DDPStrategy(start_method="spawn")

the process raises an error:

FileNotFoundError: [Errno 2] No such file or directory: '/workspace/auto'

It seems that when start_method="spawn" is used, Lightning incorrectly interprets some default configuration values such as "auto" (commonly used in accelerator="auto" or similar options) as file paths, and attempts to load them as configuration files.

What version are you seeing the problem on?

v2.5

Reproduced in studio

No response

How to reproduce the bug

main.py

import jsonargparse
from omegaconf import OmegaConf
from jsonargparse import set_loader, set_dumper

if "omegaconf" not in jsonargparse._loaders_dumpers.loaders:
    set_loader("omegaconf", OmegaConf.load)
if "omegaconf" not in jsonargparse._loaders_dumpers.dumpers:
    set_dumper("omegaconf", OmegaConf.save)

from lightning.pytorch.cli import (
    LightningCLI,
    SaveConfigCallback,
)
from lightning.pytorch.demos import BoringDataModule, BoringModel

if __name__ == "__main__":
    LightningCLI(
        model_class=BoringModel,
        datamodule_class=BoringDataModule,
        save_config_callback=SaveConfigCallback,
        parser_kwargs={"parser_mode": "omegaconf"},
    )

base.yaml

# seed_everything: 23
trainer:
  strategy: 
    class_path: DDPStrategy
    init_args:
      start_method: "spawn"


Then, run command:

python main.py fit -c base.yaml

Error messages and logs

# Error messages and logs here please
(base) root@c46491abc3b1:/workspace#  cd /workspace ; /usr/bin/env /opt/conda/bin/python /root/.vscode-server/extensions/ms-python.debugpy-2025.10.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher 45595 -- main.py fit -c base.yaml 
Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/runpy.py", line 198, in _run_module_as_main
    return _run_code(code, main_globals, None,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/runpy.py", line 88, in _run_code
    exec(code, run_globals)
  File "/root/.vscode-server/extensions/ms-python.debugpy-2025.10.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 71, in <module>
    cli.main()
  File "/root/.vscode-server/extensions/ms-python.debugpy-2025.10.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 501, in main
    run()
  File "/root/.vscode-server/extensions/ms-python.debugpy-2025.10.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 351, in run_file
    runpy.run_path(target, run_name="__main__")
  File "/root/.vscode-server/extensions/ms-python.debugpy-2025.10.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 310, in run_path
    return _run_module_code(code, init_globals, run_name, pkg_name=pkg_name, script_name=fname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.vscode-server/extensions/ms-python.debugpy-2025.10.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 127, in _run_module_code
    _run_code(code, mod_globals, init_globals, mod_name, mod_spec, pkg_name, script_name)
  File "/root/.vscode-server/extensions/ms-python.debugpy-2025.10.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 118, in _run_code
    exec(code, run_globals)
  File "main.py", line 17, in <module>
    LightningCLI(
  File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/cli.py", line 400, in __init__
    self.parse_arguments(self.parser, args)
  File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/cli.py", line 554, in parse_arguments
    self.config = parser.parse_args(args)
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_deprecated.py", line 132, in patched_parse
    cfg = parse_method(*args, _skip_validation=_skip_validation, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_core.py", line 465, in parse_args
    cfg, unk = self.parse_known_args(args=args, namespace=cfg)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_core.py", line 313, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace, **kwargs)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/argparse.py", line 2110, in _parse_known_args
    positionals_end_index = consume_positionals(start_index)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/argparse.py", line 2087, in consume_positionals
    take_action(action, args)
  File "/opt/conda/lib/python3.11/argparse.py", line 1983, in take_action
    action(self, namespace, argument_values, option_string)
  File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_actions.py", line 708, in __call__
    namespace[subcommand] = subparser.parse_args(arg_strings, namespace=subnamespace, **kwargs)
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_deprecated.py", line 132, in patched_parse
    cfg = parse_method(*args, _skip_validation=_skip_validation, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_core.py", line 460, in parse_args
    cfg = self._parse_defaults_and_environ(defaults, env)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_core.py", line 409, in _parse_defaults_and_environ
    cfg = self.get_defaults(skip_validation=True)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_core.py", line 1065, in get_defaults
    ActionTypeHint.add_sub_defaults(self, cfg)
  File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_typehints.py", line 478, in add_sub_defaults
    parser._apply_actions(cfg, skip_fn=skip_sub_defaults_apply)
  File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_core.py", line 1404, in _apply_actions
    value = self._check_value_key(action, value, action_dest, prev_cfg, append=append)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_core.py", line 1464, in _check_value_key
    value = action._check_type_(value, cfg=cfg, append=append, mode=self.parser_mode)  # type: ignore[attr-defined]
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_common.py", line 391, in _check_type_
    return self._check_type(value, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_typehints.py", line 558, in _check_type
    val, config_path = parse_value_or_config(val, enable_path=enable_path)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_util.py", line 145, in parse_value_or_config
    parsed_val = load_value(value, simple_types=simple_types)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_loaders_dumpers.py", line 208, in load_value
    loaded_value = loader(value, **load_kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/omegaconf/omegaconf.py", line 189, in load
    with io.open(os.path.abspath(file_), "r", encoding="utf-8") as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/workspace/auto'

Environment

Current environment

• CUDA:
  - GPU:
  - NVIDIA GeForce RTX 3090
  - NVIDIA GeForce RTX 3090
  - available: True
  - version: 12.6
• Lightning:
  - lightning: 2.5.5
  - lightning-utilities: 0.15.2
  - pytorch-lightning: 2.5.5
  - torch: 2.6.0+cu126
  - torchaudio: 2.6.0+cu126
  - torchelastic: 0.2.2
  - torchmetrics: 1.8.2
  - torchvision: 0.21.0+cu126
• Packages:
  - aiohappyeyeballs: 2.6.1
  - aiohttp: 3.13.0
  - aiosignal: 1.4.0
  - antlr4-python3-runtime: 4.9.3
  - archspec: 0.2.5
  - asttokens: 3.0.0
  - astunparse: 1.6.3
  - attrs: 25.1.0
  - autocommand: 2.2.2
  - backports.tarfile: 1.2.0
  - beautifulsoup4: 4.12.3
  - boltons: 24.0.0
  - brotli: 1.1.0
  - certifi: 2024.12.14
  - cffi: 1.17.1
  - chardet: 5.2.0
  - charset-normalizer: 3.4.1
  - click: 8.1.8
  - cmake: 3.31.4
  - colorama: 0.4.6
  - conda: 25.1.0
  - conda-build: 25.1.1
  - conda-index: 0.5.0
  - conda-libmamba-solver: 25.1.1
  - conda-package-handling: 2.4.0
  - conda-package-streaming: 0.11.0
  - decorator: 5.1.1
  - distro: 1.9.0
  - dnspython: 2.7.0
  - docstring-parser: 0.17.0
  - exceptiongroup: 1.2.2
  - executing: 2.1.0
  - expecttest: 0.3.0
  - filelock: 3.17.0
  - frozendict: 2.4.6
  - frozenlist: 1.8.0
  - fsspec: 2024.12.0
  - h2: 4.1.0
  - hpack: 4.0.0
  - hyperframe: 6.0.1
  - hypothesis: 6.124.7
  - idna: 3.10
  - importlib-metadata: 8.0.0
  - importlib-resources: 6.5.2
  - inflect: 7.3.1
  - ipython: 8.31.0
  - jaraco.collections: 5.1.0
  - jaraco.context: 5.3.0
  - jaraco.functools: 4.0.1
  - jaraco.text: 3.12.1
  - jedi: 0.19.2
  - jinja2: 3.1.5
  - jsonargparse: 4.41.0
  - jsonpatch: 1.33
  - jsonpointer: 3.0.0
  - jsonschema: 4.23.0
  - jsonschema-specifications: 2024.10.1
  - libarchive-c: 5.1
  - libmambapy: 2.0.5
  - lief: 0.14.1
  - lightning: 2.5.5
  - lightning-utilities: 0.15.2
  - lintrunner: 0.12.7
  - markupsafe: 3.0.2
  - matplotlib-inline: 0.1.7
  - menuinst: 2.2.0
  - more-itertools: 10.6.0
  - mpmath: 1.3.0
  - multidict: 6.7.0
  - networkx: 3.4.2
  - ninja: 1.11.1.3
  - numpy: 2.2.2
  - nvidia-cublas-cu12: 12.6.4.1
  - nvidia-cuda-cupti-cu12: 12.6.80
  - nvidia-cuda-nvrtc-cu12: 12.6.77
  - nvidia-cuda-runtime-cu12: 12.6.77
  - nvidia-cudnn-cu12: 9.5.1.17
  - nvidia-cufft-cu12: 11.3.0.4
  - nvidia-curand-cu12: 10.3.7.77
  - nvidia-cusolver-cu12: 11.7.1.2
  - nvidia-cusparse-cu12: 12.5.4.2
  - nvidia-cusparselt-cu12: 0.6.3
  - nvidia-nccl-cu12: 2.21.5
  - nvidia-nvjitlink-cu12: 12.6.85
  - nvidia-nvtx-cu12: 12.6.77
  - omegaconf: 2.3.0
  - optree: 0.14.0
  - packaging: 24.2
  - parso: 0.8.4
  - pexpect: 4.9.0
  - pickleshare: 0.7.5
  - pillow: 11.0.0
  - pip: 24.3.1
  - pkginfo: 1.12.0
  - pkgutil-resolve-name: 1.3.10
  - platformdirs: 4.3.6
  - pluggy: 1.5.0
  - prompt-toolkit: 3.0.50
  - propcache: 0.4.1
  - psutil: 6.1.1
  - ptyprocess: 0.7.0
  - pure-eval: 0.2.3
  - pycosat: 0.6.6
  - pycparser: 2.22
  - pygments: 2.19.1
  - pysocks: 1.7.1
  - python-etcd: 0.4.5
  - pytorch-lightning: 2.5.5
  - pytz: 2024.2
  - pyyaml: 6.0.2
  - referencing: 0.36.2
  - requests: 2.32.3
  - rpds-py: 0.22.3
  - ruamel.yaml: 0.18.10
  - ruamel.yaml.clib: 0.2.8
  - setuptools: 75.8.0
  - six: 1.17.0
  - sortedcontainers: 2.4.0
  - soupsieve: 2.5
  - stack-data: 0.6.3
  - sympy: 1.13.1
  - tomli: 2.0.1
  - torch: 2.6.0+cu126
  - torchaudio: 2.6.0+cu126
  - torchelastic: 0.2.2
  - torchmetrics: 1.8.2
  - torchvision: 0.21.0+cu126
  - tqdm: 4.67.1
  - traitlets: 5.14.3
  - triton: 3.2.0
  - truststore: 0.10.0
  - typeguard: 4.3.0
  - types-dataclasses: 0.6.6
  - typeshed-client: 2.8.2
  - typing-extensions: 4.12.2
  - urllib3: 2.3.0
  - wcwidth: 0.2.13
  - wheel: 0.45.1
  - yarl: 1.22.0
  - zipp: 3.21.0
  - zstandard: 0.23.0
• System:
  - OS: Linux
  - architecture:
  - 64bit
  -
  - processor: x86_64
  - python: 3.11.11
  - release: 5.15.0-139-generic
  - version: Gradient Accumulation Scheduler #149~20.04.1-Ubuntu SMP Wed Apr 16 08:29:56 UTC 2025

More info

No response

cc @mauvilsa @justusschock @lantiga

[lengmo1996]

🔄 Additional findings and diagnostics

1. start_method="fork" also fails

After further testing, the same FileNotFoundError occurs even when switching to

DDPStrategy(start_method="fork")

So the issue is not specific to "spawn" — both "spawn" and "fork" modes trigger the same error:

FileNotFoundError: [Errno 2] No such file or directory: '/workspace/auto'

This suggests that the root cause is related to how LightningCLI handles "auto" default values during argument resolution, not merely a multiprocessing problem.

2. Effect of jsonargparse / omegaconf loader registration

When I comment out the following block:

# import jsonargparse
# from omegaconf import OmegaConf
# from jsonargparse import set_loader, set_dumper
# if "omegaconf" not in jsonargparse._loaders_dumpers.loaders:

#     set_loader("omegaconf", OmegaConf.load)

# if "omegaconf" not in jsonargparse._loaders_dumpers.dumpers:

#     set_dumper("omegaconf", OmegaConf.save)

the behavior changes depending on the multiprocessing method:

DDPStrategy(start_method=...) | Behavior -- | -- "fork" | ✅ Works fine — no FileNotFoundError "spawn" | ❌ Raises KeyError: 'Validation failed: omegaconf'

This indicates that:

• The global loader/dumper registry (jsonargparse._loaders_dumpers) is not shared across processes when using "spawn".

• LightningCLI likely attempts to validate or reload configs in the new process before the omegaconf loader is re-registered, leading to KeyError or misinterpreting "auto" as a literal path.

• Under "fork", the registry state is inherited, which explains the behavioral difference.

💡 Hypothesis

The bug seems to stem from how LightningCLI manages or reuses jsonargparse’s omegaconf loader/dumper registration across subprocesses.
When using "spawn", that registration doesn’t propagate, causing either:

• Missing loader errors (KeyError: 'Validation failed: omegaconf'), or

• Misinterpretation of config defaults like "auto" as file paths (FileNotFoundError: '/workspace/auto').

A safer approach might be to re-register the omegaconf loader/dumper explicitly within each spawned process, or to avoid performing config validation in workers.

[mauvilsa]

if "omegaconf" not in jsonargparse._loaders_dumpers.loaders:
    set_loader("omegaconf", OmegaConf.load)
if "omegaconf" not in jsonargparse._loaders_dumpers.dumpers:
    set_dumper("omegaconf", OmegaConf.save)

@lengmo1996 this is incorrect, why are you doing this? set_loader expects a function that receives a string to parse. OmegaConf.load expects a path, so it is not compatible. I mean OmegaConf.load('{"x":1}') would need to work, but it fails with FileNotFoundError: [Errno 2] No such file or directory: '/tmp/{"x":1}'. And this is the error you are getting.

Similarly the use of OmegaConf.save is not valid. The jsonargparse docs could be improved, but still with this, even the most basic parser would fail. Also the _loaders_dumpers model starts with _ to indicate that is private, so users should not access it.

However, what I wonder most about is why are you using set_loader and set_dumper for OmegaConf. As explained in the docs, the support for OmegaConf comes out of the box. The only thing to do is to set the parser mode to "omegaconf". No need to set a loader or dumper.

[lengmo1996]

Thank you for your prompt reply and detailed explanation! @mauvilsa

In fact, the initial problem I encountered occurred when debugging in VSCode using DDPStrategy with the default start_method, which is 'popen'.
At that time, when num_workers > 0 and the execution reached val_run(), I encountered a rather tricky error:

"/usr/local/lib/python3.12/dist-packages/torch/utils/data/_utils/signal_handling.py", line 73, in handler
[rank0]: _error_if_any_worker_fails()
[rank0]: RuntimeError: DataLoader worker (pid 3081641) is killed by signal: Aborted.

Of course, setting num_workers=0 can avoid this issue, but after checking some references, I found that the 'popen' method may have certain compatibility problems.

Therefore, I tried switching to the 'spawn' method — which then led to the issue discussed here.
However, when using 'spawn', the first error I got was:

W1011 06:29:49.602000 13409 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 13824 via signal SIGTERM
Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/runpy.py", line 198, in _run_module_as_main
    return _run_code(code, main_globals, None,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/runpy.py", line 88, in _run_code
    exec(code, run_globals)
  File "/root/.vscode-server/extensions/ms-python.debugpy-2025.10.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/__main__.py", line 71, in <module>
    cli.main()
  File "/root/.vscode-server/extensions/ms-python.debugpy-2025.10.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 501, in main
    run()
  File "/root/.vscode-server/extensions/ms-python.debugpy-2025.10.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher/../../debugpy/../debugpy/server/cli.py", line 351, in run_file
    runpy.run_path(target, run_name="__main__")
  File "/root/.vscode-server/extensions/ms-python.debugpy-2025.10.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 310, in run_path
    return _run_module_code(code, init_globals, run_name, pkg_name=pkg_name, script_name=fname)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.vscode-server/extensions/ms-python.debugpy-2025.10.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 127, in _run_module_code
    _run_code(code, mod_globals, init_globals, mod_name, mod_spec, pkg_name, script_name)
  File "/root/.vscode-server/extensions/ms-python.debugpy-2025.10.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundle/pydevd_runpy.py", line 118, in _run_code
    exec(code, run_globals)
  File "main.py", line 17, in <module>
    LightningCLI(
  File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/cli.py", line 414, in __init__
    self._run_subcommand(self.subcommand)
  File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/cli.py", line 747, in _run_subcommand
    fn(**fn_kwargs)
  File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/trainer/trainer.py", line 560, in fit
    call._call_and_handle_interrupt(
  File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/trainer/call.py", line 48, in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/strategies/launchers/multiprocessing.py", line 144, in launch
    while not process_context.join():
              ^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 215, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException: 

-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_typehints.py", line 558, in _check_type
    val, config_path = parse_value_or_config(val, enable_path=enable_path)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_util.py", line 145, in parse_value_or_config
    parsed_val = load_value(value, simple_types=simple_types)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_loaders_dumpers.py", line 199, in load_value
    if loaded_value is not_loaded and not loader_json_superset[mode]:
                                          ~~~~~~~~~~~~~~~~~~~~^^^^^^
KeyError: 'omegaconf'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_core.py", line 1162, in validate
    check_values(cfg)
  File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_core.py", line 1144, in check_values
    self._check_value_key(action, val, key, ccfg)
  File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_core.py", line 1464, in _check_value_key
    value = action._check_type_(value, cfg=cfg, append=append, mode=self.parser_mode)  # type: ignore[attr-defined]
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_common.py", line 391, in _check_type_
    return self._check_type(value, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_typehints.py", line 559, in _check_type
    except get_loader_exceptions():
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_loaders_dumpers.py", line 168, in get_loader_exceptions
    return loader_exceptions[mode]
           ~~~~~~~~~~~~~~~~~^^^^^^
KeyError: 'omegaconf'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/conda/lib/python3.11/site-packages/torch/multiprocessing/spawn.py", line 90, in _wrap
    fn(i, *args)
  File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/strategies/launchers/multiprocessing.py", line 173, in _wrapping_function
    results = function(*args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/trainer/trainer.py", line 598, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/trainer/trainer.py", line 973, in _run
    call._call_setup_hook(self)  # allow user to set up LightningModule in accelerator environment
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/trainer/call.py", line 109, in _call_setup_hook
    _call_callback_hooks(trainer, "setup", stage=fn)
  File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/trainer/call.py", line 228, in _call_callback_hooks
    fn(trainer, trainer.lightning_module, *args, **kwargs)
  File "/opt/conda/lib/python3.11/site-packages/lightning/pytorch/cli.py", line 283, in setup
    self.parser.save(
  File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_deprecated.py", line 168, in patched_save
    return self._unpatched_save(cfg, *args, multifile=multifile, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_core.py", line 922, in save
    f.write(self.dump(cfg, **dump_kwargs))  # type: ignore[arg-type]
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_deprecated.py", line 159, in patched_dump
    return self._unpatched_dump(cfg, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_core.py", line 800, in dump
    self.validate(cfg)
  File "/opt/conda/lib/python3.11/site-packages/jsonargparse/_core.py", line 1170, in validate
    raise type(ex)(message) from ex
KeyError: 'Validation failed: omegaconf'

The commonly suggested fix online was to add the following code:

if "omegaconf" not in jsonargparse._loaders_dumpers.loaders:
    set_loader("omegaconf", OmegaConf.load)
if "omegaconf" not in jsonargparse._loaders_dumpers.dumpers:
    set_dumper("omegaconf", OmegaConf.save)

After adding this, I encountered another error:

FileNotFoundError: [Errno 2] No such file or directory: '/workspace/auto'

That’s the whole process I went through.
However, I think the key point here is that when running the code mentioned in this issue — that is, setting start_method='spawn' — it always throws

KeyError: 'Validation failed: omegaconf'

Is this behavior expected, or does it indicate that I’m doing something wrong in my setup?

[mauvilsa]

The commonly suggested fix online was to add the following code

Please share the location where this was suggested. It should be commented there that it is wrong, to avoid misleading people.

However, I think the key point here is that when running the code mentioned in this issue — that is, setting start_method='spawn' — it always throws

KeyError: 'Validation failed: omegaconf'

This does look like an actual bug. I will look into it.

[mauvilsa]

@lengmo1996 the issue should be fixed with omni-us/jsonargparse#784. You can install the current jsonargparse in develop and test it out.

The commonly suggested fix online was to add the following code

I tried to find such suggestions but with no luck. If you do have a link, please share it. Or maybe by "online" did you mean that an LLM suggested this to you?

[lengmo1996]

@mauvilsa Thanks a lot for the quick fix! Really appreciate your prompt response and hard work. 🙏
You’re right — that came from a suggestion by ChatGPT. I also checked around but couldn’t find any related sources online. My apologies for not confirming it more carefully — I hope it didn’t cause you too much inconvenience.