Custom Subcommand without Model arg

[enrico-stauss]

Bug description

I tried to register a custom subcommand with the LightningCLI as it is also documented here. My custom subcommand will not require the Model arg, so I omitted it from the signature. However, I keep getting the same validation error:

error: Validation failed: Key "cmd.model" is required but not included in config object or its value is None.

What version are you seeing the problem on?

v2.4

How to reproduce the bug

from typing import Dict, Set

import pytorch_lightning as pl
from pytorch_lightning.cli import LightningCLI


class Trainer(pl.Trainer):
    def cmd(self, datamodule:pl.LightningDataModule):
        print("Works")


class CLI(LightningCLI):
    @staticmethod
    def subcommands() -> Dict[str, Set[str]]:
        subcommands = LightningCLI.subcommands()
        subcommands["cmd"] = {"datamodule"}
        return subcommands


def main():
    CLI(trainer_class=Trainer)


main()

Run the above script like this:

python script.py cmd

Error messages and logs

2024-10-29 14:09:20,378 - LightningArgumentParser - DEBUG - Skipping parameter "model" from "__main__.Trainer.fit" because of: Parameter requested to be skipped.
2024-10-29 14:09:20,378 - LightningArgumentParser - DEBUG - Skipping parameter "train_dataloaders" from "__main__.Trainer.fit" because of: Parameter requested to be skipped.
2024-10-29 14:09:20,378 - LightningArgumentParser - DEBUG - Skipping parameter "val_dataloaders" from "__main__.Trainer.fit" because of: Parameter requested to be skipped.
2024-10-29 14:09:20,378 - LightningArgumentParser - DEBUG - Skipping parameter "datamodule" from "__main__.Trainer.fit" because of: Parameter requested to be skipped.
2024-10-29 14:09:20,402 - LightningArgumentParser - DEBUG - Skipping parameter "model" from "__main__.Trainer.validate" because of: Parameter requested to be skipped.
2024-10-29 14:09:20,402 - LightningArgumentParser - DEBUG - Skipping parameter "dataloaders" from "__main__.Trainer.validate" because of: Parameter requested to be skipped.
2024-10-29 14:09:20,403 - LightningArgumentParser - DEBUG - Skipping parameter "datamodule" from "__main__.Trainer.validate" because of: Parameter requested to be skipped.
2024-10-29 14:09:20,425 - LightningArgumentParser - DEBUG - Skipping parameter "model" from "__main__.Trainer.test" because of: Parameter requested to be skipped.
2024-10-29 14:09:20,425 - LightningArgumentParser - DEBUG - Skipping parameter "dataloaders" from "__main__.Trainer.test" because of: Parameter requested to be skipped.
2024-10-29 14:09:20,426 - LightningArgumentParser - DEBUG - Skipping parameter "datamodule" from "__main__.Trainer.test" because of: Parameter requested to be skipped.
2024-10-29 14:09:20,447 - LightningArgumentParser - DEBUG - Skipping parameter "model" from "__main__.Trainer.predict" because of: Parameter requested to be skipped.
2024-10-29 14:09:20,447 - LightningArgumentParser - DEBUG - Skipping parameter "dataloaders" from "__main__.Trainer.predict" because of: Parameter requested to be skipped.
2024-10-29 14:09:20,447 - LightningArgumentParser - DEBUG - Skipping parameter "datamodule" from "__main__.Trainer.predict" because of: Parameter requested to be skipped.
2024-10-29 14:09:20,467 - LightningArgumentParser - DEBUG - Skipping parameter "datamodule" from "__main__.Trainer.cmd" because of: Parameter requested to be skipped.
2024-10-29 14:09:20,468 - LightningArgumentParser - DEBUG - Loaded parser defaults: Namespace(config=None, subcommand=None)
2024-10-29 14:09:20,748 - LightningArgumentParser - DEBUG - Loaded parser defaults: Namespace(config=None, seed_everything=True, trainer=Namespace(accelerator='auto', strategy='auto', devices='auto', num_nodes=1, precision=None, logger=None, callbacks=None, fast_dev_run=False, max_epochs=None, min_epochs=None, max_steps=-1, min_steps=None, max_time=None, limit_train_batches=None, limit_val_batches=None, limit_test_batches=None, limit_predict_batches=None, overfit_batches=0.0, val_check_interval=None, check_val_every_n_epoch=1, num_sanity_val_steps=None, log_every_n_steps=None, enable_checkpointing=None, enable_progress_bar=None, enable_model_summary=None, accumulate_grad_batches=1, gradient_clip_val=None, gradient_clip_algorithm=None, deterministic=None, benchmark=None, inference_mode=True, use_distributed_sampler=True, profiler=None, detect_anomaly=False, barebones=False, plugins=None, sync_batchnorm=False, reload_dataloaders_every_n_epochs=0, default_root_dir=None), model=None, data=None, optimizer=None, lr_scheduler=None)
2024-10-29 14:09:20,756 - LightningArgumentParser - DEBUG - Parsed command line arguments: []
2024-10-29 14:09:20,756 - LightningArgumentParser - DEBUG - Loaded parser defaults: Namespace(config=None, seed_everything=True, trainer=Namespace(accelerator='auto', strategy='auto', devices='auto', num_nodes=1, precision=None, logger=None, callbacks=None, fast_dev_run=False, max_epochs=None, min_epochs=None, max_steps=-1, min_steps=None, max_time=None, limit_train_batches=None, limit_val_batches=None, limit_test_batches=None, limit_predict_batches=None, overfit_batches=0.0, val_check_interval=None, check_val_every_n_epoch=1, num_sanity_val_steps=None, log_every_n_steps=None, enable_checkpointing=None, enable_progress_bar=None, enable_model_summary=None, accumulate_grad_batches=1, gradient_clip_val=None, gradient_clip_algorithm=None, deterministic=None, benchmark=None, inference_mode=True, use_distributed_sampler=True, profiler=None, detect_anomaly=False, barebones=False, plugins=None, sync_batchnorm=False, reload_dataloaders_every_n_epochs=0, default_root_dir=None), model=None, data=None, optimizer=None, lr_scheduler=None)
2024-10-29 14:09:20,760 - LightningArgumentParser - ERROR - Validation failed: Key "cmd.model" is required but not included in config object or its value is None.
2024-10-29 14:09:20,760 - LightningArgumentParser - DEBUG - Debug enabled, thus raising exception instead of exit.
Traceback (most recent call last):
  File "lib\site-packages\jsonargparse\_core.py", line 1050, in check_required
    raise TypeError
TypeError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "lib\site-packages\jsonargparse\_core.py", line 1095, in check_config
    check_required(cfg, self)
  File "lib\site-packages\jsonargparse\_core.py", line 1057, in check_required
    check_required(cfg.get(subcommand), subparser, subcommand + ".")
  File "lib\site-packages\jsonargparse\_core.py", line 1052, in check_required
    raise TypeError(
TypeError: Key "cmd.model" is required but not included in config object or its value is None.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "lib\site-packages\jsonargparse\_core.py", line 400, in parse_args
    parsed_cfg = self._parse_common(
  File "lib\site-packages\jsonargparse\_core.py", line 325, in _parse_common
    self.check_config(cfg, skip_required=skip_required)
  File "ib\site-packages\jsonargparse\_core.py", line 1103, in check_config
    raise type(ex)(message) from ex
TypeError: Validation failed: Key "cmd.model" is required but not included in config object or its value is None.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "script.py", line 24, in <module>
    main()
  File "script.py", line 21, in main
    CLI(trainer_class=Trainer)
  File "lib\site-packages\pytorch_lightning\cli.py", line 383, in __init__
    self.parse_arguments(self.parser, args)
  File "lib\site-packages\pytorch_lightning\cli.py", line 534, in parse_arguments
    self.config = parser.parse_args(args)
  File "lib\site-packages\jsonargparse\_deprecated.py", line 123, in patched_parse
    cfg = parse_method(*args, _skip_check=_skip_check, **kwargs)
  File "ib\site-packages\jsonargparse\_core.py", line 409, in parse_args
    self.error(str(ex), ex)
  File lib\site-packages\jsonargparse\_core.py", line 1015, in error
    raise argument_error(message) from ex
argparse.ArgumentError: Validation failed: Key "cmd.model" is required but not included in config object or its value is None.

Environment

Current environment

• CUDA:
  - GPU: None
  - available: False
  - version: None
• Lightning:
  - lightning-utilities: 0.11.8
  - pytorch-lightning: 2.4.0
  - torch: 2.5.0
  - torchmetrics: 1.5.1
• Packages:
  - aiohappyeyeballs: 2.4.3
  - aiohttp: 3.10.10
  - aiosignal: 1.3.1
  - antlr4-python3-runtime: 4.9.3
  - astroid: 3.3.5
  - async-timeout: 4.0.3
  - attrs: 24.2.0
  - autocommand: 2.2.2
  - backports.tarfile: 1.2.0
  - black: 24.10.0
  - cfgv: 3.4.0
  - click: 8.1.7
  - codespell: 2.3.0
  - colorama: 0.4.6
  - dill: 0.3.9
  - distlib: 0.3.9
  - docstring-parser: 0.16
  - exceptiongroup: 1.2.2
  - filelock: 3.16.1
  - frozenlist: 1.5.0
  - fsspec: 2024.10.0
  - identify: 2.6.1
  - idna: 3.10
  - importlib-metadata: 8.0.0
  - importlib-resources: 6.4.5
  - inflect: 7.3.1
  - iniconfig: 2.0.0
  - isort: 5.13.2
  - jaraco.collections: 5.1.0
  - jaraco.context: 5.3.0
  - jaraco.functools: 4.0.1
  - jaraco.text: 3.12.1
  - jinja2: 3.1.4
  - jsonargparse: 4.33.2
  - lightning-utilities: 0.11.8
  - markupsafe: 3.0.2
  - mccabe: 0.7.0
  - more-itertools: 10.3.0
  - mpmath: 1.3.0
  - multidict: 6.1.0
  - mypy-extensions: 1.0.0
  - networkx: 3.4.2
  - nodeenv: 1.9.1
  - numpy: 1.26.4
  - omegaconf: 2.3.0
  - pace-fpo-iris-aiml: 0.1.0
  - packaging: 24.1
  - pathspec: 0.12.1
  - pip: 24.2
  - platformdirs: 4.3.6
  - pluggy: 1.5.0
  - pre-commit: 4.0.1
  - propcache: 0.2.0
  - pylint: 3.3.1
  - pytest: 8.3.3
  - pytorch-lightning: 2.4.0
  - pyyaml: 6.0.2
  - setuptools: 75.2.0
  - sympy: 1.13.1
  - tomli: 2.0.2
  - tomlkit: 0.13.2
  - torch: 2.5.0
  - torchmetrics: 1.5.1
  - tqdm: 4.66.6
  - typeguard: 4.3.0
  - typeshed-client: 2.7.0
  - typing-extensions: 4.12.2
  - virtualenv: 20.27.0
  - wheel: 0.43.0
  - yarl: 1.17.0
  - zipp: 3.19.2
• System:
  - OS: Windows
  - architecture:
  - 64bit
  - WindowsPE
  - processor: Intel64 Family 6 Model 142 Stepping 12, GenuineIntel
  - python: 3.10.11
  - release: 10
  - version: 10.0.26100

More info

No response

[mauvilsa]

This is not a bug. Yes you can add subcommands, but the way LightningCLI is currently implemented, the model is always required. This does not depend on the signature of the method. Support for this I guess it could be added, but it would be a feature request.

[enrico-stauss]

Understood, even though I think it could at least be documented a little more clearly.
For me it's not an issue to just provide the model and swallow it in the kwargs, but it took me a bit to figure out what's going on when I didn't provide the model.