LightningCLI: Variable interpolation and command line arguments

[mikcnt]

🐛 Bug

Hi! I'm not sure this is actually a bug or if the one doing something wrong, but here's the problem.

I'm using OmegaConf’s variable interpolation to link one parameter in the configuration file to a second one, which I want to provide with the command line. The problem is that, when using the command line to provide one of the two parameters, the second one, linked to the first one with variable interpolation, doesn't change.

To Reproduce

1. config.yaml

foo: default_value
bar: ${foo}

2. main.py

from pytorch_lightning.utilities.cli import LightningCLI

class MyLightningCLI(LightningCLI):
    def add_arguments_to_parser(self, parser):
        parser.add_argument("--foo", type=str)
        parser.add_argument("--bar", type=str)


if __name__ == "__main__":
    cli = MyLightningCLI(parser_kwargs={"parser_mode": "omegaconf"})

3. Run

python main.py fit \
    --config=config.yaml \
    --foo=some_value \
    --print_config

4. Result

The result of this is that the foo parameter is equal to the value assigned with the command line, some_value, but the bar parameter isn't, and it is instead keeping its default value set in the config file (default_value).

seed_everything: null
trainer:
  logger: true
  checkpoint_callback: null
  ...
foo: some_value
bar: default_value
...

Expected behavior

I expected the bar parameter to be equal to the foo parameter, that is to have the value some_value:

seed_everything: null
trainer:
  logger: true
  checkpoint_callback: null
  ...
foo: some_value
bar: some_value
...

Environment

• PyTorch Lightning Version (e.g., 1.5.0): 1.6.4
• PyTorch Version (e.g., 1.10): 1.10.2
• Python version (e.g., 3.9): 3.9
• OS (e.g., Linux): Mac
• GPU models and configuration: None (CPU)
• How you installed PyTorch (conda, pip, source): pip

cc @carmocca @mauvilsa @akihironitta

[lodo1995]

I agree that this would be a very nice feature to have. Unfortunately, what you are seeing is a limitation of jsonargparse, the library that Lightning uses to handle both command line arguments and configuration files.

Their documentation on variable interpolation explicitly states that (emphasis mine)

The parser_mode='omegaconf' provides support for OmegaConf’s variable interpolation in a single yaml file. It is not possible to do interpolation across multiple yaml files or in an isolated individual command line argument.

[mikcnt]

@lodo1995 Thanks a lot for pointing that out! I honestly didn't notice that, that's unfortunate 😭 Do you happen to know if there is some workaround for that?

[lodo1995]

@mikcnt I guess there must be a smart way to abuse argument linking to achieve what you want, but I'm not sure how.

Alternatively, you can "brute force" it by overriding before_instantiate_classes, manually looking for the pattern ${*} in every string argument and doing the appropriate lookup and replacement based on other arguments. Watch out for infinite recursion if you try this!

But no, I don't know of any simple workaround.

[mauvilsa]

@mikcnt with your example using foo and bar it is not known what was the real use case behind it. Could you please your actual case? There are many cases in which link_arguments should be used, but people go for variable interpolation because it is a better known concept.

[mikcnt]

@lodo1995 thanks a lot for the tips! I'll try something out, even though I still want to keep things simple.

@mauvilsa Sure, my example isn't really explanatory. My use case is similar to the the first example of #13558. Basically, I want to provide one single parameter through the command line (let's say the name of the experiment), and I want to use that to fill some other parameter as well (e.g., wandb run, directory for the checkpoints, etc).

[mauvilsa]

@mauvilsa Sure, my example isn't really explanatory. My use case is similar to the the first example of #13558. Basically, I want to provide one single parameter through the command line (let's say the name of the experiment), and I want to use that to fill some other parameter as well (e.g., wandb run, directory for the checkpoints, etc).

To me it seems that what you want is not interpolation. There is a single parameter, and its value will always be used in several specific places. It is not responsibility of who uses the CLI to do this in a config file. Is your intention that the loggers are always WandbLogger and CSVLogger, or you want the amount and which loggers to be configurable? link_arguments does not support linking to a target which is an element of a list, like the logger when there are multiple. But if the loggers are fixed I think it might be already possible to do what you want.

[mikcnt]

To me it seems that what you want is not interpolation. There is a single parameter, and its value will always be used in several specific places. It is not responsibility of who uses the CLI to do this in a config file.

I see what you're saying. The reason why I was thinking about interpolation is that I didn't want to necessarily use the same parameter for every single location, but most definitely a slight modification of this, and that was something easily achievable with interpolation. Nevertheless, I think that would suffice.

Is your intention that the loggers are always WandbLogger and CSVLogger, or you want the amount and which loggers to be configurable?

Yes, I'll use only WandbLogger for every experiment.

[mauvilsa]

There were some cases that link_arguments did not support which are required for your logger example. I implemented them but not yet in a release. To try out the code below you will need to install jsonargparse from source.

If the CLI will have one or more fixed loggers, then better to make the choice of loggers non-configurable. Only configurable would be the parameters of each fixed logger. For a single logger, this could be done as:

def save_dir_from_experiment_name(name):
    return f'logging-{name}'

class MyCLI(LightningCLI):
    def add_arguments_to_parser(self, parser):
        parser.add_argument('--experiment_name', type=str, required=True)
        parser.add_class_arguments(TensorBoardLogger, 'tensorboard')
        parser.link_arguments('experiment_name', 'tensorboard.save_dir', compute_fn=save_dir_from_experiment_name)
        parser.link_arguments('tensorboard', 'trainer.logger', apply_on='instantiate')

The save_dir_from_experiment_name function is just to illustrate that linked values can be modified, which is what I understood you wanted. The config would have a structure like:

trainer:
  ... # excluding logger
experiment_name: ...
tensorboard:
  ... # excluding save_dir

For multiple loggers it could be as:

def as_list(*items):
    return [*items]

class MyCLI(LightningCLI):
    def add_arguments_to_parser(self, parser):
        parser.add_argument('--experiment_name', type=str, required=True)
        parser.add_class_arguments(TensorBoardLogger, 'loggers.tensorboard')
        parser.add_class_arguments(CSVLogger, 'loggers.csv')
        parser.link_arguments('experiment_name', 'loggers.tensorboard.save_dir')
        parser.link_arguments('experiment_name', 'loggers.csv.save_dir')
        parser.link_arguments(
            ('loggers.tensorboard', 'loggers.csv'),
            'trainer.logger',
            apply_on='instantiate',
            compute_fn=as_list,
        )

And the config structure:

trainer:
  ... # excluding logger
experiment_name: ...
loggers:
  tensorboard:
    ... # excluding save_dir
  csv:
    ... # excluding save_dir

[mikcnt]

Sorry for the superlate response. That's exactly what I was searching for! Thanks for helping out and taking the time to write a comprehensive guide :).