composable config (multiple config files, each with a different section)

[jpcbertoldo]

Disclaimer: I tried to create a discussion about this to first check if the feature already exists but the "new" button was not working.

problem

Consider this scenario: I want to test 2 hyperparameter sets on 2 models on 2 datasets, each experiment with 5 seeds, which gives me 2 x 2 x 2 x 5 = 40 config files -- and I do want to keep files for the sake of keeping it easily reproducible.

solution

Instead of generating 8 conffig files, I would like to put the each section of the config, so I would have 2 + 2 + 2 + 5 = 11 config files, and I would make calls to the cli with the cross product combination of files at the command line level.

alternative

Generate temporary file with the full config from the pieces as described above.

[ashwinvaidya17]

From what I understand you want to perform a grid search on the combination. While multiple seeds is not supported yet, you can still try looking into the benchmarking script https://openvinotoolkit.github.io/anomalib/guides/benchmarking.html. Let me know if this works for you.

[jpcbertoldo]

I realized (long time later) it's not really that what I meant as a feature request.

[jpcbertoldo]

Thanks @ashwinvaidya17, I will check that. (Haven't read yet) Although I think the feature would still be useful for the sake of maintenance of these config files.

[jpcbertoldo]

perhaps this feature -- composing a config file from multiple (and smaller) config files -- could go along with #827?

[samet-akcay]

perhaps this feature -- composing a config file from multiple (and smaller) config files -- could go along with #827?

@jpcbertoldo, if I understand you correctly, PL CLI supports passing multiple config files such as data, model and trainer configs separately. It will be part of the CLI improvements.

[jpcbertoldo]

A ha, awesome! Ok, didn't know that.
But that's not enable yet, is it? I believe I tried to do that at some point and didn't get it.

[samet-akcay]

not yet in anomalib unfortunately