How do I ensure that my pipeline runs/results are unaffected by changing contents of external config/parameter files?

[noam-insait]

Hi Kedro community!

My team and are I trying to create an optimal setup for running experiments in parallel. Concerningly, it appears as though if we are to change the contents of a parameters file (i.e. conf/local/parameters.yml) during a run, the results of the run may be affected.

For example, let's say I set hyper_tune: False in parameters.yml, and run kedro run in the terminal. If I change the text in parameters.yml to hyper_tune: True (for example, if I am setting up the parameters for my next experiment) before the "training" node begins executing, it appears that Kedro will then read hyper_tune: True. In this example, that would mean that Kedro would execute hyperparameter tuning (despite being instructed not to do so at the beginning of the run).

Am I missing something? Is the answer as simple as passing all parameters to the pipeline one time as a whole (i.e. using a before_pipeline_runs hook) rather than to each node?

[astrojuanlu]

Hi @noam-insait!

Am I missing something? Is the answer as simple as passing all parameters to the pipeline one time as a whole (i.e. using a before_pipeline_runs hook) rather than to each node?

That's the most likely answer, I don't think Kedro supports changing parameters during the run very well. I'm paging @ankatiyar just in case she can add more from the engineering side of things.

[ankatiyar]

I think once the context is created, ideally the run should use the parameters loaded at that point. However, to vary parameters between different runs, you could use runtime parameters. Like so: kedro run --params=hyper_tune=True. This would overwrite the value in your parameters.yml file.
You could also try using different configuration environments with different parameter yaml files in them and then do kedro run --env=env_name