Difference between parameters in train_pipeline and data.train.pipeline

[UniBremenIFPElectronMicroscopy]

Hi,
sorry I'm quite new to this project, but after searching I didn't find a satisfying answer to the following question:
In the config files there are two lists of dictionaries 'train_pipeline' and 'data.train.pipeline' (similar for the test pipeline). I don't understand what is the difference between both. And more important: If I want to change the parameters, do I have to change them in both or in only one of them?
Here: https://mmdetection.readthedocs.io/en/latest/tutorials/config.html it is said that '# pipeline, this is passed by the train_pipeline created before.' So I think they should be the same. But is this true?
I hope this has not been asked somewhere before... Would be great to find an answer. Thank you!

[Maenansyn]

Short answer: Yes. It is the same. Pipeline in the data dict references the other one.

Explanation:
Afaik it's mostly for reasons of clarity and comprehensibility and to remove redundancy. E.g. taken the COCO-detection config:

train_pipeline and test_pipeline are defined beforehand and as they are quite large they would make up most parts of the data-dict config. Therefore they are defined outside and with pipeline=train_pipeline respectively pipeline=test_pipeline referenced and included that way. You can also write it directly into the data-dict config. I haven't bumped into any line of code which referenced train_pipeline and test_pipeline directly. They always reference it via the data dict.

I searched the repo for mentions of "train_pipeline" and all mentions but one are either in configs or in docs. The other one is here. If you don't use this tool, it shouldn't be a problem. And the searched key in the config there seems to be a rather customised key.

But another reason to keep it as it is: redundancy of test_pipeline. This pipeline is used twice for validation and for testing. If you make changes along the way you always have to do it twice and the probability that you forget one of both is quite high over time. A single point of truth/definition is beneficial and your code is less exposed to contain errors.

Summarised: you can probably use it merged into one dict, but I would recommend to use it the way it is, if no specific reasons need you to do otherwise.

PS. You could also outsource the separate parts of the models beforehand and only reference them in the model-dict, too. They can also be quite unclear. But I haven't done it so far.