Auto3DSeg: manual training by bash command vs Python Algo object

[pwrightkcl]

I need to run the Auto3DSeg steps separately rather than with AutoRunner (to parallelise) and am unsure which of the two methods in the documentation to use to run training, or the pros and cons of each.

1. The Algorithm Bundle documentation describes running individual bundles using the bash commands found in their internal README.md files.
2. The Auto3DSeg Reference Python APIs notebook describes loading the bundle algo history, and then iterating through each Algo object in the history and calling its train method (which could be done in parallel).

My current approach is to run data analysis and algorithm generation as per the notebook above and then use the bash commands to run training on the generated algorithms. I then plan to run ensemble as per the notebook. I am unsure if using bash rather than Python for training misses out something critical for ensemble.

In the notebook, after algorithm generation, it describes "Getting and saving the algorithm generation history to the local drive". From the description, it sounds like this is needed if you are moving your files to a remote drive. Should I include this step even if I am not moving my files? I notice that the Ensemble step at the end of the notebook imports this history. Do the generated training scripts update the history, or do I need to do this manually? Should I do it after algorithm generation, after training, or both? Can the documentation be expanded to explain a bit more about how the different steps use the history? If I train using the bash commands, is the history information lost, or can it be extracted later from the various log files? (I have tried to learn more from the API documentation but I found it hard to piece together what the history is and how it is used).

The notebook also uses the algo_to_pickle method, and says that it updates the pickle file with scores after training. It says it is optional, but does not say what cases it might be needed. Is this for hyperparameter optimisation, or do the scores have other uses? If the scores are needed, and I train using the bash commands, can I update the algo pickle file with the scores after training by importing the histories and using get_score()? That is, does get_score() get the scores from the log files, or does it use some ephemeral score variable only available right after training in Python? (If this is necessary, it would seem more efficient to just do everything in Python rather than bash.)

I notice that export_bundle_algo_history also saves to pickle: the only difference I can see is that it exports all the algos in the history object, whereas algo_to_pickle is for a single algo. Is there more to it?

I'm sorry for so many questions, but I hope it helps to enrich the documentation to make Auto3DSeg more accessible to its intended audience, which I think is people who do not have the expertise to build their own MONAI code from scratch.

[pwrightkcl]

I didn't realise there was a separate discussion section for tutorials. If someone in charge thinks this belongs there either move it or let me know and I'll move it.