Run multiple instances of spacy train in parallel

[phlobo]

I would like to run different instances of spacy train with different hyperparameters in parallel. Currently, in my project.yml I pass -o training to the script, so I guess it is currently only possible to track one project.

Is it possible to run spacy train for the same project in parallel and create subfolders for models, e.g., with different timestamps, like training/22-01-25_14-33-10?
If so, what is the syntax in the project.yml to do that?

[pmbaumgartner]

Hey @phlobo - Have you looked at spacy-ray for parallel training? That would be the first thing I'd check out. You could combine this with something like Weights and Biases to do a hyperparameter sweep (example).

We don't have anything pre-made for this in spaCy projects, so for including in a project.yml I'd first try out these commands from a shell and once you've figured out the right sequence, include them as a command.

[adrianeboyd]

To add to what Peter said, one option is to make the output directory a variable and override it on the command-line along with the other settings you're changing. This requires you to add the project directory (. or a full path) to the command before the overrides so that the options get parsed correctly:

spacy project run train . --vars.input_dir data1 --vars.output_dir output1
spacy project run train . --vars.input_dir data2 --vars.output_dir output2

You can schedule/run these jobs using whatever tool you'd like, obviously keeping your CPU/GPU limits in mind while training, and making sure none of the intermediate outputs clobber each other in your project step(s).

spacy-ray is a bit under development and more for adventurous users right now.

[phlobo]

Thank you! I think overriding output dir and GPU number will do the trick for now