Support for multi-GPU nodes (3: orchestrate different madevent executables to target different GPUs) #990
Description
As discussed at the recent CMS meeting. This is part 3 of possible multi-GPU development.
Support for multi-GPU nodes (3: orchestrate different madevent executables to target different GPUs)
Note:
- part 1 in Support for multi-GPU nodes (1: choose one of many available GPUs) #836 is simply about choosing a single GPU (should be easy, and useful)
- part 2 in Support for multi-GPU nodes (2: send MEs to more than one GPU from the same madevent executable?) #989 is about sending MEs to more than one GPU from the same executable (complex, and not necessarily useful?)
- part 3 in Support for multi-GPU nodes (3: orchestrate different madevent executables to target different GPUs) #990 is about orchestrating from python/bash the different madevent executables to target different GPUs (not to ocomplex, and sounds very useful?)
This part 3 (which needs part 1) seems to be the most interesting of the lot... not terribly difficult: it may be enough to iterate round-robin through 4 GPUs on the system when sending jobs.
This only makes sense for workflows/processes where MEs dominate, ie very complex processes.