Support for multi-worker K8S cluster [Tracker issue] #10
Description
Evaluate how to adapt the single-k8s node architecture of the PoC to a multi-worker cluster. This involves:
- Switching from the local filesystem mounts for I/O data (V1HostPathVolumeSource), to Persistent Volume Claims linked to network-level storage (e.g., a NFS).
- Add abstractions and dependency-inversion to the code so that both approaches can be used, depending on the required setup: local - single node K8S, or a K8S multi-worker cluster.
- Evaluate how this would solve the current limitations for using distributed computing resources (e.g. GPU).
- Evaluate the combination of multiple image types.