kfold cross-val: any way to average partial-epoch metrics across folds?

[pchalasani]

I am trying to use ray-tune for hyper-param opt with PTL. One issue I run into often is that the "best" hyperparams found by ray-tune are not so great once I change the random seed. It's much more useful to find hyperperams such that the average objective across random seeds is optimized.
One thought I had is to define a new objective, say avg_val_auc as the average of the val_auc across multiple folds in kfold cross-validation (I saw the kfold.py example). The important requirement is to have:

• the ability for multiple folds to be trained concurrently, in say different threads or on different cores, and
• the ability to report back (to ray-tune) the averaged val metric after each val_check_interval, i.e. before the entire fold training has completed. This enables ray-tune to kill trials that are looking bad.

Hopefullly that makes sense. Any pointers appreciated.