Simple way to CPU-fork / parallelize a particular evaluation?

[turian]

My GPU gradient steps and epochs are very fast. However, CPU evaluation is a real bottleneck.

How can I fork / parallelize off my CPU evaluation so it doesn't block training, and GPU training can proceed unimpeded?

My early stopping is based upon gnarly single-CPU code that I don't want to touch or modify. So I am trying to figure out how to increase the training time, while not blocking on CPU eval all the time.

[tchaton]

Hey @turian,

My early stopping is based upon gnarly single-CPU code that I don't want to touch or modify
Any chance you could provide more details on this?

Right now, I would say you could skip the first training epochs to save some training time, but it won't scale well.

With a hackier solution, you could run the validation loop aync in a thread/subprocess and wait to collect the result while the next training epoch starts. It would be blocking on the next validation call. This would require more granular knowledge but could provide some speedups.
This would be an interesting feature for Lightning if you managed to have a clean solution.