If one protocol fails, there should be an option to stop all of them

[lilyminium]

Describe the bug

When working with a dataset of multiple properties, one can set off a benchmarking or fitting workflow where e.g. 80% properties succeed, but 20% fail. When this happens, especially in a force field fit, an early-stopping mechanism would be helpful to avoid wasting compute and wall time on computing other properties for a run that's ultimately doomed to fail.

I'm not sure how hard this is to do though.

To Reproduce

One way to set up a test for this is:

• kick off a PreequilibratedSimulation workflow where only part of the dataset has been pre-equilibrated
• or run any workflow on a set of properties where one property cannot be assigned parameters by a published OpenFF force field

Output

Computing environment (please complete the following information):

• Operating system
• Output of running conda list

Additional context

[mattwthompson]

I'm not sure how to get started on this - no doubt a useful feature, though

[lilyminium]

Yes, it would be a non-zero amount of work. I'd probably start by looking at _process_results() (https://github.com/openforcefield/openff-evaluator/blob/main/openff/evaluator/layers/layers.py#L351) which catches errors and processes them server-side, or in _query_request_status() client-side (

openff-evaluator/openff/evaluator/server/server.py

Line 279 in 7b5714f

request_results.exceptions.extend(batch.exceptions)

). The latter is probably less work and would catch things quickly with a short polling interval.

Not super urgent if you keep an eye on things, but I did just run something for 14 hours before realising a property had crashed within 2 😞