An exception from "run_if" function terminates test with PASS outcome

[eli-zr]

Hi All,

Wanted to pay your attention to a problematic behavior when using run_if phase option.
Yes, an implementation of run_if should end up with a boolean result however when this is not the case and an exception is raised (not deliberately), test will stop with a PASS outcome. Teardowns will be skipped as well.

run_if function is not a phase and not part of the test however the behavior above is a vulnerability and in such I think test should stop

This could be reproduced with following unit test that will pass:

def run_if_with_exception():
raise Exception("run_if_with_exception")

_tr = None
def final(tr):
global _tr
_tr = tr

def test_execute_phase_with_failing_run_if(self):

@openhtf.PhaseOptions(run_if=run_if_with_exception)
def phase():
  pass

test = openhtf.Test(phase)
test.add_output_callbacks(final)
test.execute()
self.assertEqual(_tr.outcome, test_record.Outcome.PASS)

[glados-verma]

Thanks for the fix! #1203