bug(sim-flow): failed cocotb or failed simulation runtime is not propagate as an error by sim flow

[tymonx]

Some HDL simulators don't return a non-zero return code (!= 0) when something failed during simulation runtime. Including failed cocotb tests or failed cocotb initialization.

Cocotb Runner class is handling that by inspecting generated JUnit XML file. It will fail if:

• Generated JUnit XML file results.xml is not present
• Generated JUnix XML file results.xml contains non-zero failures or errors in <testsuite> XML elements

Something similar could be implemented in Edalize.

References:

• cocotb/src/cocotb_tools/runner.py
• cocotb/src/cocotb_tools/check_results.py

This should be implemented in edalize/flows/sim.py.

To reproduce it:

• Git clone https://gitlab.com/tymonx/pytest-cocotb-fusesoc repository
• Edit assertion in tests/hdl/dff/test_dff.py test from assert dut.q.value == expected_val to assert dut.q.value != expected_val
• Run simulation via fusesoc pytest -s -v -rA

The test_dff test is invoking fusesoc run dff command -> src/pytest_cocotb/fusesoc/plugin.py.

    run(["fusesoc", "run", hdl.toplevel], check=True)

It should fail not pass. The check=True argument will tell the subprocess.run function to raise an exception when invoked command returned a non-zero status code (error).