Bug: Integration tests failing to send command to GDS

[nateinaction]

Seeing builds failing when integration tests are failing to send commands to GDS.

============================= test session starts ==============================
platform linux -- Python 3.13.7, pytest-8.3.3, pluggy-1.5.0
rootdir: /home/broncospace/Documents/proves-cd/actions-runner/_work/proves-core-reference/proves-core-reference
plugins: fprime-gds-4.0.1
collected 12 items
FprimeZephyrReference/test/int/burnwire_test.py ..                       [ 16%]
FprimeZephyrReference/test/int/imu_manager_test.py ....                  [ 50%]
FprimeZephyrReference/test/int/imu_manager_test.py ....                  [ 50%]
FprimeZephyrReference/test/int/rtc_test.py ...                           [ 75%]
FprimeZephyrReference/test/int/watchdog_test.py E..                      [100%]
==================================== ERRORS ====================================
______________ ERROR at setup of test_01_watchdog_telemetry_basic ______________
fprime_test_api = <fprime_gds.common.testing_fw.api.IntegrationTestAPI object at 0x7f40b2a2c590>
    @pytest.fixture(autouse=True)
    def ensure_watchdog_running(fprime_test_api: IntegrationTestAPI):
        """Fixture to ensure watchdog is started before and after each test"""
>       start_watchdog(fprime_test_api)
FprimeZephyrReference/test/int/watchdog_test.py:17: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
FprimeZephyrReference/test/int/watchdog_test.py:24: in start_watchdog
    fprime_test_api.send_and_assert_command(
fprime-venv/lib/python3.13/site-packages/fprime_gds/common/testing_fw/api.py:572: in send_and_assert_command
    results = self.send_and_assert_event(command, args, events, timeout=timeout)
fprime-venv/lib/python3.13/site-packages/fprime_gds/common/testing_fw/api.py:635: in send_and_assert_event
    return self.assert_event_sequence(events, start=start, timeout=timeout)
fprime-venv/lib/python3.13/site-packages/fprime_gds/common/testing_fw/api.py:1072: in assert_event_sequence
    self.__assert_pred("Event sequence", len_pred, len(results), msg)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <fprime_gds.common.testing_fw.api.IntegrationTestAPI object at 0x7f40b2a2c590>
name = 'Event sequence assertion', predicate = x == 2, value = 1
msg = 'checks if sequence search found every event', expect = False
    def __assert_pred(self, name, predicate, value, msg="", expect=False):
        """
        Helper to assert that a value satisfies a predicate. Includes arguments to provide more
        descriptive messages in the logs.
        Args:
            name: short name describing the check
            predicate: an instance of predicate to determine if the assert is successful
            value: the object evaluated by the predicate
            msg: a string message to describe what the assert is checking
            expect: a boolean value: True will be have as an expect and not raise an assertion.
        Returns:
            True if the assertion was successful, False otherwise
        """
        name = name + (" expectation" if expect else " assertion")
        pred_msg = predicates.get_descriptive_string(value, predicate)
        if predicate(value):
            ast_msg = f"{name} succeeded: {msg}\nassert {pred_msg}"
            self.__log(ast_msg, TestLogger.GREEN)
            if not expect:
                assert True, pred_msg
            return True
        ast_msg = f"{name} failed: {msg}\nassert {pred_msg}"
        if not expect:
            self.__log(ast_msg, TestLogger.RED)
>           assert False, pred_msg
E           AssertionError: F(1), where F(x) evaluates
E             	 x == 2
E           assert False
fprime-venv/lib/python3.13/site-packages/fprime_gds/common/testing_fw/api.py:1422: AssertionError
---------------------------- Captured stdout setup -----------------------------
17:14:25.464438 [Test API] [STARTING CASE] test_01_watchdog_telemetry_basic
17:14:25.465505 [Test API] Clearing Test Histories
17:14:25.466520 [Test API] Sending Command: ReferenceDeployment.watchdog.START_WATCHDOG []
17:14:25.467658 [Test API] Beginning a sequence search of 2 items.
17:14:25.468552 [Test API] Sequence search now awaiting for at most 5 s.
17:14:25.513143 [GDS] Received EVR: 2025-10-10T17:14:25.995485: ReferenceDeployment.watchdog.WatchdogStart (268521472) (2(0)-1760141665:995485) EventSeverity.ACTIVITY_HI : Watchdog started
17:14:25.525786 [GDS] Received EVR: 2025-10-10T17:14:25.997579: CdhCore.cmdDisp.OpCodeDispatched (16777217) (2(0)-1760141665:997579) EventSeverity.COMMAND : Opcode 0x10015000 dispatched to port 9
17:14:25.531826 [GDS] Received EVR: 2025-10-10T17:14:25.007402: CdhCore.cmdDisp.OpCodeCompleted (16777218) (2(0)-1760141665:7402) EventSeverity.COMMAND : Opcode 0x10015000 completed
17:14:25.532847 [Test API] API detected out of order evrs!
Received First:2025-10-10T17:14:25.997579: CdhCore.cmdDisp.OpCodeDispatched (16777217) (2(0)-1760141665:997579) EventSeverity.COMMAND : Opcode 0x10015000 dispatched to port 9
Received Second:2025-10-10T17:14:25.007402: CdhCore.cmdDisp.OpCodeCompleted (16777218) (2(0)-1760141665:7402) EventSeverity.COMMAND : Opcode 0x10015000 completed
17:14:25.569306 [Test API] Sequence search found the next item: 2025-10-10T17:14:25.997579: CdhCore.cmdDisp.OpCodeDispatched EventSeverity.COMMAND : Opcode 0x10015000 dispatched to port 9
17:14:30.469627 [Test API] Sequence search timed out and ended unsuccessfully.
17:14:30.470919 [Test API] Event sequence assertion failed: checks if sequence search found every event
assert F(1), where F(x) evaluates
	 x == 2
=========================== short test summary info ============================
ERROR FprimeZephyrReference/test/int/watchdog_test.py::test_01_watchdog_telemetry_basic - AssertionError: F(1), where F(x) evaluates
  	 x == 2
assert False
========================= 11 passed, 1 error in 33.73s =========================
make: *** [Makefile:72: test-integration] Error 1

[nateinaction]

Resolved by #38. Closing.