Fix dispatcher crash on gevent LoopExit exceptions

[deepssin]

Handle gevent LoopExit exceptions gracefully to prevent dispatcher crashes. Add exception handling in main loop and lock_machines() call, with loop exit counter (max 10) to prevent infinite restarts. Isolate child processes using start_new_session=True so job supervisors continue running independently if dispatcher encounters exceptions.

[kshtsk]

Do we have any reference ticket in tracker for this issue?

[zmc]

This looks like a really well thought out changeset! Considering the dispatcher is difficult to fully validate with unit tests, can you describe briefly how you've tested this so far? Thanks!

[deepssin]

Do we have any reference ticket in tracker for this issue?
@kshtsk There isn’t an existing tracker ticket for this issue — I discovered it while debugging repeated dispatcher crashes in my local OpenStack-based Teuthology environment. The crashes were consistently triggered by gevent.LoopExit surfacing from the main loop when the dispatch cycle completed unexpectedly.

[deepssin]

This looks like a really well thought out changeset! Considering the dispatcher is difficult to fully validate with unit tests, can you describe briefly how you've tested this so far? Thanks!

@zmc For testing, I validated the changes in my OpenStack setup by:

1. Running multiple dispatcher cycles with varied queue states (empty queue, repeated job failures).
2. Reproducing the earlier crash scenario and confirming that the dispatcher no longer exits on LoopExit.
3. Verifying that child processes continue independently and the supervisor remains stable with the added loop-exit counter logic.
   Given that dispatcher behaviour is difficult to cover via unit tests, these manual integration tests helped ensure stability.

[zmc]

This looks like a really well thought out changeset! Considering the dispatcher is difficult to fully validate with unit tests, can you describe briefly how you've tested this so far? Thanks!

@zmc For testing, I validated the changes in my OpenStack setup by:

1. Running multiple dispatcher cycles with varied queue states (empty queue, repeated job failures).

2. Reproducing the earlier crash scenario and confirming that the dispatcher no longer exits on LoopExit.

3. Verifying that child processes continue independently and the supervisor remains stable with the added loop-exit counter logic.
   Given that dispatcher behaviour is difficult to cover via unit tests, these manual integration tests helped ensure stability.

Excellent - thanks for testing so thoroughly!

[zmc]

Deployed just now on teuthology.front.sepia.ceph.com without issues!