workflow status message: explain why a workflow is stopped

[oliver-sanders]

The UIS provides a "status message" for stopped workflows (because there is no scheduler around to do so). Currently this only differentiates between a workflow which has or has not yet run.

A workflow might stop for several reasons:

• It ran to completion.
• It was requested to stop (clean, kill, now, now-now, stop-after-cycle, stop-after-task).
• It hit the inactivity timeout.
• It stalled and hit the stall timeout.
• It crashed.

It would be immensely helpful to make this context available in the workflow status message.

Unfortunately, there is no obvious way to achieve this. The reason for the shutdown is recorded in the workflow log (except in the niche case that the workflow crashed because the filesystem went offline / died), but it is not recorded in a more machine-readable format.

Perhaps we should save this information (when we are able) to another file. See the same suggestion on cylc/cylc-flow#5701

[oliver-sanders]

The simplest solution I can think of to solve these problems in general, would be to rename the contact file on workflow shutdown, e.g. mv contact contact.old. We could then add a contact file field to contact.old containing the shutdown reason.

In the event that a workflow aborts (e.g. crashes, hists critical IO error or whatnot), the process that discovers the crash (likely the UIS scanning process) can mv contact contact.old; echo 'ShutdownReason=Aborted' >> contact.old.

The UIS would then read in the contact.old file for a stopped workflow and turn the ShutdownReason into the workflow's "status message" (this appears in the workflow toolbar). In the event that the ShutdownReason is Aborted, the UIS could also create a log_record at the CRITICAL level. This would cause a warning light to illuminate for the workflow (see cylc/cylc-ui#2169).

---

See also:

• Renaming the contact file for crash detection - contact: update modified time for better crash detection cylc-flow#5511 (comment)
• contact: detect crashed workflows cylc-flow#4858
• Overall workflow state / completed state cylc-flow#5701

---

Note, this doesn't go as far as cylc/cylc-flow#5701, these suggestions do not add a top-level workflow status (e.g. stopped, running, completed), however, they are a prerequisite for doing this in the future as we need a record of crash detection to differentiate between stopped (whether completed or otherwise) and aborted.

---

Logic for choosing the contact file to hold this info:

• The info we need is derived from the Scheduler state, so we need to preserve this on the FS somewhere.
• We don't really want the UIS to have to open the workflow DB to get this info because we would have to do it for all stopped workflows.
• We are already reading the contact file for running workflows anyway.
• This would also work for extreme abort scenarios (e.g. kill -9).
• This may simplify detection of workflow clean/install as we can easily tell if the workflow dir has changed from the presence of this file.