perilog: raise non-fatal job exception on epilog failure

Problem: Job epilog failures do not raise job exceptions. This made
sense when the epilog was used for administrative purposes, but now
that most of that has moved to housekeeping, the epilog is meant to
include cleanup necessary for job completion, and therefore an epilog
failure should be reflected in the job eventlog and the user notified.

Raise a non-fatal job exception on epilog failure. A non-fatal
exception is used so that it does influence the job status while
still allowing the exception to be logged.

Fixes #6249

Author: grondo

src/modules/job-manager/plugins/perilog.c

@@ -438,13 +438,28 @@ static void emit_finish_event (struct perilog_proc *proc,
                             status);
     }
     else {
+        /*
+         *  Raise a non-fatal job exception on failure so that the job
+         *   status reflects the status of the user job and not the status
+         *   of the epilog.
+         */
+        if ((status != 0 && !proc->canceled) || proc->cancel_timeout) {
+            char *errmsg = exception_errmsg (proc, status);
+            if (flux_jobtap_raise_exception (proc->p,
+                                             proc->id,
+                                             "epilog",
+                                             2,
+                                             "%s",
+                                             errmsg ?
+                                             errmsg :
+                                             "job epilog failed") < 0)
+                    flux_log_error (flux_jobtap_get_flux (proc->p),
+                                    "epilog-finish: jobtap_raise_exception");
+            free (errmsg);
+        }
         /*
          *  Epilog complete: unsubscribe this plugin from the
          *   finished job and post an epilog-finish event.
-         *
-         *  No job exception is raised since the job is already exiting,
-         *   and it is expected that the actual epilog script will
-         *   drain nodes or take other action on failure if necessary.
          */
         flux_jobtap_job_unsubscribe (proc->p, proc->id);
         if (flux_jobtap_epilog_finish (proc->p,