multiuser job notification system

[grondo]

@SteVwonder's question about --begin=time reminded me that we also need to support email notification in multi-user instances since it has been indicated that srun's --mail-type is a "widely used option" by users.

However, email as a notification system seems a bit passé, so perhaps it would be more useful to support a more generic notification framework.

Perhaps a high-level "job monitoring" system could be implemented, which could then call into a stack of notification plugins (one of which would implement the email notification). There are a multitude of ways to implement the monitoring system since most of the raw events are already all there in the eventlog. It could be implemented via a broker module or simply as a python script that gets started via rc1.

For reference, the events supported by srun --mail-type include:

NONE, BEGIN, END, FAIL, REQUEUE, ALL (equivalent to BEGIN, END, FAIL, REQUEUE, and STAGE_OUT), STAGE_OUT (burst buffer stage out and teardown completed), TIME_LIMIT, TIME_LIMIT_90 (reached 90 percent of time limit), TIME_LIMIT_80 (reached 80 percent of time limit), and TIME_LIMIT_50 (reached 50 percent of time limit).

[SteVwonder]

which could then call into a stack of notification plugins (one of which would implement the email notification).

I like that a lot! That would be a really nice way to support more "in vogue" communication systems like Slack, Teams, etc.

[grondo]

Another thought: the majority of these notification events could probably be handled by a shell plugin, which might be a lot less work if we just need to get something done.