ci: Add failure notifications to workflows

[NickLarsenNZ]

Description

Add a job at the end of each workflow which only runs on failure to send a slack notitification.

Considerations

We had the following four options to choose from:

1. Manually create threads per workflow, and hard code thread ids into each workflow.
   • Continuous log of build successes/failures along with relevant conversations.
   • Main thread stays clean.
2. Be smat and get a thread-id from a cached artifact, and if it doesn't exist, create a new one.
   • Same as the first option, but without all the manual work. Good for adding new workflows.
3. Create a thread per workflow run (begin workflow, build, manifest, done)
4. Only message on failure (higher signal-to-noise, people might be happy to receive alerts for it since each would require action)

We opted for Option 4 to keep things simple and to increase signal-to-noise (alerts are actionable).

We have a further idea of having each action write some high-level results to a file/output, and have a wrapping action that collates them all and builds the slack payload. This will be a little complicated with matrix jobs.

[soenkeliebau]

lgtm - full disclosure, have only skimmed the code, no way whatsoever of testing it :)