Implement Slack notifications for failing actions

[cka-y]

Describe the problem

Add a Slack notification system for GitHub Actions to alert the team when an action fails.

Proposed solution

• Action Inventory: Create a comprehensive list of all GitHub Actions within the repository.
• Importance Levels: Assign an importance level to each action (e.g., critical, high, medium, low) to determine notification urgency.
• Notification Setup: Configure Slack notifications to trigger when a failure occurs, with different alert formats based on the assigned importance level.

[jcpitre]

Something like this was done: #447

[cka-y]

Workflow Prioritization Table:

Group	Environment	Trigger	Severity	Workflow Dependencies	Notes
API Deployment				[2] Build & Test, Integration Tests	Manually triggered to give the person initiating control, reducing the risk of unintended issues.
				[1] Build & Test	Part of the broader release workflow and not triggered independently.
				[1] Build & Test	Managed within the release workflow, so failures are generally caught during release.
Build & Test				0	Runs on PRs to check code quality, with issues visible directly in the PR for immediate resolution.
Dataset Batch Deployment				0	Runs after merging to main. Failure notifications are essential to prevent blocking the PROD release.
				0	Part of the QA release workflow; doesn't run independently. Issues are generally resolved within the broader release context.
				0	Similar to QA, runs within the PROD release and is covered by release-level checks rather than needing separate prioritization.
Database Deployment				0	Part of the QA release workflow, where issues are generally handled within the broader release process.
				0	Similar to QA, runs within the PROD release and is managed by release-level checks.
Database Update				0	High priority in DEV as failures here would block further testing and development, impacting the release cycle.
				0	High priority, as the QA database supports integration tests on each PR. PRs may fail unexpectedly if the database isn't updated.
				0	Medium priority in PROD, as it can be triggered through a repository dispatch, allowing quicker data updates in the database and keeping MobilityDatabase in sync with the catalog.
Integration Tests	N/A			0	Low priority because it doesn't run independently and is part of other workflows.
Integration Tests - PR				0	Runs during PRs to verify integration compatibility; any failures are visible in PRs for immediate resolution.
Release				[6] Database Deployment, Database Update, API Deployment, Dataset Batch Deployment, Integration Tests, Web Deployment	High priority because it's the final step before production, affecting all other workflows in PROD. Failures here block the release.
				[6] Database Deployment, Database Update, API Deployment, Dataset Batch Deployment, Integration Tests, Web Deployment	High priority for QA, as it impacts test stability. If this fails, it suggests potential issues for the PROD release, acting as a blocker for the release.
Load Test				0	Ensures system stability under load; failures are high priority as they indicate load handling issues requiring immediate attention.
Typescript Generator Check				0	Low priority check during PRs to ensure TypeScript code generation accuracy; issues are visible directly in the PR review process.
Validator Update				0	High priority for updating validators in PROD due to its impact on validation processes across production systems.
Web App Deployer				0	Low priority in DEV as it’s manually controlled; any issues can be addressed by developers immediately.
				0	Low priority in QA, as it’s part of broader testing processes and can be managed within QA-specific workflows.
				0	Low priority in PROD due to controlled manual triggers; issues are typically addressed within the controlled PROD environment.

Based on the review and prioritization of our workflows, I recommend implementing Slack notifications for medium and high-priority workflows to ensure timely attention to critical issues.

Proposal:

• Scope of Notifications: Trigger notifications only for workflows marked as "Medium" and "High" priority, as these workflows have significant implications on production stability, database synchronization, and testing integrity.
• Notification Content: Each notification should include:
  • The name of the workflow.
  • The specific step within the workflow where the failure occurred.
  • A summary message tailored to the priority level, such as:
    • "🚧 Medium Priority Alert: [Workflow Name] encountered an issue at [Step]. This may affect ongoing integration processes."
    • "🚨 High Priority Alert: [Workflow Name] failed at [Step]. Immediate attention is required to avoid production impact."

[davidgamez]

To reduce noise, I suggest not triggering a notification by default in case of manual executions. If someone is running the workflow, this person will follow up on the status.

[davidgamez]

@cka-y, could you add this table to the repo documentation so we can keep it updated?