[feature] Add comprehensive tests and docs for CI failure bot #524

- Add complete test suite with comprehensive coverage for all bot functions
- Add user and developer documentation following OpenWISP structure
- Improve error handling in workflow to fail loudly on script errors
- Address CodeRabbit feedback on exception handling
- Follow OpenWISP documentation patterns with Inputs, Usage Example, and Note sections
- Ensure proper QA formatting with noqa directives for long Gemini prompt lines

Closes #524

Author: Srinath0916

.github/scripts/ci_failure_bot.py

@@ -7,7 +7,7 @@
 import os
 import sys
 
-import google.generativeai as genai
+import google.genai as genai
 import requests
 from github import Github, GithubException
 
@@ -127,7 +127,7 @@ def get_pr_diff(self):
                     "body": pr.body or "",
                     "diff": diff_text,
                 }
-        except (GithubException, requests.RequestException) as e:
+        except (GithubException, requests.RequestException, ValueError) as e:
             print(f"Error getting PR diff: {e}")
 
         return None
@@ -164,38 +164,38 @@ def analyze_with_gemini(self, build_logs, pr_diff, workflow_yaml):
         # Gemini prompt - ignore line length for readability
         context = f"""
 ### ROLE
-You are the "Automated Maintainer Gatekeeper." Your goal is to analyze Pull Request (PR) build failures and provide direct, technically accurate, and no-nonsense feedback to contributors.
+You are the "Automated Maintainer Gatekeeper." Your goal is to analyze Pull Request (PR) build failures and provide direct, technically accurate, and no-nonsense feedback to contributors.  # noqa: E501
 
 ### INPUT CONTEXT PROVIDED
 1. **Build Output/Logs:** {build_logs_json}
 2. **YAML Workflow:** {workflow_yaml or "Not available"}
 3. **PR Diff:** {pr_diff_json}
-4. **Project Name:** {project_name}
+4. **Project Name:** {project_name}  # noqa: E501
 5. **Repository:** {repo_url}
-6. **run-qa-checks:** {qa_checks_url}
-7. **runtests:** {runtests_url}
+6. **run-qa-checks:** {qa_checks_url}  # noqa: E501
+7. **runtests:** {runtests_url}  # noqa: E501
 
 ### TASK
-Analyze the provided context to determine why the build failed. Categorize the failure and respond according to the "Tone Guidelines" below.
+Analyze the provided context to determine why the build failed. Categorize the failure and respond according to the "Tone Guidelines" below.  # noqa: E501
 
 ### TONE GUIDELINES
 - **Direct & Honest:** Do not use "fluff" or overly polite corporate language.
 - **Firm Standards:** If a PR is low-effort, spammy, or fails to follow basic instructions, state that clearly.
-- **Action-Oriented:** Provide the exact command or file change needed to fix the error, unless the PR is spammy, in which case we should just declare the PR as potential SPAM and ask maintainers to manually review it.
+- **Action-Oriented:** Provide the exact command or file change needed to fix the error, unless the PR is spammy, in which case we should just declare the PR as potential SPAM and ask maintainers to manually review it.  # noqa: E501
 
 ### RESPONSE STRUCTURE
 1. **Status Summary:** A one-sentence blunt assessment of the failure.
 2. **Technical Diagnosis:**
    - Identify the specific line/test that failed.
    - Explain *why* it failed.
 3. **Required Action:** Provide a code block or specific steps the contributor must take.
-4. **Quality Warning (If Applicable):** If the PR appears to be "spam" (e.g., trivial README changes, AI-generated nonsense, or repeated basic errors), include a firm statement that such contributions are a drain on project resources and ping the maintainers asking them for manual review.
+4. **Quality Warning (If Applicable):** If the PR appears to be "spam" (e.g., trivial README changes, AI-generated nonsense, or repeated basic errors), include a firm statement that such contributions are a drain on project resources and ping the maintainers asking them for manual review.  # noqa: E501
 
 ### EXAMPLE RESPONSE STYLE
-"The build failed because you neglected to update the test suite to match your logic changes. This project does not accept functional changes without corresponding test updates. Refer to the log at line 452. Update tests/logic_test.py before re-submitting. We prioritize high-quality, ready-to-merge code; please ensure you run local tests before pushing."
+"The build failed because you neglected to update the test suite to match your logic changes. This project does not accept functional changes without corresponding test updates. Refer to the log at line 452. Update tests/logic_test.py before re-submitting. We prioritize high-quality, ready-to-merge code; please ensure you run local tests before pushing."  # noqa: E501
 
 Analyze the failure and provide your response:
-"""  # noqa: E501
+"""
 
         try:
             response = self.model.generate_content(context)
@@ -251,37 +251,56 @@ def post_comment(self, message):
 
     def run(self):
         """Main execution flow"""
-        print("CI Failure Bot starting - AI-powered analysis")
-
-        # Double-check: Skip if this is a dependabot PR
         try:
-            workflow_run = self.repo.get_workflow_run(self.workflow_run_id)
-            if workflow_run.actor and "dependabot" in workflow_run.actor.login.lower():
-                print(f"Skipping dependabot PR from {workflow_run.actor.login}")
+            print("CI Failure Bot starting - AI-powered analysis")
+
+            # Double-check: Skip if this is a dependabot PR
+            try:
+                workflow_run = self.repo.get_workflow_run(self.workflow_run_id)
+                if (
+                    workflow_run.actor
+                    and "dependabot" in workflow_run.actor.login.lower()
+                ):
+                    print(f"Skipping dependabot PR from {workflow_run.actor.login}")
+                    return
+            except (GithubException, AttributeError) as e:
+                print(f"Warning: Could not check actor: {e}")
+
+            # Get all context
+            build_logs = self.get_build_logs()
+            pr_diff = self.get_pr_diff()
+            workflow_yaml = self.get_workflow_yaml()
+
+            if not build_logs:
+                print("No build logs found")
                 return
-        except (GithubException, AttributeError) as e:
-            print(f"Warning: Could not check actor: {e}")
 
-        # Get all context
-        build_logs = self.get_build_logs()
-        pr_diff = self.get_pr_diff()
-        workflow_yaml = self.get_workflow_yaml()
+            print("Analyzing failure with Gemini AI...")
 
-        if not build_logs:
-            print("No build logs found")
-            return
+            # Get AI analysis
+            ai_response = self.analyze_with_gemini(build_logs, pr_diff, workflow_yaml)
 
-        print("Analyzing failure with Gemini AI...")
+            # Post intelligent comment
+            self.post_comment(ai_response)
 
-        # Get AI analysis
-        ai_response = self.analyze_with_gemini(build_logs, pr_diff, workflow_yaml)
+            print("CI Failure Bot completed successfully")
 
-        # Post intelligent comment
-        self.post_comment(ai_response)
+        except Exception as e:
+            print(f"CRITICAL ERROR in CI Failure Bot: {e}")
+            print(f"Error type: {type(e).__name__}")
+            import traceback
 
-        print("CI Failure Bot completed")
+            traceback.print_exc()
+            sys.exit(1)
 
 
 if __name__ == "__main__":
-    bot = CIFailureBot()
-    bot.run()
+    try:
+        bot = CIFailureBot()
+        bot.run()
+    except Exception as e:
+        print(f"FATAL: CI Failure Bot crashed: {e}")
+        import traceback
+
+        traceback.print_exc()
+        sys.exit(1)

.github/workflows/ci-failure-bot.yml

@@ -27,9 +27,9 @@ jobs:
           python-version: "3.11"
 
       - name: Install dependencies
-        # Pin dependency versions for reproducibility
+        # Install minimum required versions
         run: |
-          pip install requests>=2.31.0 PyGithub>=2.0.0 google-generativeai>=0.3.0
+          pip install requests>=2.32.5 PyGithub>=2.0.0 google-genai>=0.2.0
 
       - name: Run CI Failure Bot
         env:
@@ -38,4 +38,14 @@ jobs:
           WORKFLOW_RUN_ID: ${{ github.event.workflow_run.id }}
           REPOSITORY: ${{ github.repository }}
           PR_NUMBER: ${{ github.event.workflow_run.pull_requests[0].number || '' }}
-        run: python .github/scripts/ci_failure_bot.py
+        run: |
+          set -e  # Exit immediately if any command fails
+          echo "Starting CI Failure Bot..."
+          python .github/scripts/ci_failure_bot.py
+          exit_code=$?
+          if [ $exit_code -ne 0 ]; then
+            echo "ERROR: CI Failure Bot failed with exit code $exit_code"
+            echo "This indicates a critical issue with the bot logic that needs immediate attention"
+            exit $exit_code
+          fi
+          echo "CI Failure Bot completed successfully"