Add revert category extractionand exclude ghfirst reverts from stats (#6882)

Adds revert category extraction from GitHub comments and excludes
`ghfirst` reverts from precision/recall metrics.

## Changes

### 1. Added Revert Category Extraction
- New method `extract_revert_categories_batch()` in
`autorevert_checker.py`
- Extracts categories (`nosignal`, `ignoredsignal`, `landrace`, `weird`,
`ghfirst`) from GitHub issue comments
- Single batch query for performance


### 2. Enhanced `get_commits_reverted_with_info()`
- Now includes category information for each revert
- Uses batch extraction for all reverts at once


### 3. Updated Metrics Calculation
- Excludes `ghfirst` reverts from recall calculation
- Shows category breakdown in summary statistics
- Per-workflow precision now shows both total and non-ghfirst metrics


### 4. Fixed Pattern Detection Bug
- Fixed `AttributeError: 'NoneType' object has no attribute 'head_sha'`
- Created proper mapping between failures and their newer commits

<details>
<summary>Bug Fix Details</summary>

**Problem**: `newer_commit_same_job` was used outside its loop scope
**Solution**: Created `failure_to_newer_commit` dict to track mappings
```python
# Map each failure to its newer commit
failure_to_newer_commit = {}
for (rule, job) in suspected_failures:
    newer_commit_same_job, newer_same_jobs = self._find_last_commit_with_job(...)
    if newer_commit_same_job and any(...):
        failure_to_newer_commit[(rule, job)] = newer_commit_same_job

# Use mapping in pattern creation
for (failure_rule, job_name), newer_commit in failure_to_newer_commit.items():
    patterns.append({
        "newer_commits": [newer_commit.head_sha, suspected_commit1.head_sha],
        ...
    })
```
</details>

## Example Output

```
 python -m pytorch_auto_revert autorevert-checker Lint trunk pull inductor linux-binary-manywheel  --hours 720 --verbose

==================================================
SUMMARY STATISTICS
==================================================
Workflow(s): Lint, trunk, pull, inductor, linux-binary-manywheel
Timeframe: 720 hours
Commits checked: 6741
Auto revert patterns detected: 419
Actual reverts inside auto revert patterns detected (precision): 50 (11.9%)
Total revert commits in period: 121

Revert categories:
  nosignal: 46 (38.0%)
  ghfirst: 28 (23.1%)
  uncategorized: 21 (17.4%)
  ignoredsignal: 16 (13.2%)
  weird: 9 (7.4%)
  landrace: 1 (0.8%)

Total reverts excluding ghfirst: 93
Reverts (excluding ghfirst) that dont match any auto revert pattern detected (recall): 50 (53.8%)
Per workflow precision:
  Lint: 6 reverts out of 17 patterns (35.3%) [excluding ghfirst: 6 (35.3%)]
  trunk: 2 reverts out of 14 patterns (14.3%) [excluding ghfirst: 2 (14.3%)]
  pull: 40 reverts out of 354 patterns (11.3%) [excluding ghfirst: 33 (9.3%)]
  inductor: 2 reverts out of 31 patterns (6.5%) [excluding ghfirst: 2 (6.5%)]
  linux-binary-manywheel: 0 reverts out of 3 patterns (0.0%) [excluding ghfirst: 0 (0.0%)]

Reverted patterns:
  - Python RuntimeError: e1aee866 (ignoredsignal)
  - GitHub workflows weren't regenerated: 3b6569b1 (ignoredsignal)
  - GitHub workflows weren't regenerated: bbbced94 (landrace)
  - Python RuntimeError: 060838c2 (nosignal)
  - Lintrunner failure: 1a55fb0e (weird)
  - Lintrunner failure: 3239da0c (nosignal)
  - MSVC compiler error: eab45643 (ignoredsignal)
  - Bad response status code: ea7b2330 (uncategorized)
  - gtest failure: 347ace4c (nosignal)
  - pytest failure: 216bd609 (nosignal)
  - pytest failure: 84c588e5 (ignoredsignal)
  - GHA error: 863327ae (ignoredsignal)
  - GHA error: eb9efb37 (ghfirst)
  - GHA error: 9c39bc24 (ignoredsignal)
  - Python Test File RuntimeError: 6de41ce0 (uncategorized)
  - Fallback for other test failure rules: 3f920f3d (nosignal)
  - pytest failure: f179b719 (ghfirst)
  - pytest failure: 92409b6c (uncategorized)
  - pytest failure: d1b4e0fa (nosignal)
  - pytest failure: 099d0d61 (uncategorized)
  - pytest failure: c79c7bbe (nosignal)
  - pytest failure: c95f7fa8 (nosignal)
  - pytest failure: 08dae945 (weird)
  - pytest failure: fb75dea2 (uncategorized)
  - pytest failure: 9de23d0c (uncategorized)
  - pytest failure: 830a335a (ghfirst)
  - pytest failure: 6d3a4356 (ignoredsignal)
  - pytest failure: a6a3a441 (nosignal)
  - Python Test timeout (KeyboardInterrupt): 8142a028 (nosignal)
  - pr_time_benchmarks regression: 2b9d638e (weird)
  - pytest failure: dc5e8f79 (nosignal)
  - pytest failure: 5264f8cd (weird)
  - pytest failure: 8823138e (nosignal)
  - pytest failure: f154f9b3 (ignoredsignal)
  - pr_time_benchmarks regression: b07725a9 (ghfirst)
  - pr_time_benchmarks regression: d4d0ede6 (ignoredsignal)
  - Build error: 2596e3d0 (nosignal)
  - pytest failure: c1f531f0 (ghfirst)
  - GHA error: c6b4f986 (nosignal)
  - GHA error: 529e0357 (nosignal)
  - GHA error: e694280d (ghfirst)
  - GHA error: e1180c72 (weird)
  - GHA error: 7dcc77e4 (nosignal)
  - GHA error: a3098a74 (ignoredsignal)
  - GHA error: a14f427d (nosignal)
  - GHA error: bee9c70c (nosignal)
  - GHA error: 409c396a (ghfirst)
  - GHA error: 67fb9b7c (nosignal)
  - GHA error: 1b50c125 (nosignal)
  - pytest failure: 196c95d4 (nosignal)
```

Author: izaitsevfb

aws/lambda/pytorch-auto-revert/.env.example

@@ -1,7 +1,7 @@
 # ClickHouse Configuration
 CLICKHOUSE_HOST=your_clickhouse_host
 CLICKHOUSE_PORT=9000
-CLICKHOUSE_USER=default
+CLICKHOUSE_USERNAME=default
 CLICKHOUSE_PASSWORD=your_password
 CLICKHOUSE_DATABASE=pytorch
 

aws/lambda/pytorch-auto-revert/README.md

@@ -0,0 +1,92 @@
+# PyTorch Auto Revert
+
+A tool for detecting autorevert patterns in PyTorch CI workflows.
+
+## Installation
+
+1. Navigate to the project directory:
+```bash
+cd test-infra/aws/lambda/pytorch-auto-revert
+```
+
+2. Install dependencies:
+```bash
+pip install -r requirements.txt
+```
+
+## Configuration
+
+Set the following environment variables or pass them as command-line arguments:
+
+### Required:
+- **ClickHouse connection:**
+  - `CLICKHOUSE_HOST` (or `--clickhouse-host`)
+  - `CLICKHOUSE_USERNAME` (or `--clickhouse-username`)
+  - `CLICKHOUSE_PASSWORD` (or `--clickhouse-password`)
+
+- **GitHub credentials (one of):**
+  - `GITHUB_TOKEN` (or `--github-access-token`)
+  - GitHub App credentials:
+    - `GITHUB_APP_ID` (or `--github-app-id`)
+    - `GITHUB_APP_SECRET` (or `--github-app-secret`)
+    - `GITHUB_INSTALLATION_ID` (or `--github-installation-id`)
+
+### Optional:
+- `CLICKHOUSE_PORT` (default: 8443)
+- `CLICKHOUSE_DATABASE` (default: default)
+- `LOG_LEVEL` (default: INFO)
+
+## Usage
+
+Run the autorevert checker from the project directory:
+
+```bash
+python -m pytorch_auto_revert autorevert-checker <workflows> [options]
+```
+
+### Parameters:
+- `workflows`: One or more workflow names (space separated)
+- `--hours`: Lookback window in hours (default: 48)
+- `--verbose` or `-v`: Show detailed output including commit summaries
+
+### Examples:
+
+1. **Single workflow with default 48-hour lookback:**
+```bash
+python -m pytorch_auto_revert autorevert-checker pull
+```
+
+2. **Single workflow with custom lookback:**
+```bash
+python -m pytorch_auto_revert autorevert-checker trunk --hours 72
+```
+
+3. **Multiple workflows (space separated):**
+```bash
+python -m pytorch_auto_revert autorevert-checker pull trunk inductor --hours 24
+```
+
+4. **With verbose output:**
+```bash
+python -m pytorch_auto_revert autorevert-checker pull --hours 48 --verbose
+```
+
+5. **With explicit credentials:**
+```bash
+python -m pytorch_auto_revert autorevert-checker pull \
+  --clickhouse-host your-host \
+  --clickhouse-username your-user \
+  --clickhouse-password your-pass \
+  --github-access-token your-token
+```
+
+## Other Commands
+
+The tool also supports:
+- `workflow-restart-checker`: Check for restarted workflows
+- `do-restart`: Restart a workflow for a specific commit
+
+Run with `--help` for more information:
+```bash
+python -m pytorch_auto_revert --help
+```

aws/lambda/pytorch-auto-revert/pytorch_auto_revert/autorevert_checker.py

@@ -258,7 +258,9 @@ def detect_autorevert_pattern_workflow(self, workflow_name: str) -> List[Dict]:
                 for j in suspected_commit1.failed_jobs
             }
 
-            common_failures = set()
+            # Map to track newer commits for each failure
+            failure_to_newer_commit = {}
+
             for (
                 suspected_failure_class_rule,
                 suspected_failure_job_name,
@@ -270,25 +272,27 @@ def detect_autorevert_pattern_workflow(self, workflow_name: str) -> List[Dict]:
                     )
                 )
                 if not newer_commit_same_job or not newer_same_jobs:
-                    # No older commit with the same job found
+                    # No newer commit with the same job found
                     continue
 
                 if any(
                     j.classification_rule == suspected_failure_class_rule
                     for j in newer_same_jobs
                 ):
                     # The newer commit has the same job failing
-                    common_failures.add(
-                        (
-                            suspected_failure_class_rule,
-                            suspected_failure_job_name,
-                        )
+                    failure_key = (
+                        suspected_failure_class_rule,
+                        suspected_failure_job_name,
                     )
+                    failure_to_newer_commit[failure_key] = newer_commit_same_job
 
-            if not common_failures:
+            if not failure_to_newer_commit:
                 continue
 
-            for failure_rule, job_name in common_failures:
+            for (
+                failure_rule,
+                job_name,
+            ), newer_commit in failure_to_newer_commit.items():
                 last_commit_with_same_job, last_same_jobs = (
                     self._find_last_commit_with_job(
                         (commits[j] for j in range(i + 1, len(commits))), job_name
@@ -319,11 +323,11 @@ def detect_autorevert_pattern_workflow(self, workflow_name: str) -> List[Dict]:
                         "workflow_name": workflow_name,
                         "failure_rule": failure_rule,
                         "newer_commits": [
-                            "newer_commit_same_job.head_sha",
+                            newer_commit.head_sha,
                             suspected_commit1.head_sha,
                         ],
-                        "older_commit": "last_commit_with_same_job.head_sha",
-                        "failed_job_names": list("last_same_job.name"),
+                        "older_commit": last_commit_with_same_job.head_sha,
+                        "failed_job_names": [j.name for j in last_same_jobs],
                         "older_job_coverage": [],
                     }
                 )
@@ -434,3 +438,102 @@ def is_commit_reverted(self, target_commit_sha: str) -> Optional[Dict]:
                 }
 
         return None  # No revert found
+
+    def extract_revert_categories_batch(self, messages: List[str]) -> Dict[str, str]:
+        """
+        Extract categories from multiple revert commit messages in a single batch query.
+
+        Categories are specified with -c flag like:
+        - nosignal
+        - ignoredsignal
+        - landrace
+        - weird
+        - ghfirst
+
+        Args:
+            messages: List of revert commit messages
+
+        Returns:
+            Dict mapping message to category
+        """
+        # Extract all comment IDs
+        comment_ids = []
+        message_to_comment_id = {}
+
+        for message in messages:
+            comment_match = re.search(r"#issuecomment-(\d+)", message)
+            if comment_match:
+                comment_id = int(comment_match.group(1))
+                comment_ids.append(comment_id)
+                message_to_comment_id[message] = comment_id
+
+        # Batch query for all comment bodies
+        comment_id_to_category = {}
+        if comment_ids:
+            try:
+                query = """
+                SELECT id, body
+                FROM issue_comment
+                WHERE id IN {comment_ids:Array(Int64)}
+                """
+                result = CHCliFactory().client.query(
+                    query, parameters={"comment_ids": comment_ids}
+                )
+
+                for row in result.result_rows:
+                    comment_id, body = row
+                    # Look for -c flag in comment body
+                    match = re.search(r"-c\s+(\w+)", body)
+                    if match:
+                        category = match.group(1).lower()
+                        if category in [
+                            "nosignal",
+                            "ignoredsignal",
+                            "landrace",
+                            "weird",
+                            "ghfirst",
+                        ]:
+                            comment_id_to_category[comment_id] = category
+            except Exception:
+                # If query fails, continue without error
+                pass
+
+        # Map messages to categories
+        result = {}
+        for message in messages:
+            comment_id = message_to_comment_id.get(message)
+            if comment_id and comment_id in comment_id_to_category:
+                result[message] = comment_id_to_category[comment_id]
+            else:
+                result[message] = "uncategorized"
+
+        return result
+
+    def get_commits_reverted_with_info(self) -> Dict[str, Dict]:
+        """
+        Get all commits that were reverted with detailed information including categories.
+
+        Returns:
+            Dict mapping commit SHA to revert information with category
+        """
+        reverted_commits = {}
+        revert_messages = []
+
+        # First pass: collect all reverted commits and their messages
+        for commit in self.commit_history:
+            revert_info = self.is_commit_reverted(commit["sha"])
+            if revert_info:
+                reverted_commits[commit["sha"]] = revert_info
+                revert_messages.append(revert_info["revert_message"])
+
+        # Batch extract categories
+        if revert_messages:
+            message_to_category = self.extract_revert_categories_batch(revert_messages)
+
+            # Update revert info with categories
+            for _, info in reverted_commits.items():
+                info["category"] = message_to_category.get(
+                    info["revert_message"], "uncategorized"
+                )
+
+        return reverted_commits