|
1 | | -#scoring workflow |
| 1 | +# 🧮 Scoring Workflow: PyTorch Ambassador Program |
| 2 | + |
| 3 | +This document outlines the **scoring system** used to evaluate nominations for the PyTorch Ambassador Program. It is intended for **reviewers, maintainers, and workflow administrators** who participate in or manage the review process. |
| 4 | + |
| 5 | +--- |
| 6 | + |
| 7 | +## 📌 Purpose |
| 8 | + |
| 9 | +The scoring workflow enables reviewers to evaluate nominee submissions using a **1–5 scale** via GitHub issue comments. It supports a structured review process by: |
| 10 | + |
| 11 | +- Acknowledging valid scores immediately |
| 12 | +- Accepting only one score per reviewer |
| 13 | +- Allowing a review coordination phase |
| 14 | +- Calculating average scores |
| 15 | +- Applying a final decision (`approved` or `rejected`) |
| 16 | +- Closing rejected issues |
| 17 | +- Adding a `scoring-complete` label for tracking |
| 18 | + |
| 19 | +--- |
| 20 | + |
| 21 | +## 🔁 Overview of the Scoring Flow |
| 22 | + |
| 23 | +| Phase | Labels Used | |
| 24 | +|-------|-------------| |
| 25 | +| Nomination submitted | `pending-review` | |
| 26 | +| Review begins (manual trigger) | `under-review`, `scoring-in-progress` | |
| 27 | +| Review finalized | `approved` / `rejected`, `scoring-complete` | |
| 28 | + |
| 29 | +--- |
| 30 | + |
| 31 | +## 🧾 Step 1: Submitting a Score |
| 32 | + |
| 33 | +Reviewers comment directly on a nomination issue using this format: |
| 34 | + |
| 35 | +Score: X |
| 36 | + |
| 37 | +Where `X` is an integer from **1 to 5**. The comment is **not case-sensitive**. |
| 38 | + |
| 39 | +✅ Only the **first valid score per reviewer** is counted. |
| 40 | +⚠️ A warning will appear if additional scores are submitted by the same user. |
| 41 | + |
| 42 | +--- |
| 43 | + |
| 44 | +## ✅ Step 2: Acknowledgement Comment |
| 45 | + |
| 46 | +When a valid score is submitted, the bot responds with: |
| 47 | + |
| 48 | +📝 Score received from @reviewer: X |
| 49 | +⏳ Final decision will be calculated and posted after all reviewers have submitted or in approximately 2 hours. |
| 50 | +🚨 Only your first score is counted. Please discuss with other reviewers before submitting. |
| 51 | + |
| 52 | + |
| 53 | +This encourages team coordination **before** scoring and reinforces the single-score policy. |
| 54 | + |
| 55 | +--- |
| 56 | + |
| 57 | +## 🧮 Step 3: Finalizing the Score |
| 58 | + |
| 59 | +The finalization step is **manually triggered** by a reviewer or maintainer. It: |
| 60 | + |
| 61 | +1. Gathers all valid `Score: X` comments |
| 62 | +2. Filters out duplicate scores (only first per user) |
| 63 | +3. Calculates the **average score** |
| 64 | +4. Applies the final label: |
| 65 | + |
| 66 | +| Average Score | Outcome | |
| 67 | +|---------------|---------| |
| 68 | +| ≥ 3.0 | ✅ `approved` | |
| 69 | +| < 3.0 | ❌ `rejected` | |
| 70 | + |
| 71 | +5. Adds the `scoring-complete` label |
| 72 | +6. Removes temporary labels: `pending-review`, `under-review`, `scoring-in-progress` |
| 73 | +7. Closes the issue if it is rejected |
| 74 | + |
| 75 | +--- |
| 76 | + |
| 77 | +### 💬 Example Summary Comment |
| 78 | + |
| 79 | +🧮 Final average score from 3 reviewer(s): 4.33 |
| 80 | +👥 Reviewed by: @alice, @bob, @carol |
| 81 | +📌 Final decision: APPROVED |
| 82 | + |
| 83 | +--- |
| 84 | + |
| 85 | +## 🛠 Workflow Files Involved |
| 86 | + |
| 87 | +- `.github/workflows/acknowledge-score.yml` |
| 88 | + Acknowledges and limits reviewer scores |
| 89 | + |
| 90 | +- `.github/workflows/finalize-scoring.yml` |
| 91 | + Calculates score average and applies final decision |
| 92 | + |
| 93 | +- `.github/workflows/mark-under-review.yml` |
| 94 | + Manually removes `pending-review` and adds `under-review`, `scoring-in-progress` |
| 95 | + |
| 96 | +--- |
| 97 | + |
| 98 | +## 🧪 How to Test the Workflow |
| 99 | + |
| 100 | +1. Create a test nomination issue with the `nomination` label |
| 101 | +2. Trigger the manual **under-review** workflow |
| 102 | +3. Submit `Score: X` comments from different users |
| 103 | +4. Trigger **Finalize Scoring** manually |
| 104 | +5. Confirm: |
| 105 | + - Decision label (`approved`/`rejected`) is added |
| 106 | + - `scoring-complete` label is added |
| 107 | + - Summary comment is posted |
| 108 | + - Temporary review labels are removed |
| 109 | + - Rejected issues are closed |
| 110 | + |
| 111 | +--- |
| 112 | + |
| 113 | +## 🚧 Future Enhancements |
| 114 | + |
| 115 | +- Per-issue review deadlines |
| 116 | +- Reviewer coordination UI |
| 117 | +- Score override logic with audit trail |
| 118 | +- Discord or email alerts when scoring is finalized |
| 119 | + |
| 120 | +--- |
| 121 | + |
| 122 | +## 🙋 Questions? |
| 123 | + |
| 124 | +If you need help using or maintaining these workflows, contact a maintainer or open an issue in this repository. |
0 commit comments