Skip to content

Commit fb1dec1

Browse files
Update scoring_workflow.md
1 parent b5902c2 commit fb1dec1

File tree

1 file changed

+124
-1
lines changed

1 file changed

+124
-1
lines changed
Lines changed: 124 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,124 @@
1-
#scoring workflow
1+
# 🧮 Scoring Workflow: PyTorch Ambassador Program
2+
3+
This document outlines the **scoring system** used to evaluate nominations for the PyTorch Ambassador Program. It is intended for **reviewers, maintainers, and workflow administrators** who participate in or manage the review process.
4+
5+
---
6+
7+
## 📌 Purpose
8+
9+
The scoring workflow enables reviewers to evaluate nominee submissions using a **1–5 scale** via GitHub issue comments. It supports a structured review process by:
10+
11+
- Acknowledging valid scores immediately
12+
- Accepting only one score per reviewer
13+
- Allowing a review coordination phase
14+
- Calculating average scores
15+
- Applying a final decision (`approved` or `rejected`)
16+
- Closing rejected issues
17+
- Adding a `scoring-complete` label for tracking
18+
19+
---
20+
21+
## 🔁 Overview of the Scoring Flow
22+
23+
| Phase | Labels Used |
24+
|-------|-------------|
25+
| Nomination submitted | `pending-review` |
26+
| Review begins (manual trigger) | `under-review`, `scoring-in-progress` |
27+
| Review finalized | `approved` / `rejected`, `scoring-complete` |
28+
29+
---
30+
31+
## 🧾 Step 1: Submitting a Score
32+
33+
Reviewers comment directly on a nomination issue using this format:
34+
35+
Score: X
36+
37+
Where `X` is an integer from **1 to 5**. The comment is **not case-sensitive**.
38+
39+
✅ Only the **first valid score per reviewer** is counted.
40+
⚠️ A warning will appear if additional scores are submitted by the same user.
41+
42+
---
43+
44+
## ✅ Step 2: Acknowledgement Comment
45+
46+
When a valid score is submitted, the bot responds with:
47+
48+
📝 Score received from @reviewer: X
49+
⏳ Final decision will be calculated and posted after all reviewers have submitted or in approximately 2 hours.
50+
🚨 Only your first score is counted. Please discuss with other reviewers before submitting.
51+
52+
53+
This encourages team coordination **before** scoring and reinforces the single-score policy.
54+
55+
---
56+
57+
## 🧮 Step 3: Finalizing the Score
58+
59+
The finalization step is **manually triggered** by a reviewer or maintainer. It:
60+
61+
1. Gathers all valid `Score: X` comments
62+
2. Filters out duplicate scores (only first per user)
63+
3. Calculates the **average score**
64+
4. Applies the final label:
65+
66+
| Average Score | Outcome |
67+
|---------------|---------|
68+
| ≥ 3.0 |`approved` |
69+
| < 3.0 |`rejected` |
70+
71+
5. Adds the `scoring-complete` label
72+
6. Removes temporary labels: `pending-review`, `under-review`, `scoring-in-progress`
73+
7. Closes the issue if it is rejected
74+
75+
---
76+
77+
### 💬 Example Summary Comment
78+
79+
🧮 Final average score from 3 reviewer(s): 4.33
80+
👥 Reviewed by: @alice, @bob, @carol
81+
📌 Final decision: APPROVED
82+
83+
---
84+
85+
## 🛠 Workflow Files Involved
86+
87+
- `.github/workflows/acknowledge-score.yml`
88+
Acknowledges and limits reviewer scores
89+
90+
- `.github/workflows/finalize-scoring.yml`
91+
Calculates score average and applies final decision
92+
93+
- `.github/workflows/mark-under-review.yml`
94+
Manually removes `pending-review` and adds `under-review`, `scoring-in-progress`
95+
96+
---
97+
98+
## 🧪 How to Test the Workflow
99+
100+
1. Create a test nomination issue with the `nomination` label
101+
2. Trigger the manual **under-review** workflow
102+
3. Submit `Score: X` comments from different users
103+
4. Trigger **Finalize Scoring** manually
104+
5. Confirm:
105+
- Decision label (`approved`/`rejected`) is added
106+
- `scoring-complete` label is added
107+
- Summary comment is posted
108+
- Temporary review labels are removed
109+
- Rejected issues are closed
110+
111+
---
112+
113+
## 🚧 Future Enhancements
114+
115+
- Per-issue review deadlines
116+
- Reviewer coordination UI
117+
- Score override logic with audit trail
118+
- Discord or email alerts when scoring is finalized
119+
120+
---
121+
122+
## 🙋 Questions?
123+
124+
If you need help using or maintaining these workflows, contact a maintainer or open an issue in this repository.

0 commit comments

Comments
 (0)