Skip to content

Commit 27775cf

Browse files
anthonydmaysanthonydmays
authored
feat: adds Google AppsScript utility for automating grading (#7)
* chore: adds useful links to syllabus Signed-off-by: Anthony D. Mays <[email protected]> * feat: adds initial grader apps script project Signed-off-by: Anthony D. Mays <[email protected]> * feat: completes pr hyperlink functionality Signed-off-by: Anthony D. Mays <[email protected]> * feat: adds ChatGPT automated grading Signed-off-by: Anthony D. Mays <[email protected]> * chore: changes pr comment so that it appears correctly Signed-off-by: Anthony D. Mays <[email protected]> * chore: marks submission as 'Y' if already graded Signed-off-by: Anthony D. Mays <[email protected]> --------- Signed-off-by: Anthony D. Mays <[email protected]>
1 parent 0273187 commit 27775cf

File tree

10 files changed

+4951
-0
lines changed

10 files changed

+4951
-0
lines changed

‎lib/javascript/grader/.clasp.json

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
{"scriptId":"17mrtU0kXm3s51qeefzeBH6V12yzNztLayWorp8pTOMW25QHscqIkiU3L","rootDir":"/Users/anthonymays/source/forks/code-society-25-2/lib/javascript/grader"}

‎lib/javascript/grader/.gitignore

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
# Dependencies
2+
node_modules/
3+
4+
# Build output
5+
dist/
6+
7+
# Logs
8+
*.log
9+
npm-debug.log*
10+
11+
# Runtime data
12+
pids
13+
*.pid
14+
*.seed
15+
*.pid.lock
16+
17+
# Coverage directory used by tools like istanbul
18+
coverage/
19+
20+
# nyc test coverage
21+
.nyc_output
22+
23+
# Dependency directories
24+
node_modules/
25+
26+
# Optional npm cache directory
27+
.npm
28+
29+
# Optional REPL history
30+
.node_repl_history
31+
32+
# Output of 'npm pack'
33+
*.tgz
34+
35+
# Yarn Integrity file
36+
.yarn-integrity
37+
38+
# dotenv environment variables file
39+
.env
40+
41+
# macOS
42+
.DS_Store
43+
44+
# Windows
45+
Thumbs.db
46+
ehthumbs.db
47+
48+
# VS Code
49+
.vscode/settings.json
50+
.vscode/launch.json
51+
52+
# Keep clasp.json for project configuration
53+
# .clasp.json

‎lib/javascript/grader/GRADING_ENHANCEMENT.md

Lines changed: 211 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,211 @@
1+
# GitHub Grader - Auto-Grading Enhancement
2+
3+
This enhancement adds automatic grading functionality to the GitHub Grader system. Here's how it works:
4+
5+
## Overview
6+
7+
The enhanced grader now performs these steps:
8+
9+
1. **Early Exit**: If a PR link has already been recorded and grading status is "GRADED", processing stops
10+
2. **Grading Instructions**: Fetches `GRADING.md` from the corresponding lesson folder
11+
3. **Automatic Grading**: If GRADING.md is found, analyzes PR changes against grading criteria
12+
4. **Draft Review**: Creates a draft review comment on the original PR
13+
5. **Score Recording**: Records functional and technical scores in the spreadsheet
14+
15+
## Key Feature: GRADING.md-Based Activation
16+
17+
**Grading only occurs when a `GRADING.md` file exists in the lesson folder.** This means:
18+
- ✅ Lessons with `lesson_XX/GRADING.md` will be automatically graded
19+
- ⏭️ Lessons without `GRADING.md` will only have PR links recorded (no grading)
20+
- 🎯 No configuration needed - simply add GRADING.md files when ready to grade
21+
22+
## New Spreadsheet Columns
23+
24+
The system now uses these additional columns:
25+
- **Column M (13)**: Functional Score (0-10)
26+
- **Column N (14)**: Technical Score (0-10)
27+
- **Column O (15)**: Grading Status ("GRADED", "ERROR", or empty)
28+
29+
## Required Setup
30+
31+
### 1. Create GRADING.md Files
32+
Each lesson folder should contain a `GRADING.md` file with grading criteria:
33+
34+
```
35+
lesson_00/
36+
GRADING.md
37+
lesson_01/
38+
GRADING.md
39+
lesson_02/
40+
GRADING.md
41+
```
42+
43+
### 2. GRADING.md Format
44+
See the example in `lesson_00/GRADING.md`. Include:
45+
- Functional requirements with point values
46+
- Technical requirements with point values
47+
- Common issues to check
48+
- Notes for graders
49+
50+
### 3. GitHub Token Permissions
51+
Your GitHub token needs these permissions:
52+
- `repo` scope (for private repos) or `public_repo` (for public repos)
53+
- `pull_requests:write` (to create review comments)
54+
55+
## Menu Options
56+
57+
The "GitHub Grader" menu includes:
58+
- **🔄 Sync GitHub PRs Now**: Manual sync of all recent PRs
59+
- **⚙️ Setup GitHub Token**: Configure GitHub access
60+
- **🕐 Setup Auto-Sync (Hourly)**: Enable automatic syncing
61+
- **� Disable Auto-Sync**: Disable automatic syncing
62+
- **🧪 Test Student Lookup**: Test student name resolution
63+
- **🧪 Test Grading System**: Test grading on a specific PR
64+
- **📋 View Sync Logs**: Instructions for viewing execution logs
65+
66+
## How Grading Works
67+
68+
### 1. Early Exit Logic
69+
```typescript
70+
if (currentPrUrl === prHyperlink && gradingStatus === "GRADED") {
71+
return {updated: false, message: "Already graded"};
72+
}
73+
```
74+
75+
### 2. Automatic Grading Detection
76+
The system automatically attempts grading for any ungraded PR:
77+
- Tries to fetch `lesson_XX/GRADING.md` from the repository
78+
- If found: proceeds with grading analysis
79+
- If not found: skips grading (no error, just logs the absence)
80+
81+
### 3. Basic Grading Heuristics
82+
Current implementation includes:
83+
- Tests presence check (+2 functional points)
84+
- Documentation check (+1 technical point)
85+
- File count analysis (penalty for too many files)
86+
87+
## Enhancing the Grading Logic
88+
89+
### Option 1: Rule-Based Enhancement
90+
Enhance `analyzeSubmission()` function with more sophisticated rules:
91+
92+
```typescript
93+
function analyzeSubmission(prContent: PRContent, gradingInstructions: string, changedFiles: string[]): GradingAnalysis {
94+
// Add checks for:
95+
// - Code quality metrics
96+
// - Specific file requirements
97+
// - Naming conventions
98+
// - Git commit quality
99+
// - Code complexity
100+
}
101+
```
102+
103+
### Option 2: AI Integration
104+
Integrate with OpenAI or other AI services:
105+
106+
```typescript
107+
function analyzeWithAI(prContent: string, gradingInstructions: string): Promise<GradingAnalysis> {
108+
const apiKey = PropertiesService.getScriptProperties().getProperty("OPENAI_API_KEY");
109+
110+
const response = UrlFetchApp.fetch("https://api.openai.com/v1/chat/completions", {
111+
method: "post",
112+
headers: {
113+
"Authorization": "Bearer " + apiKey,
114+
"Content-Type": "application/json"
115+
},
116+
payload: JSON.stringify({
117+
model: "gpt-4",
118+
messages: [
119+
{
120+
role: "system",
121+
content: "You are a code grader. Analyze the PR content against the grading criteria and provide scores."
122+
},
123+
{
124+
role: "user",
125+
content: `Grading Criteria:\n${gradingInstructions}\n\nPR Content:\n${prContent}`
126+
}
127+
]
128+
})
129+
});
130+
131+
// Parse response and extract scores
132+
}
133+
```
134+
135+
## Draft Review Format
136+
137+
The system creates draft reviews with this format:
138+
139+
```markdown
140+
## 🎓 Automated Grading Report
141+
142+
**Student:** John Doe
143+
**Date:** 7/31/2025
144+
145+
### Scores
146+
- **Functional Score:** 7/10
147+
- **Technical Score:** 8/10
148+
149+
### Feedback
150+
✅ Includes test files
151+
✅ Includes documentation
152+
⚠️ Large number of changed files - consider smaller commits
153+
154+
### Grading Criteria Used
155+
[First 500 characters of GRADING.md content]...
156+
157+
---
158+
*This is an automated preliminary review. Please review and adjust before finalizing.*
159+
```
160+
161+
## Best Practices
162+
163+
### For Instructors
164+
1. **Review Draft Comments**: Always review and edit before publishing
165+
2. **Consistent GRADING.md**: Keep grading criteria consistent across lessons
166+
3. **Regular Testing**: Use the test function to verify grading logic
167+
4. **Gradual Rollout**: Add GRADING.md files to lessons when ready to grade them
168+
5. **Selective Grading**: Only lessons with GRADING.md files will be graded
169+
170+
### For GRADING.md Files
171+
1. **Clear Criteria**: Specify exactly what earns each point
172+
2. **Point Distribution**: Aim for 70% functional, 30% technical
173+
3. **Common Issues**: List frequently seen problems
174+
4. **Examples**: Include examples of good/bad implementations
175+
176+
### For Maintenance
177+
1. **Monitor Logs**: Check execution logs regularly
178+
2. **Error Handling**: The system gracefully handles missing files
179+
3. **Backup Scores**: Spreadsheet serves as permanent record
180+
4. **Rate Limits**: GitHub API has rate limits (5000 requests/hour)
181+
182+
## Troubleshooting
183+
184+
### Common Issues
185+
1. **Missing GRADING.md**: System logs info message, skips grading (no error)
186+
2. **GitHub API Errors**: Check token permissions and rate limits
187+
3. **Draft Review Fails**: Verify token has pull request write access
188+
4. **No Lesson Detected**: Ensure PR changes files in `lesson_XX/` folders
189+
190+
### Error Messages
191+
- `"No grading instructions found for L00"`: No `lesson_00/GRADING.md` file (this is normal if grading not needed)
192+
- `"GitHub API error: 403"`: Check token permissions
193+
- `"Failed to create review"`: Verify pull request write access
194+
195+
## Future Enhancements
196+
197+
1. **AI Integration**: Use GPT-4 or Claude for sophisticated analysis
198+
2. **Custom Rubrics**: Allow per-lesson custom grading logic
199+
3. **Plagiarism Detection**: Compare submissions across students
200+
4. **Code Quality Metrics**: Integrate with ESLint, SonarQube, etc.
201+
5. **Automated Testing**: Run unit tests and grade based on results
202+
6. **Peer Review**: Allow students to review each other's work
203+
204+
## Security Considerations
205+
206+
1. **Token Storage**: GitHub tokens stored securely in Script Properties
207+
2. **Draft Reviews**: Reviews are drafts by default, require manual publishing
208+
3. **Data Privacy**: No student data leaves Google Sheets environment
209+
4. **Access Control**: Only spreadsheet collaborators can access grading functions
210+
211+
This enhancement provides a solid foundation for automated grading while maintaining instructor control over the final grades and feedback.

0 commit comments

Comments
 (0)