Skip to content

Commit 4eb447e

Browse files
realmarcinrealmarcin
committed
Test rubric10 fix script - document structural issues
- Updated fix_evaluation_scores.py to support both rubric10 and rubric20 - Tested with 16 rubric10 evaluation files - Found only 2/16 files have correct structure (12.5% success rate) - 14/16 files have incompatible structures (wrong field names, data types) - Created detailed test results report in RUBRIC10_FIX_SCRIPT_TEST_RESULTS.md Key findings: - AI_READI_claudecode_agent: 32/50 (64%) - Successfully calculated - VOICE_claudecode_agent: 47/50 (94%) - Successfully calculated - CHORUS_claudecode_agent: Script crashed (element_scores is dict not list) - 13 other files: Returned 0/0 due to missing/wrong structure Recommendation: Complete regeneration using schema-compliant prompt required Fix script cannot salvage files with fundamental structural inconsistencies
1 parent f5127ea commit 4eb447e

File tree

2 files changed

+424
-118
lines changed

2 files changed

+424
-118
lines changed

‎RUBRIC10_FIX_SCRIPT_TEST_RESULTS.md‎

Lines changed: 181 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,181 @@
1+
# Rubric10 Fix Script Test Results
2+
3+
## Test Date: 2025-12-25
4+
5+
## Summary
6+
7+
Tested `scripts/fix_evaluation_scores.py` (updated to support rubric10 format) against 16 rubric10 evaluation files in `data/evaluation_llm/rubric10_semantic/concatenated/`.
8+
9+
**Results:**
10+
- ✅ Successfully processed: 2/16 files (12.5%)
11+
- ❌ Structural issues preventing fix: 14/16 files (87.5%)
12+
- 🔥 Script crashed on: 1 file (CHORUS_claudecode_agent)
13+
14+
## Detailed Findings
15+
16+
### Files Successfully Processed (2/16)
17+
18+
Only files with correct `element_scores` list structure could be processed:
19+
20+
| File | Calculated Score | Status |
21+
|------|------------------|--------|
22+
| AI_READI_claudecode_agent | 32.0/50.0 (64%) | ✅ FIXED |
23+
| VOICE_claudecode_agent | 47.0/50.0 (94%) | ✅ FIXED |
24+
25+
### Files with Structural Issues (14/16)
26+
27+
These files have incompatible structures and returned 0/0 or crashed:
28+
29+
#### Wrong Field Names (6 files)
30+
31+
**CM4AI_claudecode_agent** - Uses `element_evaluations` instead of `element_scores`
32+
```json
33+
{
34+
"overall_scores": {...},
35+
"element_evaluations": {...} // Should be "element_scores": [...]
36+
}
37+
```
38+
39+
**AI_READI_gpt5** - No `element_scores` field at all
40+
```json
41+
{
42+
"scoring_summary": {...},
43+
"detailed_scores": {...} // Completely different structure
44+
}
45+
```
46+
47+
Similar issues in:
48+
- AI_READI_claudecode_assistant
49+
- AI_READI_claudecode
50+
- CM4AI_gpt5
51+
- CM4AI_claudecode_assistant
52+
53+
#### Wrong Data Types (1 file)
54+
55+
**CHORUS_claudecode_agent** - `element_scores` is dict instead of list
56+
```json
57+
{
58+
"element_scores": { // Should be array: [...]
59+
// ...
60+
}
61+
}
62+
```
63+
**Error**: Script crashed with `AttributeError: 'str' object has no attribute 'get'`
64+
65+
#### Other Structural Variations (7 files)
66+
67+
Files that returned 0/0 due to missing or incompatible structures:
68+
- CHORUS_claudecode_assistant
69+
- CHORUS_claudecode
70+
- CHORUS_gpt5
71+
- CM4AI_claudecode
72+
- VOICE_claudecode_assistant
73+
- VOICE_claudecode
74+
- VOICE_gpt5
75+
76+
## Field Name Variations Observed
77+
78+
Across 16 files, we found these different field names for the same concept:
79+
80+
### Overall Score Field
81+
- `overall_score` (dict) - AI_READI_claudecode_agent
82+
- `overall_scores` (dict) - CM4AI_claudecode_agent
83+
- `overall_summary` (?) - CHORUS files
84+
- `overall_assessment` (string) - VOICE_claudecode_agent
85+
- `scoring_summary` (?) - AI_READI_gpt5
86+
- **EXPECTED**: `summary_scores` (dict with total_score, total_max_score, overall_percentage)
87+
88+
### Element Scores Field
89+
- `element_scores` (list) ✅ - 2 files (correct)
90+
- `element_scores` (dict) ❌ - 1 file (CHORUS)
91+
- `element_evaluations` ❌ - 1 file (CM4AI)
92+
- `detailed_scores` ❌ - 1 file (AI_READI_gpt5)
93+
- Missing entirely ❌ - 11 files
94+
95+
## Successful Structure Example
96+
97+
Only these 2 files had the correct structure:
98+
99+
```json
100+
{
101+
"element_scores": [ // Must be array
102+
{
103+
"id": 1,
104+
"name": "Element Name",
105+
"sub_elements": [ // Must be array
106+
{
107+
"name": "Sub-element Name",
108+
"score": 1, // Binary: 0 or 1
109+
"evidence": "...",
110+
"quality_note": "..."
111+
}
112+
// ... 4 more sub-elements
113+
],
114+
"element_score": 5,
115+
"element_max": 5
116+
}
117+
// ... 9 more elements
118+
]
119+
}
120+
```
121+
122+
## Conclusion
123+
124+
**The fix script CANNOT salvage these files.**
125+
126+
### Why?
127+
128+
1. **87.5% have incompatible structures** - Different field names, wrong data types
129+
2. **Only 2 files can be processed** - AI_READI and VOICE claudecode_agent
130+
3. **No consistent pattern** - Each generation method produces different structure
131+
4. **Script would need 14 different structure handlers** - Not maintainable
132+
133+
### Recommendation
134+
135+
**Complete regeneration required** using the new schema-compliant prompt:
136+
- Prompt: `RUBRIC10_EVALUATION_PROMPT_FINAL.md`
137+
- Schema: `src/download/prompts/rubric10_semantic_schema.json`
138+
- Post-processing: `scripts/fix_evaluation_scores.py` (for math errors only)
139+
- Validation: `scripts/validate_evaluation_schema.py`
140+
141+
### What About the 2 Working Files?
142+
143+
Even the "working" files (AI_READI and VOICE claudecode_agent) have issues:
144+
- Missing required schema fields (rubric, version, project, method, timestamp, model)
145+
- Use `overall_score` or `overall_assessment` instead of `summary_scores`
146+
- Cannot be rendered by HTML generator without fixing field names
147+
148+
**They should also be regenerated for consistency.**
149+
150+
## Comparison to Rubric20
151+
152+
Rubric20 situation was **much better**:
153+
- All 4 files had identical structure
154+
- Only issue: LLM math errors (summing scores incorrectly)
155+
- Fix script corrected all 4 files successfully
156+
- HTML generation worked perfectly after fix
157+
158+
Rubric10 situation is **much worse**:
159+
- 16 different structures across 16 files
160+
- Fundamental schema inconsistencies
161+
- Fix script can only process 2 files
162+
- HTML generation impossible without schema compliance
163+
164+
## Next Steps
165+
166+
1. ✅ Document findings (this report)
167+
2. ⏭️ Proceed with regeneration using new prompt
168+
3. ⏭️ Start with VOICE project (test case)
169+
4. ⏭️ Validate against schema before HTML generation
170+
5. ⏭️ Generate HTML from schema-compliant evaluations
171+
172+
---
173+
174+
**Testing Command Used:**
175+
```bash
176+
poetry run python scripts/fix_evaluation_scores.py \
177+
--input-dir data/evaluation_llm/rubric10_semantic/concatenated \
178+
--dry-run
179+
```
180+
181+
**Script Version**: Updated 2025-12-25 to support both rubric10 and rubric20 formats

0 commit comments

Comments
 (0)