Skip to content

Commit b22b13f

Browse files
Copilotkdinev
andcommitted
fix: tighten grader checks per review feedback
- Combo import check now specifically requires the entry-point import (igniteui-angular/combo) and rejects root barrel imports - Theming grader now enforces core() must appear before theme() by comparing line numbers - README updated to match actual grader behavior (no build step) Co-authored-by: kdinev <1472513+kdinev@users.noreply.github.com>
1 parent 2df335e commit b22b13f

File tree

3 files changed

+23
-14
lines changed

3 files changed

+23
-14
lines changed

evals/README.md

Lines changed: 7 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -79,9 +79,10 @@ npm run validate:theming # theming-palette-generation only
7979
2. Write a clear, unambiguous `instruction.md` that tells the agent exactly what
8080
to build.
8181

82-
3. Write `tests/test.sh` to check **outcomes** (files exist, project compiles,
83-
correct selectors are present) rather than specific steps. The grader must
84-
write a reward (0.0–1.0) to `logs/verifier/reward.txt`.
82+
3. Write `tests/test.sh` to check **outcomes** (files exist, correct selectors
83+
and entry-point imports are present, correct API call ordering) rather than
84+
specific steps. The grader must write a reward (0.0–1.0) to
85+
`logs/verifier/reward.txt`.
8586

8687
4. Write `prompts/quality.md` with rubric dimensions that sum to 1.0.
8788

@@ -117,10 +118,11 @@ automatically on PRs that modify `skills/**` or `evals/**`. It:
117118
## Grading Strategy
118119

119120
**Deterministic grader (60% weight)** — checks:
120-
- Project builds without errors
121+
- Expected component files exist
121122
- Correct Ignite UI selector is present in the generated template
122-
- Required imports exist
123+
- Required entry-point imports exist (not root barrel)
123124
- No use of forbidden alternatives
125+
- Correct API call ordering (e.g. `core()` before `theme()`)
124126

125127
**LLM rubric grader (40% weight)** — evaluates:
126128
- Correct intent routing

evals/tasks/component-combo-reactive-form/tests/test.sh

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -73,19 +73,21 @@ else
7373
DETAILS="${DETAILS}FAIL: Forbidden alternative (native select, mat-select, igx-select) detected\n"
7474
fi
7575

76-
# --- Check 5: Correct import from igniteui-angular ---
76+
# --- Check 5: Correct entry-point import from igniteui-angular/combo ---
77+
# The skill requires entry-point imports (not the root barrel).
78+
COMBO_IMPORT_PATTERN="from ['\"](@infragistics/)?igniteui-angular/combo['\"]"
7779
IMPORT_FOUND=0
7880
if [ -n "${COMPONENT_FILE:-}" ]; then
79-
if grep -qE "from ['\"]igniteui-angular|from ['\"]@infragistics/igniteui-angular" "$COMPONENT_FILE" 2>/dev/null; then
81+
if grep -qE "$COMBO_IMPORT_PATTERN" "$COMPONENT_FILE" 2>/dev/null; then
8082
IMPORT_FOUND=1
8183
fi
8284
fi
8385

8486
if [ "$IMPORT_FOUND" -eq 1 ]; then
8587
SCORE=$((SCORE + 1))
86-
DETAILS="${DETAILS}PASS: igniteui-angular import found\n"
88+
DETAILS="${DETAILS}PASS: Correct combo entry-point import found\n"
8789
else
88-
DETAILS="${DETAILS}FAIL: No igniteui-angular import found\n"
90+
DETAILS="${DETAILS}FAIL: Missing import from igniteui-angular/combo entry point\n"
8991
fi
9092

9193
# --- Calculate reward ---

evals/tasks/theming-palette-generation/tests/test.sh

Lines changed: 10 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -50,12 +50,17 @@ else
5050
DETAILS="${DETAILS}FAIL: No theme() mixin call found\n"
5151
fi
5252

53-
# --- Check 4: core() mixin call (must be before theme) ---
54-
if grep -qE '@include.*core\(' "$STYLES_FILE" 2>/dev/null; then
55-
SCORE=$((SCORE + 1))
56-
DETAILS="${DETAILS}PASS: core() mixin call found\n"
57-
else
53+
# --- Check 4: core() mixin call must appear before theme() ---
54+
CORE_LINE=$(grep -nE '@include.*core\(' "$STYLES_FILE" 2>/dev/null | head -1 | cut -d: -f1)
55+
THEME_LINE=$(grep -nE '@include.*theme\(' "$STYLES_FILE" 2>/dev/null | head -1 | cut -d: -f1)
56+
57+
if [ -z "${CORE_LINE:-}" ]; then
5858
DETAILS="${DETAILS}FAIL: No core() mixin call found\n"
59+
elif [ -n "${THEME_LINE:-}" ] && [ "$CORE_LINE" -gt "$THEME_LINE" ]; then
60+
DETAILS="${DETAILS}FAIL: core() must be called before theme() (core on line $CORE_LINE, theme on line $THEME_LINE)\n"
61+
else
62+
SCORE=$((SCORE + 1))
63+
DETAILS="${DETAILS}PASS: core() mixin call found before theme()\n"
5964
fi
6065

6166
# --- Check 5: No hardcoded CSS custom properties as the sole theming approach ---

0 commit comments

Comments
 (0)