File tree Expand file tree Collapse file tree 3 files changed +17
-2
lines changed Expand file tree Collapse file tree 3 files changed +17
-2
lines changed Original file line number Diff line number Diff line change @@ -35,8 +35,11 @@ result = await judge.evaluate(
35
35
```
36
36
37
37
** What happens behind the scenes:**
38
+
38
39
- Judge creates a prompt asking to evaluate the response based on your criteria
40
+
39
41
- The LLM provides a score (typically 1-10) and reasoning
42
+
40
43
- You get a structured result with ` decision ` , ` reasoning ` , and ` score `
41
44
42
45
## Level 2: Adding Structure with Scales and Rubrics
Original file line number Diff line number Diff line change @@ -5,10 +5,15 @@ vLLM Judge provides 20+ pre-built metrics for common evaluation tasks. This guid
5
5
## What are Metrics?
6
6
7
7
Metrics are pre-configured evaluation templates that encapsulate:
8
+
8
9
- ** Criteria** : What to evaluate
10
+
9
11
- ** Scale** : Numeric range (if applicable)
12
+
10
13
- ** Rubric** : Judgement/Scoring guidelines
14
+
11
15
- ** System Prompt** : Evaluator expertise
16
+
12
17
- ** Examples** : Few-shot guidance
13
18
14
19
## Using Pre-built Metrics
Original file line number Diff line number Diff line change @@ -5,9 +5,13 @@ Template variables make your evaluations dynamic and reusable. This guide shows
5
5
## Why Use Templates?
6
6
7
7
Templates allow you to:
8
+
8
9
- ** Reuse** evaluation logic with different contexts
10
+
9
11
- ** Parameterize** criteria for different scenarios
12
+
10
13
- ** Scale** evaluations across varied use cases
14
+
11
15
- ** Maintain** consistency while allowing flexibility
12
16
13
17
## Basic Template Usage
@@ -640,11 +644,14 @@ Template variables provide powerful flexibility for:
640
644
- ** Complex logic** with Jinja2 templates
641
645
642
646
Key takeaways:
647
+
643
648
1 . Start with simple ` {variable} ` substitution
649
+
644
650
2 . Use template metrics for reusability
651
+
645
652
3 . Leverage Jinja2 for complex logic
646
- 4 . Always validate required variables
647
- 5 . Document your templates clearly
653
+
654
+ 4 . Validate required variables
648
655
649
656
<!-- ## Next Steps
650
657
You can’t perform that action at this time.
0 commit comments