Skip to content

Commit 7245143

Browse files
authored
DEV: Add additional evals (#4)
This update adds a few additional evals: - simple topic summarization eval to check if topics are summarized in a relevant and concise manner - simple smart dates eval to check AI helper smart dates - simple explain eval to check AI helper explain
1 parent e6ce72e commit 7245143

File tree

3 files changed

+71
-0
lines changed

3 files changed

+71
-0
lines changed

ai_helper/simple_explain.yml

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
id: simple_explain
2+
name: Simple Explain
3+
description: A simple explain evaluation
4+
type: helper
5+
args:
6+
name: explain
7+
input: "I need to submit my PR on GitHub."
8+
judge:
9+
llm: gpt-4o
10+
pass_rating: 8
11+
prompt: |
12+
Is the following:
13+
14+
[[[
15+
{{output}}
16+
]]]
17+
18+
a faithful explanation about Pull Requests on GitHub based on the following text:
19+
20+
[[[
21+
{{input}}
22+
]]]

ai_helper/simple_smart_dates.yml

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
id: simple_smart_dates
2+
name: Simple Smart Dates
3+
description: A simple smart dates evaluation
4+
type: helper
5+
args:
6+
name: replace_dates
7+
input: "Discourse was officially released on the 26th of August of 2014."
8+
judge:
9+
llm: gpt-40
10+
pass_rating: 8
11+
prompt: |
12+
Is the following:
13+
14+
[[[
15+
{{output}}
16+
]]]
17+
18+
correctly replacing the date in the following text:
19+
20+
[[[
21+
{{input}}
22+
]]]
23+
24+
to a format like [date=2014-08-26 timezone="UTC"] with the correct date?
Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
id: simple_summarization
2+
name: Simple Summarization
3+
description: A basic evaluation of topic summarization
4+
type: summarization
5+
args:
6+
input: |
7+
Dogs are man's best friend. They are loyal, loving, and protective.
8+
They are also very intelligent and can be trained to do many things.
9+
Dogs are used as service animals, therapy animals, and even as police dogs.
10+
They are also great companions and can help reduce stress and anxiety. Dogs are truly amazing animals.
11+
judge:
12+
llm: gpt-40
13+
pass_rating: 8
14+
prompt: |
15+
Is the following:
16+
17+
[[[
18+
{{output}}
19+
]]]
20+
21+
a faithful summary of the following text:
22+
23+
[[[
24+
{{input}}
25+
]]]

0 commit comments

Comments
 (0)