Skip to content
This repository was archived by the owner on Sep 9, 2025. It is now read-only.

Commit 9157839

Browse files
russellbbjhargrave
authored andcommitted
skill: Add reasoning skill about siblings
This skill was inspired by one of the tests included in mt-bench. It included some questions that the model answered incorrectly. When given the same question manually, I got the following incorrect answer: > Q: David has three sisters. Each of them has one brother. How many > brothers does David have? > A: David has three sisters, and each of them has one brother, so it > must be the same brother in all cases. Therefore, David has 1 brother. Signed-off-by: Russell Bryant <rbryant@redhat.com> Signed-off-by: BJ Hargrave <hargrave@us.ibm.com>
1 parent 9ae7bfa commit 9157839

File tree

1 file changed

+28
-0
lines changed
  • compositional_skills/extraction/inference/qualitative/siblings

1 file changed

+28
-0
lines changed
Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
created_by: russellb
2+
task_description: Reasoning about sibling relationships
3+
seed_examples:
4+
- answer: |
5+
David has three sisters, and each of them has one brother.
6+
David is their one brother. So, David has no brothers.
7+
question: |
8+
David has three sisters. Each of them has one brother. How many brothers does David have?
9+
- answer: |
10+
David has three sisters, and each of them has two brothers.
11+
David is one of their brothers. So, David has one brother.
12+
question: |
13+
David has three sisters. Each of them has two brothers. How many brothers does David have?
14+
- answer: |
15+
Sally has three brothers, and each of them has one sister.
16+
Sally is their one sister. So, Sally has no sisters.
17+
question: |
18+
Sally has three brothers. Each of them has one sister. How many sisters does Sally have?
19+
- answer: |
20+
Sally has three brothers, and each of them has two sisters.
21+
Sally is their one sister. So, Sally has one sister.
22+
question: |
23+
Sally has three brothers. Each of them has two sisters. How many sisters does Sally have?
24+
- answer: |
25+
Sally has one brother and one sister. There are two sisters and one brother.
26+
Sally's brother has two sisters.
27+
question: |
28+
Sally has one brother and one sister. How many sisters does Sally's brother have?

0 commit comments

Comments
 (0)