Skip to content
This repository was archived by the owner on Sep 9, 2025. It is now read-only.

Commit 3bfb1be

Browse files
russellbbjhargrave
authored andcommitted
skill: Add reasoning skill about siblings
This skill was inspired by one of the tests included in mt-bench. It included some questions that the model answered incorrectly. When given the same question manually, I got the following incorrect answer: > Q: David has three sisters. Each of them has one brother. How many > brothers does David have? > A: David has three sisters, and each of them has one brother, so it > must be the same brother in all cases. Therefore, David has 1 brother. Signed-off-by: Russell Bryant <rbryant@redhat.com> Signed-off-by: BJ Hargrave <hargrave@us.ibm.com>
1 parent 2631714 commit 3bfb1be

File tree

2 files changed

+30
-0
lines changed

2 files changed

+30
-0
lines changed
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Self-authored
Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
created_by: russellb
2+
version: 3
3+
task_description: Inductive reasoning about sibling relationships
4+
seed_examples:
5+
- question: |
6+
David has three sisters. Each of them has one brother. How many brothers does David have?
7+
answer: |
8+
David has three sisters, and each of them has one brother.
9+
David is their one brother. So, David has no brothers.
10+
- question: |
11+
David has three sisters. Each of them has two brothers. How many brothers does David have?
12+
answer: |
13+
David has three sisters, and each of them has two brothers.
14+
David is one of their brothers. So, David has one brother.
15+
- question: |
16+
Sally has three brothers. Each of them has one sister. How many sisters does Sally have?
17+
answer: |
18+
Sally has three brothers, and each of them has one sister.
19+
Sally is their one sister. So, Sally has no sisters.
20+
- question: |
21+
Sally has three brothers. Each of them has two sisters. How many sisters does Sally have?
22+
answer: |
23+
Sally has three brothers, and each of them has two sisters.
24+
Sally is their one sister. So, Sally has one sister.
25+
- question: |
26+
Sally has one brother and one sister. How many sisters does Sally's brother have?
27+
answer: |
28+
Sally has one brother and one sister. There are two sisters and one brother.
29+
Sally's brother has two sisters.

0 commit comments

Comments
 (0)