Skip to content

Commit 7b665f6

Browse files
add 5 prompts to lama dataset (#715)
* add5 prompts to lama dataset * add missing metric + other Co-authored-by: VictorSanh <[email protected]>
1 parent 9ea89d1 commit 7b665f6

File tree

1 file changed

+71
-0
lines changed

1 file changed

+71
-0
lines changed
Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
dataset: lama
2+
subset: trex
3+
templates:
4+
27c2da31-bd1a-48d4-9e34-c530e42c9f00: !Template
5+
answer_choices: null
6+
id: 27c2da31-bd1a-48d4-9e34-c530e42c9f00
7+
jinja: '{{masked_sentence}} Fill the mask with the missing word. ||| {{obj_label}}'
8+
metadata: !TemplateMetadata
9+
choices_in_prompt: false
10+
metrics:
11+
- Accuracy
12+
original_task: false
13+
name: fill_mask
14+
reference: ''
15+
52469d4c-6c46-4686-a36b-7af1801e1eec: !Template
16+
answer_choices: null
17+
id: 52469d4c-6c46-4686-a36b-7af1801e1eec
18+
jinja: 'Given the following paragraph : {{ masked_sentence | replace("[MASK]",obj_label)}}.
19+
what is the relationship between {{obj_label}} and {{sub_label}} ?
20+
21+
||| {{ template | replace("[X]",sub_label) | replace("[Y]", obj_surface)}} '
22+
metadata: !TemplateMetadata
23+
choices_in_prompt: false
24+
metrics:
25+
- Accuracy
26+
- BLEU
27+
- Other
28+
original_task: false
29+
name: find_relation
30+
reference: ''
31+
8cb6ee9d-bcf7-4d82-9acf-b93072c7384b: !Template
32+
answer_choices: null
33+
id: 8cb6ee9d-bcf7-4d82-9acf-b93072c7384b
34+
jinja: 'Replace the mask with the correct word: {{masked_sentence}} ||| {{obj_label}}'
35+
metadata: !TemplateMetadata
36+
choices_in_prompt: false
37+
metrics:
38+
- Accuracy
39+
original_task: false
40+
name: replace_mask
41+
reference: ''
42+
cc07e0dc-b970-4f9d-b76a-05e72a86490e: !Template
43+
answer_choices: null
44+
id: cc07e0dc-b970-4f9d-b76a-05e72a86490e
45+
jinja: "write the negation of the following statements : {{ template | replace(\"\
46+
[X]\",sub_surface) | replace(\"[Y]\", obj_surface)}} \n||| {{ template_negated\
47+
\ | replace(\"[X]\",sub_surface) | replace(\"[Y]\", obj_surface)}} "
48+
metadata: !TemplateMetadata
49+
choices_in_prompt: false
50+
metrics:
51+
- Accuracy
52+
- BLEU
53+
- Other
54+
original_task: false
55+
name: write_negation
56+
reference: ''
57+
e054c5b2-56fd-451a-aba5-fcd105e70bce: !Template
58+
answer_choices: null
59+
id: e054c5b2-56fd-451a-aba5-fcd105e70bce
60+
jinja: "Negate the following statement : {{ template | replace(\"[X]\",sub_surface)\
61+
\ | replace(\"[Y]\", obj_surface)}} \n||| {{ template_negated | replace(\"\
62+
[X]\",sub_surface) | replace(\"[Y]\", obj_surface)}} "
63+
metadata: !TemplateMetadata
64+
choices_in_prompt: false
65+
metrics:
66+
- Accuracy
67+
- BLEU
68+
- Other
69+
original_task: false
70+
name: negate_sentence
71+
reference: ''

0 commit comments

Comments
 (0)