|
37 | 37 | NLI_STATEMENTS_MESSAGE = HumanMessagePromptTemplate.from_template( |
38 | 38 | """ |
39 | 39 | Prompt: Natural language inference |
40 | | -Consider the following context: |
41 | | -Context: |
42 | | -John is a student at XYZ University. He is pursuing a degree in Computer Science. He is enrolled in several courses this semester, including Data Structures, Algorithms, and Database Management. John is a diligent student and spends a significant amount of time studying and completing assignments. He often stays late in the library to work on his projects. |
43 | | -Now, read the following statements and determine whether they are supported by the information present in the context. Provide a brief explanation for each statement. Also provide a Final Answer (Yes/No) at the end. |
| 40 | +Consider the given context and following statements, then determine whether they are supported by the information present in the context.Provide a brief explanation for each statement before arriving at the verdict (Yes/No). Provide a final verdict for each statement in order at the end in the given format. Do not deviate from the specified format. |
| 41 | +
|
| 42 | +Context:\nJohn is a student at XYZ University. He is pursuing a degree in Computer Science. He is enrolled in several courses this semester, including Data Structures, Algorithms, and Database Management. John is a diligent student and spends a significant amount of time studying and completing assignments. He often stays late in the library to work on his projects. |
44 | 43 | statements:\n1. John is majoring in Biology.\n2. John is taking a course on Artificial Intelligence.\n3. John is a dedicated student.\n4. John has a part-time job.\n5. John is interested in computer programming.\n |
45 | 44 | Answer: |
46 | 45 | 1. John is majoring in Biology. |
47 | | -Explanation: John's major is explicitly mentioned as Computer Science. There is no information suggesting he is majoring in Biology. So answer is No. |
| 46 | +Explanation: John's major is explicitly mentioned as Computer Science. There is no information suggesting he is majoring in Biology. Verdict: No. |
48 | 47 | 2. John is taking a course on Artificial Intelligence. |
49 | | -Explanation: The context mentions the courses John is currently enrolled in, and Artificial Intelligence is not mentioned. Therefore, it cannot be deduced that John is taking a course on AI.So answer is No. |
| 48 | +Explanation: The context mentions the courses John is currently enrolled in, and Artificial Intelligence is not mentioned. Therefore, it cannot be deduced that John is taking a course on AI. Verdict: No. |
50 | 49 | 3. John is a dedicated student. |
51 | | -Explanation: The prompt states that he spends a significant amount of time studying and completing assignments. Additionally, it mentions that he often stays late in the library to work on his projects, which implies dedication.So answer is Yes. |
| 50 | +Explanation: The prompt states that he spends a significant amount of time studying and completing assignments. Additionally, it mentions that he often stays late in the library to work on his projects, which implies dedication. Verdict: Yes. |
52 | 51 | 4. John has a part-time job. |
53 | | -Explanation: There is no information given in the context about John having a part-time job. Therefore, it cannot be deduced that John has a part-time job. So answer is No. |
| 52 | +Explanation: There is no information given in the context about John having a part-time job. Therefore, it cannot be deduced that John has a part-time job. Verdict: No. |
54 | 53 | 5. John is interested in computer programming. |
55 | | -Explanation: The context states that John is pursuing a degree in Computer Science, which implies an interest in computer programming.So answer is Yes. |
56 | | -Final answer: No. No. Yes. No. Yes. |
| 54 | +Explanation: The context states that John is pursuing a degree in Computer Science, which implies an interest in computer programming. Verdict: Yes. |
| 55 | +Final verdict for each statement in order: No. No. Yes. No. Yes. |
57 | 56 | context:\n{context} |
58 | 57 | statements:\n{statements} |
59 | | -Now, read the following statements and determine whether they are supported by the information present in the context. Provide a brief explanation for each statement. Also provide a Final Answer (Yes/No) at the end. |
60 | 58 | Answer: |
61 | 59 | """ # noqa: E501 |
62 | 60 | ) |
@@ -113,20 +111,20 @@ def _score_batch(self: t.Self, ds: Dataset) -> Dataset: |
113 | 111 | outputs = result.generations |
114 | 112 |
|
115 | 113 | scores = [] |
| 114 | + final_answer = "Final verdict for each statement in order:" |
| 115 | + final_answer = final_answer.lower() |
116 | 116 | for i, output in enumerate(outputs): |
117 | 117 | output = output[0].text.lower().strip() |
118 | | - if output.find("final answer:") != -1: |
119 | | - output = output[output.find("final answer:") + len("final answer:") :] |
| 118 | + if output.find(final_answer) != -1: |
| 119 | + output = output[output.find(final_answer) + len(final_answer) :] |
120 | 120 | score = sum( |
121 | 121 | 0 if "yes" in answer else 1 |
122 | 122 | for answer in output.strip().split(".") |
123 | 123 | if answer != "" |
124 | 124 | ) |
125 | 125 | score = score / len(list_statements[i]) |
126 | 126 | else: |
127 | | - score = max(0, output.count("so answer is no")) / len( |
128 | | - list_statements[i] |
129 | | - ) |
| 127 | + score = max(0, output.count("verdict: no")) / len(list_statements[i]) |
130 | 128 |
|
131 | 129 | scores.append(1 - score) |
132 | 130 |
|
|
0 commit comments