Skip to content

Commit c1c3768

Browse files
authored
added improved NLI prompt (#81)
Fixes #80 ## What Improved NLI prompt in faithfulness metrics ## Why The earlier NLI prompt was deviating from the specified format. ## How Changed prompt format and made instructions more clear.
1 parent 4a2804e commit c1c3768

File tree

1 file changed

+14
-16
lines changed

1 file changed

+14
-16
lines changed

src/ragas/metrics/faithfulnes.py

Lines changed: 14 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -37,26 +37,24 @@
3737
NLI_STATEMENTS_MESSAGE = HumanMessagePromptTemplate.from_template(
3838
"""
3939
Prompt: Natural language inference
40-
Consider the following context:
41-
Context:
42-
John is a student at XYZ University. He is pursuing a degree in Computer Science. He is enrolled in several courses this semester, including Data Structures, Algorithms, and Database Management. John is a diligent student and spends a significant amount of time studying and completing assignments. He often stays late in the library to work on his projects.
43-
Now, read the following statements and determine whether they are supported by the information present in the context. Provide a brief explanation for each statement. Also provide a Final Answer (Yes/No) at the end.
40+
Consider the given context and following statements, then determine whether they are supported by the information present in the context.Provide a brief explanation for each statement before arriving at the verdict (Yes/No). Provide a final verdict for each statement in order at the end in the given format. Do not deviate from the specified format.
41+
42+
Context:\nJohn is a student at XYZ University. He is pursuing a degree in Computer Science. He is enrolled in several courses this semester, including Data Structures, Algorithms, and Database Management. John is a diligent student and spends a significant amount of time studying and completing assignments. He often stays late in the library to work on his projects.
4443
statements:\n1. John is majoring in Biology.\n2. John is taking a course on Artificial Intelligence.\n3. John is a dedicated student.\n4. John has a part-time job.\n5. John is interested in computer programming.\n
4544
Answer:
4645
1. John is majoring in Biology.
47-
Explanation: John's major is explicitly mentioned as Computer Science. There is no information suggesting he is majoring in Biology. So answer is No.
46+
Explanation: John's major is explicitly mentioned as Computer Science. There is no information suggesting he is majoring in Biology. Verdict: No.
4847
2. John is taking a course on Artificial Intelligence.
49-
Explanation: The context mentions the courses John is currently enrolled in, and Artificial Intelligence is not mentioned. Therefore, it cannot be deduced that John is taking a course on AI.So answer is No.
48+
Explanation: The context mentions the courses John is currently enrolled in, and Artificial Intelligence is not mentioned. Therefore, it cannot be deduced that John is taking a course on AI. Verdict: No.
5049
3. John is a dedicated student.
51-
Explanation: The prompt states that he spends a significant amount of time studying and completing assignments. Additionally, it mentions that he often stays late in the library to work on his projects, which implies dedication.So answer is Yes.
50+
Explanation: The prompt states that he spends a significant amount of time studying and completing assignments. Additionally, it mentions that he often stays late in the library to work on his projects, which implies dedication. Verdict: Yes.
5251
4. John has a part-time job.
53-
Explanation: There is no information given in the context about John having a part-time job. Therefore, it cannot be deduced that John has a part-time job. So answer is No.
52+
Explanation: There is no information given in the context about John having a part-time job. Therefore, it cannot be deduced that John has a part-time job. Verdict: No.
5453
5. John is interested in computer programming.
55-
Explanation: The context states that John is pursuing a degree in Computer Science, which implies an interest in computer programming.So answer is Yes.
56-
Final answer: No. No. Yes. No. Yes.
54+
Explanation: The context states that John is pursuing a degree in Computer Science, which implies an interest in computer programming. Verdict: Yes.
55+
Final verdict for each statement in order: No. No. Yes. No. Yes.
5756
context:\n{context}
5857
statements:\n{statements}
59-
Now, read the following statements and determine whether they are supported by the information present in the context. Provide a brief explanation for each statement. Also provide a Final Answer (Yes/No) at the end.
6058
Answer:
6159
""" # noqa: E501
6260
)
@@ -113,20 +111,20 @@ def _score_batch(self: t.Self, ds: Dataset) -> Dataset:
113111
outputs = result.generations
114112

115113
scores = []
114+
final_answer = "Final verdict for each statement in order:"
115+
final_answer = final_answer.lower()
116116
for i, output in enumerate(outputs):
117117
output = output[0].text.lower().strip()
118-
if output.find("final answer:") != -1:
119-
output = output[output.find("final answer:") + len("final answer:") :]
118+
if output.find(final_answer) != -1:
119+
output = output[output.find(final_answer) + len(final_answer) :]
120120
score = sum(
121121
0 if "yes" in answer else 1
122122
for answer in output.strip().split(".")
123123
if answer != ""
124124
)
125125
score = score / len(list_statements[i])
126126
else:
127-
score = max(0, output.count("so answer is no")) / len(
128-
list_statements[i]
129-
)
127+
score = max(0, output.count("verdict: no")) / len(list_statements[i])
130128

131129
scores.append(1 - score)
132130

0 commit comments

Comments
 (0)