Skip to content

Commit 717039d

Browse files
authored
Improve context relevancy (#112)
## What Improve context relevancy prompt ## Why LLM has trouble doing candidate sentence extraction. The current prompt has caused issues where context relevancy becomes zero due to a suboptimal prompt. This prompt is tested on data from Arxiv, StackOverflow, etc. fixes: #109
1 parent cd9dbc4 commit 717039d

File tree

1 file changed

+7
-17
lines changed

1 file changed

+7
-17
lines changed

src/ragas/metrics/context_relevance.py

Lines changed: 7 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -16,25 +16,11 @@
1616

1717
CONTEXT_RELEVANCE = HumanMessagePromptTemplate.from_template(
1818
"""\
19-
Task: Candidate sentence extraction.
20-
Given the question and context, extract minimum number of sentences from context required to answer the question. If the context do not contain information required to answer the question return "No candidate sentences found".
21-
22-
question: Which equation is known as worlds most famous equation?
23-
context:\nAlbert Einstein (14 March 1879 – 18 April 1955) was a German-born theoretical physicist,[5] widely ranked among the greatest and most influential scientists of all time. Best known for developing the theory of relativity, he also made important contributions to quantum mechanics, and was thus a central figure in the revolutionary reshaping of the scientific understanding of nature that modern physics accomplished in the first decades of the twentieth century.
24-
His mass–energy equivalence formula E = mc2, which arises from relativity theory, has been called "the world's most famous equation".
25-
sentences:His mass–energy equivalence formula E = mc2, which arises from relativity theory, has been called "the world's most famous equation".
26-
27-
question: Were Scott Derrickson and Ed Wood of the same nationality?
28-
context :\nScott Derrickson (born July 16, 1966) is an American director, screenwriter and producer He lives in Los Angeles, California He is best known for directing horror films such as "Sinister", "The Exorcism of Emily Rose", and "Deliver Us From Evil", as well as the 2016 Marvel Cinematic Universe installment, "Doctor Strange"Tyler Bates is an American musician, music producer, and composer for films, television, and video games. Adam Collis is an American filmmaker and actor.Conrad Brooks is an American actor.Edward Davis Wood Jr. (October 10, 1924 – December 10, 1978) was an American filmmaker, actor, writer, producer, and director.
29-
sentences:Scott Derrickson (born July 16, 1966) is an American director, screenwriter and producer. Edward Davis Wood Jr. (October 10, 1924 – December 10, 1978) was an American filmmaker, actor, writer, producer, and director.
30-
31-
question: How many were killed in the Tiananmen Square incident?
32-
context:\nTiananmen Square incident, also called June Fourth incident or 6/4, series of protests and demonstrations in China in the spring of 1989 that culminated on the night of June 3–4 with a government crackdown on the demonstrators in Tiananmen Square in Beijing.
33-
sentences: No candidate sentences found.
19+
Please extract relevant sentences from the provided context that can potentially help answer the following question. If no relevant sentences are found, or if you believe the question cannot be answered from the given context, return the phrase "Insufficient Information". While extracting candidate sentences you're not allowed to make any changes to sentences from given context.
3420
3521
question:{question}
3622
context:\n{context}
37-
sentences:""" # noqa: E501
23+
candidate sentences:\n""" # noqa: E501
3824
)
3925

4026

@@ -159,7 +145,11 @@ def _score_batch(
159145
overlap_scores = []
160146
context_sents = sent_tokenize(context)
161147
for output in n_response:
162-
indices = sent_tokenize(output)
148+
indices = (
149+
output.split("\n")
150+
if output.lower() != "insufficient information."
151+
else []
152+
)
163153
score = min(len(indices) / len(context_sents), 1)
164154
overlap_scores.append(score)
165155
if self.strictness > 1:

0 commit comments

Comments
 (0)