-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Closed
Labels
bugSomething isn't workingSomething isn't working
Description
- I have checked the documentation and related resources and couldn't resolve my bug.
Describe the bug
EvaluatorChain errors out for summariztion_score metric. ValueError: Missing some input keys: {'answer'}
Ragas version: 0.1.15
Python version: 3.9
Code to Reproduce
from ragas.integrations.langchain import EvaluatorChain
from ragas.metrics import summarization_score
chain=EvaluatorChain(metric=summarization_score)
chain.run({"contexts": ["hello"], "summary": "hello"})
Error trace
File /.../python3.9/site-packages/langchain/chains/base.py:284, in Chain._validate_inputs(self, inputs)
282 missing_keys = set(self.input_keys).difference(inputs)
283 if missing_keys:
--> 284 raise ValueError(f"Missing some input keys: {missing_keys}")
ValueError: Missing some input keys: {'answer'}
Expected behavior
Expect summarization_score could work with LangChain (EvaluatorChain)
Additional context
The same setting above works for other ragas metrics like faithfulness. Example
from ragas.integrations.langchain import EvaluatorChain
from ragas.metrics import faithfulness
chain=EvaluatorChain(metric=faithfulness)
chain.run({"question": "What is the capital of France?",
"answer": "The capital of France is Paris.",
"contexts": [
"France, in Western Europe, encompasses medieval cities, alpine villages and Mediterranean beaches. Paris, its capital, is famed for its fashion houses, classical art museums including the Louvre and monuments like the Eiffel Tower.",
"Paris is the capital and most populous city of France. It has an area of 105 square kilometers and a population of over 2 million residents."
],
"ground_truth": "Paris"
})
dosubot
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't working