Skip to content

LangChain integration does not work for summarization_scoreΒ #1230

@Peilun-Li

Description

@Peilun-Li
  • I have checked the documentation and related resources and couldn't resolve my bug.

Describe the bug
EvaluatorChain errors out for summariztion_score metric. ValueError: Missing some input keys: {'answer'}

Ragas version: 0.1.15
Python version: 3.9

Code to Reproduce

from ragas.integrations.langchain import EvaluatorChain
from ragas.metrics import summarization_score 

chain=EvaluatorChain(metric=summarization_score)

chain.run({"contexts": ["hello"], "summary": "hello"})

Error trace

File /.../python3.9/site-packages/langchain/chains/base.py:284, in Chain._validate_inputs(self, inputs)
    282 missing_keys = set(self.input_keys).difference(inputs)
    283 if missing_keys:
--> 284     raise ValueError(f"Missing some input keys: {missing_keys}")
ValueError: Missing some input keys: {'answer'}

Expected behavior
Expect summarization_score could work with LangChain (EvaluatorChain)

Additional context
The same setting above works for other ragas metrics like faithfulness. Example

from ragas.integrations.langchain import EvaluatorChain
from ragas.metrics import faithfulness 

chain=EvaluatorChain(metric=faithfulness)

chain.run({"question": "What is the capital of France?",
    "answer": "The capital of France is Paris.",
    "contexts": [
        "France, in Western Europe, encompasses medieval cities, alpine villages and Mediterranean beaches. Paris, its capital, is famed for its fashion houses, classical art museums including the Louvre and monuments like the Eiffel Tower.",
        "Paris is the capital and most populous city of France. It has an area of 105 square kilometers and a population of over 2 million residents."
    ],
    "ground_truth": "Paris"
})

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions