Why faithfullness is NaN most of the times even when context and answer both makes sense.

[repl-mira-agarwal]

I am using Bedrock for my RAG and faithfullness is NaN most of the times even when context and answer both makes sense. The same problem is also there for the amnesty dataset shared in ragas docs.

Ragas version: 0.1.0

Code to Reproduce
from datasets import load_dataset
from ragas import evaluate

amnesty_qa = load_dataset("explodinggradients/amnesty_qa", "english_v2")
result = evaluate(amnesty_qa["eval"],metrics=[context_precision, faithfulness, answer_relevancy])
result.to_pandas()

Error trace

I see similar issue in my own dataset as well, but it's not clear why failthfulness is NaN. Could you please give some clarity what is the issue here.

[christdahl]

I have experienced the same issue. faithfullness is always 0 when gpt-4 is used as judge, and NaN when gpt-3.5 is used as judge. version 0.1.0. The same behaviour occurs when running the 0.0.21 version and changing the llm to gpt-4 (default is 'gpt-3.5-turbo-16k')

from ragas.llms.openai import OpenAI
faithfullness.llm = OpenAI(model="gpt-4")

[shahules786]

hey @christdahl , I am not able to reproduce the issue using gpt4/3.5
If you're trying to customise models, please follow the updated docs here for 0.1 version

[shahules786]

Hi @repl-mira-agarwal Found the issue with bedrock, but this has been known to us for a while now. Bedrock as model seems completely incapable of producing valid JSON outputs. for example even with clear description and demonstrations this is how bedrock response to a query in faithfulness.

 Here are my responses:

statement_1: Criminalizing abortion disproportionately impacts marginalized groups, exacerbating existing inequalities and barriers to healthcare.
Verdict: Yes 
Reason: The context indicates that marginalized groups are disproportionately affected by abortion criminalization, supporting this statement.

statement_2: When abortion is illegal, marginalized individuals may resort to unsafe methods, facing severe health risks and even death.
Verdict: Yes
Reason: The context states that criminalization leads to unsafe abortions and increased maternal mortality, supporting this statement.

statement_3: The economic burden of accessing abortion increases for marginalized individuals when it is criminalized.
Verdict: Null
Reason: The context does not provide information to support or refute this statement.

statement_4: Criminalizing abortion limits reproductive autonomy and choices for marginalized groups. 
Verdict: Yes
Reason: The context indicates criminalization undermines access to services, supporting limited autonomy.

statement_5: It can lead to the criminalization of pregnancy itself, resulting in discrimination and punishment of marginalized women.
Verdict: Null 
Reason: The context does not directly address criminalization of pregnancy.

statement_6

Is there a way that you know of to improve this to help produce json outputs from bedrock?

[shahules786]

Hey guys, just made a couple of changes that improved the filled values.
#588 #581
feel free to try out by installing from source once merged.

[pberger514]

feel free to try out by installing from source once merged.

Can we get a (minor?) release for these? I'm also seeing some problems with faithfulness and context_recall on gpt4 where often only the first item of the list gets parsed.

(This is probably due to the "find outermost json" function, actually a fix for this would be to run "find outermost json" recursively until no json is found -- I'd be happy to write up this feature if there's interest).

[shahules786]

Hey @pberger514 we can do a minor release tomorrow.

I'd be happy to write up this feature if there's interest

that would be great, please do :)

[shabie]

What model is being used for Bedrock? Also what prompt?

[repl-mira-agarwal]

Hey guys, just made a couple of changes that improved the filled values. #588 #581 feel free to try out by installing from source once merged.

Thank you ShahuI for looking into this. I see this PR is merged but when I try installing updated RAGAS in Bedrock, it is now failing in importing ragas.langchain which is a wrapper llm required to pass bedrock model to RAGAS evaluate function. Could you please point out what could be the issue here.

[repl-mira-agarwal]

What model is being used for Bedrock? Also what prompt?

Model I used for bedrock is meta.llama2-70b-chat-v1 and prompt is ChatPromptTemplate.

[rogeriochaves]

hey there, I've tried version v0.1.2 but still getting nan issues most of the time with gpt-3.5-turbo-1106 and those very simple inputs:

    dataset = Dataset.from_dict(
        {
            "question": ["What is the capital of France?"],
            "answer": ["The capital of France is Paris."],
            "contexts": [["France is a country in Europe.", "Paris is a city in France."]],
            "ground_truth": [""],
        }
    )

if I remove the second context example ("Paris is a city in France."), then it works just fine, which is quite odd

for the meantime I'm reverting back to v0.1.0 where it all works fine

edit: my bad, on v0.1.0 it also happens, maybe less, but I see issues in prod

[rere950303]

@rogeriochaves @shahules786 thanks for your work.

When performing ragas test evaluation, using gpt-4-0125-preview, and likewise, sometimes I get a nan value for faithfulness metric. Is it a bug?

[jlonge4]

@shahules786 using Bedrock (more specifically any of the Anthropic Claude models v1-2.1) seems to always result in NaN as Claude is incapable of adhering to a specific output format. May have to get creative with that. Any ideas?

[prufrock349]

I am having this issue of nan faithfulness score with seemingly random inputs/outputs. I am using text-embedding-3-small. Is there any update on how to handle this?
Version 0.1.4, Python v 3.10.13