Make expectations in FactCheckingEvaluator's prompt more explicit #1969
Description
FactCheckingEvaluator has the following prompt template:
Document: \n {document}\n
Claim: \n {claim}
But there is no clear expectation set as to how the LLM should respond. Meanwhile, the decision as to whether the evaluation passes or not, is based simply on whether the response to that prompt be "yes" or not.
What I've found is that more often than not, the response to that prompt is a long-winded explanation as to why (or why not) the claim is supported by the document. Even if the response includes the word "yes", it's not solely "yes" and therefore equalsIgnoreCase() returns false.
I propose that the prompt be modified to include very explicit instructions to direct it to return "yes" or "no". Something like this:
Respond with only the single word "yes" if the document supports the claim made. Otherwise respond "no".
Document: \n {document}\n
Claim: \n {claim}
When run against Minstral:7B and Llama3.2 in Ollama, the result is more consistently in line with the factuality of the claim.