Add agreement between different LLM calls #11
Description
Description
The implementation at this moment is using the relevance label coming from one LLM call. This could lead to labels that are not correct if the LLM hallucinates. To try to avoid this fact, we could use more LLM execution and try to implement Inter-Rater Reliability, that is "the degree of agreement among independent observers who rate, code, or assess the same phenomenon"(Wikipedia). If the Inter-Rater Reliability is too low, we must discard the label.
Idea
To assess the quality of the label assigned by the LLM, we could run the LLM multiple times and see if we have an 80% (for example) Inter-Rater Reliability on the score. If not, we discard the label since it’s not confident enough.