Add `faithfulness` check

[linear]

Summary

Add a built-in LLM-based check that evaluates whether the model's answer is faithful to the provided source material. Complementary to Groundedness — while Groundedness checks if claims are supported by context, Faithfulness focuses on whether the answer accurately represents the source without distortion.

Motivation

Faithfulness is a standard RAG metric shipped by Ragas (Faithfulness), DeepEval, and Opik. It's distinct from Groundedness in that it focuses on accurate representation rather than just grounding.

Implementation Guide

Steps

1. Create template: src/giskard/checks/prompts/judges/faithfulness.j2
   • Given an answer and source material, evaluate faithfulness
   • Check for: misrepresentation, selective quoting, distortion, unsupported claims
2. Create check: src/giskard/checks/judges/faithfulness.py
   • Subclass BaseLLMCheck, register as "faithfulness"
   • Support:
     • answer_key: JSONPathStr — JSONPath for answer (default: trace.last.outputs)
     • source: str | list[str] | None = None — source material
     • source_key: JSONPathStr | None = None — JSONPath for source
3. Add tests

Distinction from Groundedness

• Groundedness: "Is every claim in the answer supported by the context?" (binary per-claim)
• Faithfulness: "Does the answer accurately represent the source material?" (holistic assessment including distortion, misrepresentation)

Example usage

from giskard.checks import Faithfulness, Scenario

scenario = (
    Scenario(name="faithful_answer")
    .interact(
        inputs="Summarize this document",
        outputs="The document states that...",
        metadata={"source": "Original document text..."}
    )
    .check(Faithfulness(source_key="trace.last.metadata.source"))
)

Related issues

• Bump @vue/cli-plugin-unit-jest from 4.5.15 to 5.0.3 in /frontend #22 (answer_relevance), Bump pytest from 6.2.5 to 7.1.1 in /backend/app #23 (context_relevance), Giskard v.1 : AI Test #25 (hallucination)
• Bug/design partner ai inspect bugs #33 (RAGET v3)

Acceptance Criteria

• Evaluates faithfulness of answer to source material
• Detects misrepresentation and distortion
• Distinct behavior from Groundedness check
• Tests cover: faithful answer passes, distorted answer fails, partial faithfulness

[linear]

OSS-52

[mithilesh1909]

Hi, I encountered an error when running pip install -e .:

Multiple top-level packages discovered: ['libs', 'readme']

This seems to be due to setuptools auto-discovery picking up unintended directories. Restricting package discovery to libs/ in pyproject.toml should resolve it.

Happy to open a PR if this approach looks good.

[JohnnyTarrr]

Related to my comment on #2369 — VeroQ Shield handles faithfulness checking by verifying each claim in the output against provided context or live evidence. Returns a trust score (0-1) and per-claim breakdown.

The verify_output function accepts both the text and optional context/source material:

from veroq import verify_output
result = verify_output(text, context="source document text...")

Returns claim-level verdicts showing exactly which claims are grounded in the source vs. hallucinated.