Will LLM-as-a-Judge Process Uploaded Files in a Multi-LLM Chat Application?

[harshjhajarvis-svg]

Describe your question

I’m building a multi-LLM chat application, and I had a question about how Langfuse’s LLM-as-a-Judge works.

If a user uploads or passes a file into the system, does the LLM-as-a-Judge in Langfuse directly read and analyze the file contents, or does it only evaluate the textual responses generated by the LLMs?

I haven’t tested this yet. We’re currently planning to use Langfuse mainly for observability and monitoring, and I’m working on a proof of concept. While doing so, this question came up.

Langfuse Cloud or Self-Hosted?

Langfuse Cloud

If Self-Hosted

No response

If Langfuse Cloud

No response

SDK and integration versions

No response

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[Lotte-Verheyden]

Hi @harshjhajarvis-svg, thanks for raising this!
Currently, Langfuse's built-in LLM-as-a-Judge evaluators do not support processing uploaded files or images. When you log a trace that includes uploaded files in your multi-LLM chat application, the evaluator will only see the text representation of the messages, not the actual file content.

A workaround for this is to use the SDK to run experiments outside of the Langfuse UI:

• fetch the trace via the SDK
• run your evaluation with the full multimodal context
• post scores back to Langfuse