initial commit

Author: mtullalizardi

content/en/llm_observability/evaluations/custom_llm_as_a_judge_evaluations.md

@@ -0,0 +1,169 @@
+---
+title: Custom LLM-as-a-Judge Evaluations
+description: Learn how to create custom evaluations for your LLM applications using LLM-as-a-Judge.
+further_reading:
+- link: "/llm_observability/terms/"
+  tag: "Documentation"
+  text: "Learn about LLM Observability terms and concepts"
+- link: "/llm_observability/setup"
+  tag: "Documentation"
+  text: "Learn how to set up LLM Observability"
+- link: "/llm_observability/evaluations/managed_evaluations"
+  tag: "Documentation"
+  text: "Learn about Managed Evaluations"
+- link: "https://www.datadoghq.com/blog/llm-evaluation-framework-best-practices/"
+  tag: "Blog"
+  text: "Building an LLM evaluation framework: best practices"
+---
+
+## Overview
+
+Custom LLM-as-a-Judge Evaluations let you define your own evaluation logic to automatically assess your LLM applications. You can use natural language prompts to capture subjective or objective criteria—like tone, helpfulness, or factuality—and run them at scale across your traces and spans.
+
+This provides a flexible, automated way to monitor model quality, detect regressions, and track improvements over time.
+
+## How it works
+
+Custom LLM-as-a-Judge Evaluations use an LLM to judge the performance of another LLM.
+
+You define:
+- The criteria (via prompt text)
+- What is evaluated (for example, a span's output)
+- The model (for example, GPT-4o)
+- The output type (Boolean, numeric score, or categorical label)
+
+Datadog then runs this evaluation logic automatically against your spans, recording results as structured metrics that you can query, visualize, and monitor.
+
+## Create a custom evaluation
+
+You can create and manage custom evaluations from the [Evaluations page][1] in LLM Observability.
+
+### 1. Name your evaluation
+
+Give your evaluation a clear, descriptive name (for example, `factuality-check` or `tone-eval`).
+
+You'll use this name later when querying evaluation results. The name has to be unique within your application.
+
+### 2. Choose a model provider and model
+
+Select your LLM account and the model you wish to use for the evaluation. If you do not have an LLM account already integrated with LLM Observability, follow [these instructions][2] to connect an LLM provider.
+
+### 3. Define the evaluation prompt
+
+In the **Evaluation Prompt** section, you can either:
+- Use preexisting prompt templates, including:
+  - Failure to Answer
+  - Prompt Injection
+  - Sentiment
+  - Topic Relevancy
+  - Toxicity
+- Or create an evaluation from scratch by writing your own criteria.
+
+Templates can be used as-is or fully customized to match your specific evaluation logic.
+
+#### Writing a custom prompt
+
+In the **System Prompt** field, write clear instructions describing what the evaluator should assess.
+
+- Focus on a single evaluation goal
+- Include 2–3 few-shot examples showing input/output pairs, expected results, and reasoning.
+
+In the **User Prompt** field, explicitly specify what parts of the span to evaluate: Span Input, Output, or Both.
+
+**Example System Prompt:**
+
+{{< code-block lang="text" >}}
+You will be looking at interactions between a user and a budgeting AI agent. Your job is to classify the user's intent when it comes to using the budgeting AI agent.
+
+You will be given a Span Input, which represents the user's message to the agent, which you will then classify. Here are some examples.
+
+Span Input: What are the core things I should know about budgeting?
+Classification: general_financial_advice
+
+Span Input: Did I go over budget with my grocery bills last month?
+Classification: budgeting_question
+
+Span Input: What is the category for which I have the highest budget?
+Classification: budgeting_question
+
+Span Input: Based on my past months, what is my ideal budget for subscriptions?
+Classification: budgeting_advice
+
+Span Input: Raise my restaurant budget by $50
+Classification: budgeting_request
+
+Span Input: Help me plan a trip to the Maldives
+Classification: unrelated
+{{< /code-block >}}
+
+**Example User Message:**
+
+{{< code-block lang="text" >}}
+Span Input: {{span_input}}
+{{< /code-block >}}
+
+### 4. Choose output type
+
+Define the expected output schema for the evaluator:
+
+- **Boolean** – True/False results (for example, "Did the model follow instructions?")
+- **Score** – Numeric rating (for example, 1–5 scale for helpfulness)
+- **Categorical** – Discrete labels (for example, "Good", "Bad", "Neutral")
+
+The schema ensures your results are structured for querying and dashboarding. For Anthropic and Bedrock models, only Boolean output types are available.
+
+You can preview and refine your logic in the **Test Evaluation** panel by providing sample span input/output and clicking **Run Evaluation** to verify outputs before saving.
+
+### 5. Configure filters and sampling
+
+Choose which application and spans to evaluate:
+
+- **Traces** – Evaluate only root spans
+- **All Spans** – Include all spans
+- **Span Names** – Target spans by name
+- **Tags** – Limit evaluation to spans with certain tags
+
+Optionally, apply sampling (for example, 10%) to control evaluation cost.
+
+### 6. Test your evaluation
+
+Use the **Test Evaluation** panel on the right to preview how your evaluator performs.
+
+You can input sample `{{span_input}}` and `{{span_output}}` values, then click **Run Evaluation** to see the LLM-as-a-Judge's output before saving. Modify your evaluation until you are satisfied with the results.
+
+## Viewing and using results
+
+After an evaluation has been saved, it will automatically run on targeted spans and results will be available across LLM Observability in near-realtime.
+
+Use the syntax `@evaluations.custom.<evaluation_name>` to query or visualize results.
+
+For example:
+```
+@evaluations.custom.helpfulness-check
+```
+
+You can:
+- Filter traces by evaluation results
+- Use evaluation results as facets
+- View aggregate results in the LLM Observability Overview page's Evaluation section
+- Create monitors to alert on performance changes or regression
+
+## Best practices for reliable custom evaluations
+
+- **Start small**: Target a single, well-defined failure mode before scaling.
+- **Iterate**: Run, inspect outputs, and refine your prompt.
+- **Validate**: Periodically check evaluator accuracy using sampled traces.
+- **Document your rubric**: Clearly define what "Pass" and "Fail" mean to avoid drift over time.
+- **Re-align your evaluator**: Reassess prompt and few-shot examples when the underlying LLM updates.
+
+For more resources on best practices, check out [Building an LLM evaluation framework: best practices][3] and [Using LLM-as-a-judge for automated and versatile evaluation][4].
+
+## Further Reading
+
+{{< partial name="whats-next/whats-next.html" >}}
+
+[1]: https://app.datadoghq.com/llm/settings/evaluations
+[2]: /llm_observability/evaluations/managed_evaluations#connect-your-llm-provider-account
+[3]: https://www.datadoghq.com/blog/llm-evaluation-framework-best-practices/
+[4]: https://huggingface.co/learn/cookbook/llm_judge
+

content/en/llm_observability/evaluations/managed_evaluations.md

@@ -134,8 +134,8 @@ Each of these metrics has `ml_app`, `model_server`, `model_provider`, `model_nam
 #### Topic relevancy
 
 This check identifies and flags user inputs that deviate from the configured acceptable input topics. This ensures that interactions stay pertinent to the LLM's designated purpose and scope.
-  
-| Evaluation Stage | Evaluation Method | Evaluation Definition | 
+
+| Evaluation Stage | Evaluation Method | Evaluation Definition |
 |---|---|---|
 | Evaluated on Input | Evaluated using LLM | Topic relevancy assesses whether each prompt-response pair remains aligned with the intended subject matter of the Large Language Model (LLM) application. For instance, an e-commerce chatbot receiving a question about a pizza recipe would be flagged as irrelevant.  |
 
@@ -156,7 +156,7 @@ This check identifies instances where the LLM makes a claim that disagrees with
 
 {{< img src="llm_observability/evaluations/hallucination_1.png" alt="A Hallucination evaluation detected by an LLM in LLM Observability" style="width:100%;" >}}
 
-| Evaluation Stage | Evaluation Method | Evaluation Definition | 
+| Evaluation Stage | Evaluation Method | Evaluation Definition |
 |---|---|---|
 | Evaluated on Output | Evaluated using LLM | Hallucination flags any output that disagrees with the context provided to the LLM. |
 
@@ -217,13 +217,13 @@ This check identifies instances where the LLM fails to deliver an appropriate re
 
 {{< img src="llm_observability/evaluations/failure_to_answer_1.png" alt="A Failure to Answer evaluation detected by an LLM in LLM Observability" style="width:100%;" >}}
 
-| Evaluation Stage | Evaluation Method | Evaluation Definition | 
+| Evaluation Stage | Evaluation Method | Evaluation Definition |
 |---|---|---|
 | Evaluated on Output | Evaluated using LLM | Failure To Answer flags whether each prompt-response pair demonstrates that the LLM application has provided a relevant and satisfactory answer to the user's question.  |
 
 ##### Failure to Answer Configuration
 <div class="alert alert-info">Configuring failure to answer evaluation categories is supported if OpenAI or Azure OpenAI is selected as your LLM provider.</div>
-You can configure the Failure to Answer evaluation to use specific categories of failure to answer, listed in the following table. 
+You can configure the Failure to Answer evaluation to use specific categories of failure to answer, listed in the following table.
 
 | Configuration Option | Description | Example(s) |
 |---|---|---|
@@ -245,7 +245,7 @@ Afrikaans, Albanian, Arabic, Armenian, Azerbaijani, Belarusian, Bengali, Norwegi
 
 {{< img src="llm_observability/evaluations/language_mismatch_1.png" alt="A Language Mismatch evaluation detected by an open source model in LLM Observability" style="width:100%;" >}}
 
-| Evaluation Stage | Evaluation Method | Evaluation Definition | 
+| Evaluation Stage | Evaluation Method | Evaluation Definition |
 |---|---|---|
 | Evaluated on Input and Output | Evaluated using Open Source Model | Language Mismatch flags whether each prompt-response pair demonstrates that the LLM application answered the user's question in the same language that the user used.  |
 
@@ -255,7 +255,7 @@ This check helps understand the overall mood of the conversation, gauge user sat
 
 {{< img src="llm_observability/evaluations/sentiment_1.png" alt="A Sentiment evaluation detected by an LLM in LLM Observability" style="width:100%;" >}}
 
-| Evaluation Stage | Evaluation Method | Evaluation Definition | 
+| Evaluation Stage | Evaluation Method | Evaluation Definition |
 |---|---|---|
 | Evaluated on Input and Output | Evaluated using LLM | Sentiment flags the emotional tone or attitude expressed in the text, categorizing it as positive, negative, or neutral.   |
 
@@ -265,7 +265,7 @@ This check evaluates whether your LLM chatbot can successfully carry out a full
 
 {{< img src="llm_observability/evaluations/goal_completeness.png" alt="A Goal Completeness evaluation detected by an LLM in LLM Observability" style="width:100%;" >}}
 
-| Evaluation Stage | Evaluation Method | Evaluation Definition | 
+| Evaluation Stage | Evaluation Method | Evaluation Definition |
 |---|---|---|
 | Evaluated on session | Evaluated using LLM | Goal Completeness assesses whether all user intentions within a multi-turn interaction were successfully resolved. The evaluation identifies resolved and unresolved intentions, providing a completeness score based on the ratio of unresolved to total intentions. |
 
@@ -314,7 +314,7 @@ This check evaluates whether the agent has successfully selected the appropriate
 
 {{< img src="llm_observability/evaluations/tool_selection_failure.png" alt="A tool selection failure detected by the evaluation in LLM Observability" style="width:100%;" >}}
 
-| Evaluation Stage | Evaluation Method | Evaluation Definition | 
+| Evaluation Stage | Evaluation Method | Evaluation Definition |
 |---|---|---|
 | Evaluated on LLM spans| Evaluated using LLM | Tool Selection verifies that the tools chosen by the LLM align with the user's request and the available tools. The evaluation identifies cases where irrelevant or incorrect tool calls were made.|
 
@@ -339,9 +339,9 @@ def subtract_numbers(a: int, b: int) -> int:
     Subtracts two numbers.
     """
     return a - b
-    
 
-# List of tools available to the agent 
+
+# List of tools available to the agent
 math_tutor_agent = Agent(
     name="Math Tutor",
     handoff_description="Specialist agent for math questions",
@@ -360,21 +360,21 @@ history_tutor_agent = Agent(
 )
 
 # The triage agent decides which specialized agent to hand off the task to — another type of tool selection covered by this evaluation.
-triage_agent = Agent(  
+triage_agent = Agent(
     'openai:gpt-4o',
     model_settings=ModelSettings(temperature=0),
-    instructions='What is the sum of 1 to 10?',  
+    instructions='What is the sum of 1 to 10?',
     handoffs=[math_tutor_agent, history_tutor_agent],
 )
 {{< /code-block >}}
 
 #### Tool argument correctness
 
-This check looks at the arguments provided to a selected tool, and it evaluates whether these arguments match the expected type and make sense given the tool's context. 
+This check looks at the arguments provided to a selected tool, and it evaluates whether these arguments match the expected type and make sense given the tool's context.
 
 {{< img src="llm_observability/evaluations/tool_argument_correctness_error.png" alt="A tool argument correctness error detected by the evaluation in LLM Observability" style="width:100%;" >}}
 
-| Evaluation Stage | Evaluation Method | Evaluation Definition | 
+| Evaluation Stage | Evaluation Method | Evaluation Definition |
 |---|---|---|
 | Evaluated on LLM spans| Evaluated using LLM | Tool Argument Correctness verifies that the arguments provided to a tool by the LLM are correct and contextually relevant. This evaluation identifies cases where the arguments provided to the tool are incorrect according to the tool schema (for example: the argument is expected to be an integer rather than a string) and are not relevant (for example: the argument is a country, but the model provides the name of a city).|
 
@@ -403,7 +403,7 @@ def subtract_numbers(a: int, b: int) -> int:
     """
     return a - b
 
-    
+
 def multiply_numbers(a: int, b: int) -> int:
     """
     Multiplies two numbers.
@@ -441,7 +441,7 @@ history_tutor_agent = Agent(
 )
 
 # Create the triage agent
-# Note: pydantic_ai handles handoffs differently - you'd typically use result_type 
+# Note: pydantic_ai handles handoffs differently - you'd typically use result_type
 # or custom logic to route between agents
 triage_agent = Agent(
     'openai:gpt-5-nano',
@@ -470,27 +470,27 @@ result = triage_agent.run_sync(
 This check evaluates each input prompt from the user and the response from the LLM application for toxic content. This check identifies and flags toxic content to ensure that interactions remain respectful and safe.
 
 {{< img src="llm_observability/evaluations/toxicity_1.png" alt="A Toxicity evaluation detected by an LLM in LLM Observability" style="width:100%;" >}}
-  
-| Evaluation Stage | Evaluation Method | Evaluation Definition | 
+
+| Evaluation Stage | Evaluation Method | Evaluation Definition |
 |---|---|---|
 | Evaluated on Input and Output | Evaluated using LLM | Toxicity flags any language or behavior that is harmful, offensive, or inappropriate, including but not limited to hate speech, harassment, threats, and other forms of harmful communication. |
 
 ##### Toxicity configuration
 
 <div class="alert alert-info">Configuring toxicity evaluation categories is supported if OpenAI or Azure OpenAI is selected as your LLM provider.</div>
-You can configure toxicity evaluations to use specific categories of toxicity, listed in the following table. 
+You can configure toxicity evaluations to use specific categories of toxicity, listed in the following table.
 
-| Category | Description | 
+| Category | Description |
 |---|---|
-| Discriminatory Content | Content that discriminates against a particular group, including based on race, gender, sexual orientation, culture, etc.  | 
-| Harassment | Content that expresses, incites, or promotes negative or intrusive behavior toward an individual or group. | 
-| Hate | Content that expresses, incites, or promotes hate based on race, gender, ethnicity, religion, nationality, sexual orientation, disability status, or caste. | 
-| Illicit | Content that asks, gives advice, or instruction on how to commit illicit acts. | 
-| Self Harm | Content that promotes, encourages, or depicts acts of self-harm, such as suicide, cutting, and eating disorders. | 
-| Sexual | Content that describes or alludes to sexual activity.  | 
-| Violence | Content that discusses death, violence, or physical injury. | 
-| Profanity | Content containing profanity. | 
-| User Dissatisfaction | Content containing criticism towards the model. *This category is only available for evaluating input toxicity.* | 
+| Discriminatory Content | Content that discriminates against a particular group, including based on race, gender, sexual orientation, culture, etc.  |
+| Harassment | Content that expresses, incites, or promotes negative or intrusive behavior toward an individual or group. |
+| Hate | Content that expresses, incites, or promotes hate based on race, gender, ethnicity, religion, nationality, sexual orientation, disability status, or caste. |
+| Illicit | Content that asks, gives advice, or instruction on how to commit illicit acts. |
+| Self Harm | Content that promotes, encourages, or depicts acts of self-harm, such as suicide, cutting, and eating disorders. |
+| Sexual | Content that describes or alludes to sexual activity.  |
+| Violence | Content that discusses death, violence, or physical injury. |
+| Profanity | Content containing profanity. |
+| User Dissatisfaction | Content containing criticism towards the model. *This category is only available for evaluating input toxicity.* |
 
 The toxicity categories in this table are informed by: [Banko et al. (2020)][14], [Inan et al. (2023)][15], [Ghosh et al. (2024)][16], [Zheng et al. (2024)][17].
 
@@ -500,13 +500,13 @@ This check identifies attempts by unauthorized or malicious authors to manipulat
 
 {{< img src="llm_observability/evaluations/prompt_injection_1.png" alt="A Prompt Injection evaluation detected by an LLM in LLM Observability" style="width:100%;" >}}
 
-| Evaluation Stage | Evaluation Method | Evaluation Definition | 
+| Evaluation Stage | Evaluation Method | Evaluation Definition |
 |---|---|---|
 | Evaluated on Input | Evaluated using LLM | [Prompt Injection][13] flags any unauthorized or malicious insertion of prompts or cues into the conversation by an external party or user. |
 
 ##### Prompt injection configuration
 <div class="alert alert-info">Configuring prompt injection evaluation categories is supported if OpenAI or Azure OpenAI is selected as your LLM provider.</div>
-You can configure the prompt injection evaluation to use specific categories of prompt injection, listed in the following table. 
+You can configure the prompt injection evaluation to use specific categories of prompt injection, listed in the following table.
 
 | Configuration Option | Description | Example(s) |
 |---|---|---|
@@ -520,8 +520,8 @@ You can configure the prompt injection evaluation to use specific categories of
 This check ensures that sensitive information is handled appropriately and securely, reducing the risk of data breaches or unauthorized access.
 
 {{< img src="llm_observability/evaluations/sensitive_data_scanning_1.png" alt="A Security and Safety evaluation detected by the Sensitive Data Scanner in LLM Observability" style="width:100%;" >}}
-  
-| Evaluation Stage | Evaluation Method | Evaluation Definition | 
+
+| Evaluation Stage | Evaluation Method | Evaluation Definition |
 |---|---|---|
 | Evaluated on Input and Output | Sensitive Data Scanner | Powered by the [Sensitive Data Scanner][4], LLM Observability scans, identifies, and redacts sensitive information within every LLM application's prompt-response pairs. This includes personal information, financial data, health records, or any other data that requires protection due to privacy or security concerns. |