Documentation for Custom LLM-as-a-judge Evaluations (#32018)

* initial commit

* improve

* undo changes to managed_evaluations file

* add to side panel menu

* be more consistent with capitalization

* remove from other languages

* greg comments

* updates

* missed capitalization

* last few edits

---------

Co-authored-by: cecilia saixue watt <[email protected]>

Author: mtullalizardi

config/_default/menus/main.en.yaml

@@ -344,7 +344,7 @@ menu:
       url: agent/supported_platforms/heroku/
       weight: 405
       parent: agent_supported_platforms
-    - name: MacOS 
+    - name: MacOS
       identifier: basic_agent_usage_osx
       url: agent/supported_platforms/osx/
       weight: 406
@@ -4780,6 +4780,11 @@ menu:
       parent: llm_obs
       identifier: llm_obs_evaluations
       weight: 4
+    - name: Custom LLM-as-a-Judge
+      url: llm_observability/evaluations/custom_llm_as_a_judge_evaluations
+      parent: llm_obs_evaluations
+      identifier: llm_obs_custom_llm_as_a_judge_evaluations
+      weight: 400
     - name: Managed
       url: llm_observability/evaluations/managed_evaluations
       parent: llm_obs_evaluations

content/en/llm_observability/evaluations/_index.md

@@ -8,34 +8,39 @@ aliases:
 
 ## Overview
 
-LLM Observability offers several ways to support evaluations. They can be configured by navigating to [**AI Observability > Settings > Evaluations**][7].
+LLM Observability offers several ways to support evaluations. They can be configured by navigating to [**AI Observability > Settings > Evaluations**][8].
 
-### Managed Evaluations
+### Custom LLM-as-a-judge evaluations
 
-Datadog builds and supports [Managed Evaluations][1] to support common use cases. You can enable and configure them within the LLM Observability application.
+[Custom LLM-as-a-judge evaluations][1] allow you to define your own evaluation logic using natural language prompts. You can create custom evaluations to assess subjective or objective criteria (like tone, helpfulness, or factuality) and run them at scale across your traces and spans.
 
-### Submit External Evaluations
+### Managed evaluations
 
-You can also submit [External Evaluations][2] using Datadog's API. This mechanism is great if you have your own evaluation system, but would like to centralize that information within Datadog.
+Datadog builds and supports [managed evaluations][2] to support common use cases. You can enable and configure them within the LLM Observability application.
 
-### Evaluation Integrations
+### Submit external evaluations
 
-Datadog also supports integrations with some 3rd party evaluation frameworks, such as [Ragas][3] and [NeMo][4].
+You can also submit [external evaluations][3] using Datadog's API. This mechanism is great if you have your own evaluation system, but would like to centralize that information within Datadog.
+
+### Evaluation integrations
+
+Datadog also supports integrations with some 3rd party evaluation frameworks, such as [Ragas][4] and [NeMo][5].
 
 ### Sensitive Data Scanner integration
 
-In addition to evaluating the input and output of LLM requests, agents, workflows, or the application, LLM Observability integrates with [Sensitive Data Scanner][5], which helps prevent data leakage by identifying and redacting any sensitive information (such as personal data, financial details, or proprietary information) that may be present in any step of your LLM application. 
+In addition to evaluating the input and output of LLM requests, agents, workflows, or the application, LLM Observability integrates with [Sensitive Data Scanner][6], which helps prevent data leakage by identifying and redacting any sensitive information (such as personal data, financial details, or proprietary information) that may be present in any step of your LLM application.
 
 By proactively scanning for sensitive data, LLM Observability ensures that conversations remain secure and compliant with data protection regulations. This additional layer of security reinforces Datadog's commitment to maintaining the confidentiality and integration of user interactions with LLMs.
 
 ### Permissions
 
-[LLM Observability Write permissions][6] are necessary to configure evaluations.
+[`LLM Observability Write` permissions][7] are necessary to configure evaluations.
 
-[1]: /llm_observability/evaluations/managed_evaluations
-[2]: /llm_observability/evaluations/external_evaluations
-[3]: /llm_observability/evaluations/ragas_evaluations
-[4]: /llm_observability/evaluations/submit_nemo_evaluations
-[5]: /security/sensitive_data_scanner/
-[6]: /account_management/rbac/permissions/#llm-observability
-[7]: https://app.datadoghq.com/llm/settings/evaluations
+[1]: /llm_observability/evaluations/custom_llm_as_a_judge_evaluations
+[2]: /llm_observability/evaluations/managed_evaluations
+[3]: /llm_observability/evaluations/external_evaluations
+[4]: /llm_observability/evaluations/ragas_evaluations
+[5]: /llm_observability/evaluations/submit_nemo_evaluations
+[6]: /security/sensitive_data_scanner/
+[7]: /account_management/rbac/permissions/#llm-observability
+[8]: https://app.datadoghq.com/llm/settings/evaluations

content/en/llm_observability/evaluations/custom_llm_as_a_judge_evaluations.md

@@ -0,0 +1,161 @@
+---
+title: Custom LLM-as-a-Judge Evaluations
+description: How to create custom LLM-as-a-judge evaluations, and how to use these evaluation results across LLM Observability.
+further_reading:
+- link: "/llm_observability/terms/"
+  tag: "Documentation"
+  text: "Learn about LLM Observability terms and concepts"
+- link: "/llm_observability/setup"
+  tag: "Documentation"
+  text: "Learn how to set up LLM Observability"
+- link: "/llm_observability/evaluations/managed_evaluations"
+  tag: "Documentation"
+  text: "Learn about managed evaluations"
+- link: "https://www.datadoghq.com/blog/llm-evaluation-framework-best-practices/"
+  tag: "Blog"
+  text: "Building an LLM evaluation framework: best practices"
+- link: "https://huggingface.co/learn/cookbook/llm_judge"
+  tag: "Hugging Face"
+  text: "Using LLM-as-a-judge for an automated and versatile evaluation"
+---
+
+Custom LLM-as-a-judge evaluations use an LLM to judge the performance of another LLM. You can define evaluation logic with natural language prompts, capture subjective or objective criteria (like tone, helpfulness, or factuality), and run these evaluations at scale across your traces and spans. 
+
+## Create a custom LLM-as-a-judge evaluation
+
+You can create and manage custom evaluations from the [Evaluations page][1] in LLM Observability.
+
+1. In Datadog, navigate to the LLM Observability [Evaluations page][1]. Select **Create Evaluation**, then select **Create your own**.
+   {{< img src="llm_observability/evaluations/custom_llm_judge_1.png" alt="The LLM Observability Evaluations page with the Create Evaluation side panel opened. The first item, 'Create your own,' is selected. " style="width:100%;" >}}
+
+1. Provide a clear, descriptive **evaluation name** (for example, `factuality-check` or `tone-eval`). You will use this name when querying evaluation results. The name must be unique within your application.
+
+1. Use the **Account** drop-down menu to select the LLM provider and corresponding account to use for your LLM judge. To connect a new account, see [connect an LLM provider][2].
+
+1. Use the **Model** drop-down menu to select a model to use for your LLM judge.
+
+1. Under **Evaluation Prompt** section, use the **Prompt Template** drop-down menu:
+   - **Create from scratch**: Use your own custom prompt (defined in the next step).
+   - **Failure to Answer**, **Prompt Injection**, **Sentiment**, etc.: Populate a pre-existing prompt template. You can use these templates as-is, or modify them to match your specific evaluation logic.
+
+1. In the **System Prompt** field, enter your custom prompt or modify a prompt template.
+   For custom prompts, provide clear instructions describing what the evaluator should assess. 
+
+   - Focus on a single evaluation goal 
+   - Include 2–3 few-shot examples showing input/output pairs, expected results, and reasoning.
+
+{{% collapse-content title="Example custom prompt" level="h4" expanded=false id="custom-prompt-example" %}}
+**System Prompt**
+```
+You will be looking at interactions between a user and a budgeting AI agent. Your job is to classify the user's intent when it comes to using the budgeting AI agent.
+
+You will be given a Span Input, which represents the user's message to the agent, which you will then classify. Here are some examples.
+
+Span Input: What are the core things I should know about budgeting?
+Classification: general_financial_advice
+
+Span Input: Did I go over budget with my grocery bills last month?
+Classification: budgeting_question
+
+Span Input: What is the category for which I have the highest budget?
+Classification: budgeting_question
+
+Span Input: Based on my past months, what is my ideal budget for subscriptions?
+Classification: budgeting_advice
+
+Span Input: Raise my restaurant budget by $50
+Classification: budgeting_request
+
+Span Input: Help me plan a trip to the Maldives
+Classification: unrelated
+```
+
+**User**
+
+```
+Span Input: {{span_input}}
+```
+{{% /collapse-content %}}
+
+7. In the **User** field, provide your user prompt. Explicitly specify what parts of the span to evaluate: Span Input (`{{span_input}}`), Output (`{{span_output}}`), or both.
+
+7. Select an evaluation output type:
+
+   - **Boolean**: True/false results (for example, "Did the model follow instructions?")
+   - **Score**: Numeric ratings (for example, a 1–5 scale for helpfulness)
+   - **Categorical**: Discrete labels (for example, "Good", "Bad", "Neutral")
+   <div class="alert alert-info">For Anthropic and Amazon Bedrock models, only the <strong>Boolean</strong> output type is available.</div>
+
+7. Define the structure of your output.
+
+   {{< tabs >}}
+   {{% tab "OpenAI" %}}
+   {{% llm-eval-output-json %}}
+   {{% /tab %}}
+
+   {{% tab "Azure OpenAI" %}}
+   {{% llm-eval-output-json %}}
+   {{% /tab %}}
+
+   {{% tab "Anthropic" %}}
+   {{% llm-eval-output-keyword %}}
+   {{% /tab %}}
+
+   {{% tab "Amazon Bedrock" %}}
+   {{% llm-eval-output-keyword %}}
+   {{% /tab %}}
+   {{< /tabs >}}
+
+7. Under **Evaluation Scope**, define the scope of your evaluation:
+   - **Application**: Select the application you want to evaluate.
+   - **Evaluate On**: Choose one of the following:
+      - **Traces**: Evaluate only root spans
+      - **All Spans**: Evaluate both root and child spans
+   - **Span Names**: (Optional) Limit evaluation to spans with certain names.
+   - **Tags**: (Optional) Limit evaluation to spans with certain tags.
+   - **Sampling Rate**: (Optional) Apply sampling (for example, 10%) to control evaluation cost.
+
+7. Use the **Test Evaluation** panel on the right to preview how your evaluator performs. You can input sample `{{span_input}}` and `{{span_output}}` values, then click **Run Evaluation** to see the LLM-as-a-Judge's output before saving. Modify your evaluation until you are satisfied with the results.
+
+{{< img src="llm_observability/evaluations/custom_llm_judge_2.png" alt="Creation flow for a custom LLM-as-a-judge evaluation. On the right, under Test Evaluation, sample span_input and span_output have been provided. An Evaluation Result textbox below displays a sample result." style="width:100%;" >}}
+
+
+## Viewing and using results
+
+After you save your evaluation, Datadog automatically runs your evaluation on targeted spans. Results are available across LLM Observability in near-real-time. You can find your custom LLM-as-a-judge results for a specific span in the **Evaluations** tab, next to all other evaluations.
+
+{{< img src="llm_observability/evaluations/custom_llm_judge_3.png" alt="The Evaluations tab of a trace, displaying custom evaluation results alongside managed evaluations." style="width:100%;" >}}
+
+Use the syntax `@evaluations.custom.<evaluation_name>` to query or visualize results.
+
+For example:
+```
+@evaluations.custom.helpfulness-check
+```
+
+{{< img src="llm_observability/evaluations/custom_llm_judge_4.png" alt="The LLM Observability Traces view. In the search box, the user has entered `@evaluations.custom.budget-guru-intent-classifier:budgeting_question` and results are populated below." style="width:100%;" >}}
+
+
+You can:
+- Filter traces by evaluation results
+- Use evaluation results as [facets][3]
+- View aggregate results in the LLM Observability Overview page's Evaluation section
+- Create [monitors][4] to alert on performance changes or regression
+
+## Best practices for reliable custom evaluations
+
+- **Start small**: Target a single, well-defined failure mode before scaling.
+- **Iterate**: Run, inspect outputs, and refine your prompt.
+- **Validate**: Periodically check evaluator accuracy using sampled traces.
+- **Document your rubric**: Clearly define what "Pass" and "Fail" mean to avoid drift over time.
+- **Re-align your evaluator**: Reassess prompt and few-shot examples when the underlying LLM updates.
+
+## Further Reading
+
+{{< partial name="whats-next/whats-next.html" >}}
+
+[1]: https://app.datadoghq.com/llm/settings/evaluations
+[2]: /llm_observability/evaluations/managed_evaluations#connect-your-llm-provider-account
+[3]: /service_management/events/explorer/facets/
+[4]: /monitors/
+

layouts/shortcodes/llm-eval-output-json.en.md

@@ -0,0 +1,56 @@
+Edit a JSON schema that defines your evaluations output type.
+
+#### Boolean
+- Edit the `description` field to further explain what true and false mean in your use case.
+
+#### Score
+- Set a `min` and `max` score for your evaluation.
+- Edit the `description` field to further explain the scale of your evaluation.
+
+#### Categorical
+- Add or remove categories by editing the JSON schema
+- Edit category names
+- Edit the `description` field of categories to further explain what they mean in the context of your evaluation.
+
+An example schema for a categorical evaluation:
+
+```
+{
+    "name": "categorical_eval",
+    "schema": {
+        "type": "object",
+        "required": [
+            "categorical_eval"
+        ],
+        "properties": {
+            "categorical_eval": {
+                "type": "string",
+                "anyOf": [
+                    {
+                        "const": "budgeting_question",
+                        "description": "The user is asking a question about their budget. The answer can be directly determined by looking at their budget and spending."
+                    },
+                    {
+                        "const": "budgeting_request",
+                        "description": "The user is asking to change something about their budget. This should involve an action that changes their budget."
+                    },
+                    {
+                        "const": "budgeting_advice",
+                        "description": "The user is asking for advice on their budget. This should not require a change to their budget, but it should require an analysis of their budget and spending."
+                    },
+                    {
+                        "const": "general_financial_advice",
+                        "description": "The user is asking for general financial advice which is not directly related to their specific budget. However, this can include advice about budgeting in general."
+                    },
+                    {
+                        "const": "unrelated",
+                        "description": "This is a catch-all category for things not related to budgeting or financial advice."
+                    }
+                ]
+            }
+        },
+        "additionalProperties": false
+    },
+    "strict": true
+}
+```

layouts/shortcodes/llm-eval-output-keyword.en.md

@@ -0,0 +1,10 @@
+Provide **True keywords** and **False keywords** that define when the evaluation result is true or false, respectively. 
+
+Datadog searches the LLM-as-a-judge's response text for your defined keywords and provides the appropriate results for the evaluation. For this reason, you should instruct the LLM to respond with your chosen keywords. 
+
+For example, if you set:
+
+- **True keywords**: Yes, yes
+- **False keywords**: No, no
+
+Then your system prompt should include something like `Respond with "yes" or "no"`.