Added DocumentRetrievalEvaluator to Azure AI Evaluation to support evaluation of document search (Azure#39929)

* Added new evaluator code for Azure AI Evaluation

* Added TypedDict for input validation and created json schema specs for input and output schemas

* Added a temporary hack to make the example runnable; updated schema

* Implementation improvements to align with applied science recommendations

* Added docstrings and cleaned up input schema file

* Updates based on in-person feedback

* Addressed comments from the PR and SDK review

* small fix for threshold dict update

* Updates to support complex object inputs in DocumentRetrievalEvaluator

* Silence cspell errors for metric names'

* Updates to cspell.json

* Some updates for style enforcement; removed json schema files

* Reformatted with black

* Added tests, addressed a few comments and handled some edge cases

* Updates to tests and a few code fixes

* Docstring updates and added samples

* PR comments

* A few small test updates

---------

Co-authored-by: Abby Hartman <[email protected]>

Author: abhahn

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_common/_base_eval.py

@@ -495,7 +495,8 @@ def __init__(self, real_call):  # DO NOT ADD TYPEHINT PROMPT FLOW WILL SCREAM AT
     # are known to throw at this, mash them into kwargs, and then pass them into the real call.
     async def __call__(
         self, *, query=None, response=None, context=None, conversation=None, ground_truth=None,
-            tool_call=None, tool_definitions=None, messages=None, **kwargs
+            tool_call=None, tool_definitions=None, messages=None, retrieval_ground_truth=None,
+            retrieved_documents=None,**kwargs
     ):
         if conversation is not None:
             kwargs["conversation"] = conversation
@@ -515,5 +516,9 @@ async def __call__(
             kwargs["tool_definitions"] = tool_definitions
         if messages is not None:
             kwargs["messages"] = messages
+        if retrieval_ground_truth is not None:
+            kwargs["retrieval_ground_truth"] = retrieval_ground_truth
+        if retrieved_documents is not None:
+            kwargs["retrieved_documents"] = retrieved_documents
 
         return await self._real_call(**kwargs)

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_document_retrieval/__init__.py

@@ -0,0 +1,11 @@
+# ---------------------------------------------------------
+# Copyright (c) Microsoft Corporation. All rights reserved.
+# ---------------------------------------------------------
+
+from ._document_retrieval import DocumentRetrievalEvaluator, RetrievalGroundTruthDocument, RetrievedDocument
+
+__all__ = [
+    "DocumentRetrievalEvaluator",
+    "RetrievalGroundTruthDocument",
+    "RetrievedDocument"
+]