[evaluation] tests: Migrate azure-ai-evaluations tests (Azure#37201)

* chore: Add pf-azure extra as dev dependency

* tests: Copy tests verbatim from Microsoft/promptflow

* tests: Re-sync tests

* chore: Re-sync tests again

* fix: Change imports from "promptflow.evals" to "azure.ai.evaluation"

* tests,refactor: Replace promptflow-recording.{is_live,is_record,...} with az-for-python equivalents

* tests,refactor: Make SanitizedValues enum

* tests: Replace "vcr_recording" with azp equivalent

* tests,refactor: Move test data from `recordings/` -> `test_configs`

* tests, fix: Explicitly set expected caplog level

    Some unittests inspect the log messages captured by the caplog
    pytest fixture.

    Explicitly setting the caplog level resolves test failures
    when the `log_level` config value is changed from the default.

* tests,refactor: Remove setup_recording_injection_if_enabled

* style: Run isort

* tests,refactor: Remove RecordStorage

    RecordStorage appears to be a caching mechanism to accelerate
    recordings that involve flows.

* tests,refactor: Remove variable_recorder fixture

    This shadows the implementation used by the azure-sdk-for-python
    infrastructure.

* tests,refactor: Make a dev_connections fixture

* style: Run isort + black

* tests,chore: Move tests up a directory

* chore: Remove tests/e2etests/README.md

* ci: Re-enable tests in ci

* chore: Mock dev_connections when not live

* tests,fix: Don't hardcode azure_deployment

* tests,feat: Add support for recording openai requests

* fix: response.status -> response.status_code

* fix: Don't await response.text()

* tests,fix: Redirect traffic from AsyncioRequestsTransport to test proxy

    Currently, the azure-sdk-for-python infra:

      * Only patches a single async transport (AsyncioTransport)
      * Only patches the async transport when the test itself is async

    This fixture gets requested unconditionally, and patches the
    default transport the SDK uses.

    Ideally this would be part of the azure-sdk-for-python
    test infra.

* tests: Request "recorded_test" for more e2e tests

* test,refactor: Update mock config values

* tests,refactor: Remove redundant mock project_scope

* tests: Add some sanitizers

    Were taken from promptflow-recording

* chore: Add assets.json

* tests,fix: Use a FakeTokenCredential when not live

* ci,fix: Exclude tests from packages so verify_sdist finds py.typed

* tests: Add sanitizers for stainless headers and x-cv

* tests: Add a sanitizer for values from connections.json

* tests: Add return type to get_cred

* chore: Update assets.json

* tests,fix: late import NISTTokenizer

    nltk does not bundle all the data it uses in its pip install, and
    instead requires that the user manually installs them
    (`nltk.download`).

    nltk will error on the import of any class the depends on some external
    resource to work.

    The azure-sdk-for-python team's test proxy uses it's own certificate
    bundle to enable https connections to the test proxy, but this
    seems to cause `nltk.download` to fail.

    late importing NISTTokenizer allows tests to run in CI without
    immediately crashing on the import

* fix: Fix broken IndirectAttackEvaluator imports

* fix: Fix broken EvaluationMetrics import

* chore: Update assets.json

* docs: Fix docstring for IndirectAttackEvaluator

* tests,fix: Coerce string enum values to string

    Otherwise the string in the dict is the qualified name of the enum
    value.

* chore: Update assets.json

* tests: Temp skip tests

* chore: Add a minimum bound to azure-identity dependency

* chore: Bump minimum bound of numpy

         1.26.4 fixes a bug that prevented numpy from installing on
         python3.12

* chore: Bump nltk lower bound

    * 3.8.1 crashes on python3.12
    * 3.9.0 can't be imported (`import nltk`) without downloading
      "wordnet"

* ci: Temporarily disable windows tests

* ci: Temp disable python3.12 test

* chore: Bump numpy minimum bound

         On Python 3.11, pandas depends on numpy>=1.23.2

* ci: Run pypy39 on ubuntu

Author: kdestin

sdk/evaluation/azure-ai-evaluation/assets.json

@@ -0,0 +1,6 @@
+{
+  "AssetsRepo": "Azure/azure-sdk-assets",
+  "AssetsRepoPrefixPath": "python",
+  "TagPrefix": "python/evaluation/azure-ai-evaluation",
+  "Tag": "python/evaluation/azure-ai-evaluation_9ac3e64c3e"
+}

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_common/utils.py

@@ -8,16 +8,8 @@
 
 from typing import List
 
-import nltk
 import numpy as np
 
-try:
-    from nltk.tokenize.nist import NISTTokenizer
-except LookupError:
-    nltk.download("perluniprops")
-    nltk.download("punkt")
-    nltk.download("punkt_tab")
-    from nltk.tokenize.nist import NISTTokenizer
 
 
 def get_harm_severity_level(harm_score: int) -> str:
@@ -45,6 +37,16 @@ def get_harm_severity_level(harm_score: int) -> str:
 def nltk_tokenize(text: str) -> List[str]:
     """Tokenize the input text using the NLTK tokenizer."""
 
+    import nltk
+
+    try:
+        from nltk.tokenize.nist import NISTTokenizer
+    except LookupError:
+        nltk.download("perluniprops")
+        nltk.download("punkt")
+        nltk.download("punkt_tab")
+        from nltk.tokenize.nist import NISTTokenizer
+
     if not text.isascii():
         # Use NISTTokenizer for international tokenization
         tokens = NISTTokenizer().international_tokenize(text)

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/evaluators/__init__.py

@@ -23,6 +23,7 @@
 from ._relevance import RelevanceEvaluator
 from ._rouge import RougeScoreEvaluator, RougeType
 from ._similarity import SimilarityEvaluator
+from ._xpia import IndirectAttackEvaluator
 
 __all__ = [
     "CoherenceEvaluator",

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/evaluators/_xpia/__init__.py

@@ -1,4 +1,4 @@
-from ._xpia import IndirectAttackEvaluator
+from .xpia import IndirectAttackEvaluator
 
 __all__ = [
     "IndirectAttackEvaluator",

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/evaluators/_xpia/xpia.py

@@ -6,42 +6,48 @@
 
 from promptflow._utils.async_utils import async_run_allowing_running_loop
 
-from azure.ai._common.constants import EvaluationMetrics
+from azure.ai.evaluation._common.constants import EvaluationMetrics
 from azure.ai.evaluation._common.rai_service import evaluate_with_rai_service
 
 logger = logging.getLogger(__name__)
 
 
 class IndirectAttackEvaluator:
-    """
-    Initializes an XPIA (cross domain prompt injected attack) jailbreak evaluator to detect whether cross domain
-    injected attacks are present in your AI system's response.
-    :param project_scope: The scope of the Azure AI project.
-        It contains subscription id, resource group, and project name.
+    """A Cross-Domain Prompt Injection Attack (XPIA) jailbreak evaluator.
+
+    Detect whether cross domain injected attacks are present in your AI system's response.
+
+    :param project_scope: The scope of the Azure AI project. It contains subscription id, resource group, and project
+        name.
     :type project_scope: dict
     :param eval_last_turn: Set to True to evaluate only the most recent exchange in the dialogue,
         focusing on the latest user inquiry and the assistant's corresponding response. Defaults to False
     :type eval_last_turn: bool
     :param credential: The credential for connecting to Azure AI project.
     :type credential: ~azure.core.credentials.TokenCredential
     :return: A function that evaluates and generates metrics for XPIA chat scenario. Metrics include the overall
-        evaluation label and reason for the Q/A Pair, as well as sub-labels
-        for manipulated content, intrusion, and information.
+        evaluation label and reason for the Q/A Pair, as well as sub-labels for manipulated content, intrusion, and
+        information.
     :rtype: Callable
-    **Usage**
-    .. code-block:: python
-        eval_fn = IndirectAttackEvaluator(model_config)
-        result = eval_fn(question="What is the capital of France?", answer="Paris.")
-    **Output format for question-answer pair**
-    .. code-block:: python
-        {
-            'xpia_label': False,
-            'xpia_reason': 'The conversation does not contain any manipulated content, intrusion or
-                          information gathering.'
-            'xpia_information_gathering': False,
-            'xpia_intrusion': False
-            'xpia_manipulated_content': False
-        }
+        **Usage**
+
+        .. code-block:: python
+
+            eval_fn = IndirectAttackEvaluator(model_config)
+            result = eval_fn(question="What is the capital of France?", answer="Paris.")
+
+        **Output format for question-answer pair**
+
+        .. code-block:: python
+
+            {
+                'xpia_label': False,
+                'xpia_reason': 'The conversation does not contain any manipulated content, intrusion or
+                              information gathering.'
+                'xpia_information_gathering': False,
+                'xpia_intrusion': False
+                'xpia_manipulated_content': False
+            }
     """
 
     def __init__(self, project_scope: dict, eval_last_turn: bool = False, credential=None):

sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/synthetic/_model_tools/_proxy_completion_model.py

@@ -182,7 +182,7 @@ async def request_api(
 
         if response.status_code != 202:
             raise HttpResponseError(
-                message=f"Received unexpected HTTP status: {response.status} {await response.text()}", response=response
+                message=f"Received unexpected HTTP status: {response.status_code} {response.text()}", response=response
             )
 
         response = response.json()

sdk/evaluation/azure-ai-evaluation/dev_requirements.txt

@@ -6,3 +6,4 @@ pytest-asyncio
 pytest-cov
 pytest-mock
 pytest-xdist
+-e ../azure-ai-evaluation[pf-azure]

sdk/evaluation/azure-ai-evaluation/setup.py

@@ -57,7 +57,7 @@
     include_package_data=True,
     packages=find_packages(
         exclude=[
-            "tests",
+            "tests*",
             # Exclude packages that will be covered by PEP420 or nspkg
             "azure",
             "azure.ai",
@@ -69,11 +69,11 @@
         "promptflow-core>=1.15.0",
         "websocket-client>=1.2.0",
         "jsonpath_ng>=1.5.0",
-        "numpy>=1.22",
+        "numpy>=1.23.2",
         "pyjwt>=2.8.0",
-        "azure-identity",
+        "azure-identity>=1.12.0",
         "azure-core>=1.30.2",
-        "nltk>=3.8.1",
+        "nltk>=3.9.1",
         "rouge-score>=0.1.2",
     ],
     extras_require={

sdk/evaluation/azure-ai-evaluation/tests/__init__.py

@@ -0,0 +1,118 @@
+"""Implementation of an httpx.Client that forwards traffic to the Azure SDK test-proxy.
+
+.. note::
+
+    This module has side-effects!
+
+    Importing this module will replace the default httpx.Client used
+    by the openai package with one that can redirect it's traffic
+    to the Azure SDK test-proxy on demand.
+
+"""
+
+from contextlib import contextmanager
+from dataclasses import dataclass
+from typing import Iterable, Iterator, Literal, Optional
+
+import httpx
+import openai._base_client
+from typing_extensions import override
+
+
+@dataclass
+class TestProxyConfig:
+    recording_id: str
+    """The ID for the ongoing test recording."""
+
+    recording_mode: Literal["playback", "record"]
+    """The current recording mode."""
+
+    proxy_url: str
+    """The url for the Azure SDK test proxy."""
+
+
+class TestProxyHttpxClientBase:
+    recording_config: Optional[TestProxyConfig] = None
+
+    @classmethod
+    def is_recording(cls) -> bool:
+        """Whether we are forwarding requests to the test proxy
+
+        :return: True if forwarding, False otherwise
+        :rtype: bool
+        """
+        return cls.recording_config is not None
+
+    @classmethod
+    @contextmanager
+    def record_with_proxy(cls, config: TestProxyConfig) -> Iterable[None]:
+        """Forward all requests made within the scope of context manager to test-proxy.
+
+        :param TestProxyConfig config: The test proxy configuration
+        """
+        cls.recording_config = config
+
+        yield
+
+        cls.recording_config = None
+
+    @contextmanager
+    def _reroute_to_proxy(self, request: httpx.Request) -> Iterator[None]:
+        """Temporarily re-route a request to be sent throught the test proxy.
+
+        The request is modified in place, but is restored once the contextmanager exits
+
+        :param httpx.Request request: The request to update
+        :return: None
+        :rtype: None
+        """
+        assert self.is_recording(), f"{self._reroute_to_proxy.__qualname__} should only be called while recording"
+        config = self.recording_config
+        original_url = request.url
+
+        request_path = original_url.copy_with(scheme="", netloc=b"")
+        request.url = httpx.URL(config.proxy_url).join(request_path)
+
+        original_headers = request.headers
+        request.headers = request.headers.copy()
+        request.headers.setdefault(
+            "x-recording-upstream-base-uri", str(httpx.URL(scheme=original_url.scheme, netloc=original_url.netloc))
+        )
+        request.headers["x-recording-id"] = config.recording_id
+        request.headers["x-recording-mode"] = config.recording_mode
+
+        yield
+
+        request.url = original_url
+        request.headers = original_headers
+
+
+class TestProxyHttpxClient(TestProxyHttpxClientBase, openai._base_client.SyncHttpxClientWrapper):
+    @override
+    def send(self, request: httpx.Request, **kwargs) -> httpx.Response:
+        if self.is_recording():
+            with self._reroute_to_proxy(request):
+                response = super().send(request, **kwargs)
+
+            response.request.url = request.url
+            return response
+        else:
+            return super().send(request, **kwargs)
+
+
+class TestProxyAsyncHttpxClient(TestProxyHttpxClientBase, openai._base_client.AsyncHttpxClientWrapper):
+    @override
+    async def send(self, request: httpx.Request, **kwargs) -> httpx.Response:
+        if self.is_recording():
+            with self._reroute_to_proxy(request):
+                response = await super().send(request, **kwargs)
+
+            response.request.url = request.url
+            return response
+        else:
+            return await super().send(request, **kwargs)
+
+
+# openai._base_client.{Async,Sync}HttpxClientWrapper are default httpx.Clients instantiated by openai
+openai._base_client.SyncHttpxClientWrapper = TestProxyHttpxClient
+openai._base_client.AsyncHttpxClientWrapper = TestProxyAsyncHttpxClient