Skip to content

Commit d4042ff

Browse files
slister1001Nagkumar Arkalgudnagkumar91Nagkumar ArkalgudMilesHolland
authored
Introducing RedTeam (Azure#39898)
* remove redundant quotes * Fix typo * pylint fix * Update broken tests * Include the grounding json in the manifest * Fix typo * Come on package * Release 1.0.0b5 * Notice from Chang * Remove adv_conv template parameters from the outputs * Update chanagelog * Experimental tags on adv scenarios * Readme fix onbreaking change * Add the category and both user and assistant context to the response of qr_json_lines * Update changelog * Rename _kwargs to _options * _options as prefix * update troubleshooting for simulator * Rename according to suggestions * Clean up readme * more links * Bugfix: zip_longest created null parameters * Updated changelog * zip does the job * remove ununsed import * Fix changelog merge * Remove print statements * Adding pyrit dependency * updates * updates * Make pyrit extra * Set up pyrit as extra correctly * Limit the number of turns * adding sample * baseline sample with evals * updates * Update setup to use roman's branch * callback chat target * sample update * add local callback chat target * Revert to adding main pyrit as extra * add option for simulation only * budget first pass error on model target * low budget working (without many shot) * Adding RedTeamAgent * updates for parallelism and cleanup * add different converters, dispose of memory * add typespec autogen files * updates * removing unused orchestrators and ensuring baseline always included * first attempt to get attack objectives * retrieve attack objectives sucessfully * cachine to retrieve multiple objectives for each of the attack strategy * Add generated client * Add content * Remove params argument from get method call * scorecard and more targets * I did mess up that merge oops * Make attack objective generator mandatory * Remove debug statement from _red_team_agent.py * get rai call working * nits * remove change not needed * whitespace * mock call to evaluate * Remove caching for objectives, get num_objectives from the attack objective generator * update the sample with attack objective generator * scorecard updates * remove baseline from detailed_joint_risk_attack_asr * smaller changes * Update all the content safety evalutors to have a pass/fail result and treshold * Update groundedness service based * Binary results for prompt based evaluators * Update changelog * new typespec gen and readme * Newly generated client responds with objectives properly * mlflow run * Pass -> pass Fail -> fail * Add thresholds to NLP evals * mlflow updates * init idea, waiting for tunnels or int to work * Update sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_gleu/_gleu.py Co-authored-by: Copilot <[email protected]> * Update sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_evaluators/_common/_base_eval.py Co-authored-by: Copilot <[email protected]> * print scorecard init call to evaluate and mock attackobjectives until int is fixed * Make a call to jailbreak and prepend response * flip call to evaluate instead of mock * Binarization in rouge * Adding threshold to all evaluators * adding progress bar * fixing and supressing errors * more updates * syntax error * More syntax fifxes * Typo fixes * print a message if exception occurs for binary result calc * Final typo * start MLFlow run earlier * Update built in evals test * updates for call to evaluate * draft of binarization * RE add the previously removed _label * Trying a fix for the test * Why ar we checking len of keys instead of the keys themselves * refactoring * Update sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/red_team_agent/red_team_agent_result.py Co-authored-by: Nagkumar Arkalgud <[email protected]> * Update sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/red_team_agent/red_team_agent_result.py Co-authored-by: Nagkumar Arkalgud <[email protected]> * address feedback * Update redundant comment and change to * Yaay tests passed * Fix bug * uncomment recording * updates * Update * use the values for accessing keys * updates * updates * parsing risk category * Handle data only scenario * update * Fix the keys for risk category to be lower case in result and show int url for ai studio * Fix the way we render results * updates * updates * Updates to use the new generated client * updates * version pin pyrit * Update to include autogen files in setup * Add init files and update setup * Removed mocks * risk category to output and update to asr calc * revert unnecessary changes * debug content filter error for open ai target * minor updates * mark startergies that are not supported * update jailbreak retrieval * update objective filtering logic * fix safety eval unit tests * remove empty utils subfolder * init attempt to move things to diff files and add tests * Fix the cspell error * feat(security): Add RedTeamAgent for AI system vulnerability assessment Implements a comprehensive RedTeamAgent feature for systematically testing AI system security vulnerabilities using various attack strategies. Key additions: - Red Team Agent class with support for multiple attack strategies (Base64, ROT13, Jailbreak, etc.) - Risk category assessment across Violence, SelfHarm, Sexual, and HateUnfairness domains - MLflow integration for experiment tracking and result visualization - Comprehensive scoring metrics including Attack Success Rate (ASR) - Detailed test coverage for all major components - Updated CHANGELOG.md with feature documentation The RedTeamAgent helps security teams and AI developers evaluate system robustness against potential attacks and provides detailed analytics on vulnerabilities. * Revert all the changes to difficulty * refactor(redteam): Simplify scorecard formatting output Removed redundant 'Scorecard:' header and studio URL from output for cleaner display. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * test(redteam): Update formatting_utils tests Updated test_formatting_utils.py to match the simplified scorecard format by removing assertions for removed elements. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * test(redteam): Fix Red Team Agent unit tests Updated test_red_team_agent.py to properly mock logging, file handlers, and tempfile operations to support scan-specific output folders and MLflow integration. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * feat(redteam): Add output directory support to logging Modified logging_utils.py to accept output_dir parameter for scan-specific log files. 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <[email protected]> * making it work, thanks claude * Allow custom attac objectives * keeping a count of timeout, add a test * Update init param and test * update to aog and scan name * Skip work in progress tests * fix timeout and other small tweaks * add init file to utils * Make redteam agent tests optional CI stage * trying to update ci to require installation of redteam extra * update naming, unit tests * minor updates * update sample with release installation * remove sample notebook * Please pass CI analyze * Again CI analyze please pass * Again CI analyze please pass --------- Co-authored-by: Nagkumar Arkalgud <[email protected]> Co-authored-by: Nagkumar Arkalgud <[email protected]> Co-authored-by: Nagkumar Arkalgud <[email protected]> Co-authored-by: Miles Holland <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: Claude <[email protected]>
1 parent 093efca commit d4042ff

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

58 files changed

+13332
-163
lines changed

‎sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_exceptions.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -84,6 +84,7 @@ class ErrorTarget(Enum):
8484
UNKNOWN = "Unknown"
8585
CONVERSATION = "Conversation"
8686
TOOL_CALL_ACCURACY_EVALUATOR = "ToolCallAccuracyEvaluator"
87+
RED_TEAM = "RedTeam"
8788

8889

8990
class EvaluationException(AzureError):

‎sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_safety_evaluation/__init__.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,3 @@
11
# ---------------------------------------------------------
22
# Copyright (c) Microsoft Corporation. All rights reserved.
3-
# ---------------------------------------------------------
3+
# ---------------------------------------------------------

‎sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_safety_evaluation/_generated_rai_client.py‎

Whitespace-only changes.

‎sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/_safety_evaluation/_safety_evaluation.py‎

Lines changed: 142 additions & 147 deletions
Large diffs are not rendered by default.

‎sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/autogen/README.md‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# Azure AI Evaluation client library for Python
2+
3+
## References
4+
5+
- [How to generate client libraries](https://azure.github.io/typespec-azure/docs/howtos/generate-client-libraries/00howtogen/)
6+
7+
# Getting started
8+
9+
## Typespec setup
10+
11+
- Install the typespec [compiler](https://typespec.io/docs/) by following the "Install tsp" section. You should be able to run `tsp` in your terminal once this is done.
12+
13+
## Raiclient
14+
15+
TODO: Have someone who did more than blindly stumble their way to a solution double check this, especially the step about modifying the tspconfig.
16+
17+
[Source](https://github.com/Azure/azure-rest-api-specs/tree/gaugup/AddTypeSpecRAISvc/specification/ai/Azure.AI.Projects/RAISvc) Note, currently not on main. Use main once committed.
18+
19+
- Navigate to `azure-rest-api-specs/specification/ai/Azure.AI.Projects/RAISvc` and make sure you're in the right repo branch.
20+
- The contents of `./tspconfig.yaml` are set to emit autorest files. Temporarily replace the file with the following:
21+
```yaml
22+
emit:
23+
- "@azure-tools/typespec-python"
24+
options:
25+
"@azure-tools/typespec-python":
26+
package-name: "rai_client"
27+
output-dir: "./rai_client"
28+
```
29+
- Run from run ` tsp compile main.tsp`
30+
- Check the output. Autogenerated files should be in `tsp-output/@azure-tools/typespec-python/raiclient/`
31+
- Replace the contents of `azure-sdk-for-python/sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/autogen/raiclient` with the newly generated code.
32+
- Cleanup lingering files and revert the config changed.
33+
34+
# Key concepts
35+
# Examples
36+
# Troubleshooting
37+
# Next steps
38+
# Contributing

‎sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/autogen/__init__.py‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
# ---------------------------------------------------------
2+
# Copyright (c) Microsoft Corporation. All rights reserved.
3+
# ---------------------------------------------------------
4+
5+
from .raiclient import MachineLearningServicesClient
6+
7+
__all__ = ["MachineLearningServicesClient"]

‎sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/autogen/raiclient/__init__.py‎

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# coding=utf-8
2+
# --------------------------------------------------------------------------
3+
# Copyright (c) Microsoft Corporation. All rights reserved.
4+
# Licensed under the MIT License. See License.txt in the project root for license information.
5+
# Code generated by Microsoft (R) Python Code Generator.
6+
# Changes may cause incorrect behavior and will be lost if the code is regenerated.
7+
# --------------------------------------------------------------------------
8+
# pylint: disable=wrong-import-position
9+
10+
from typing import TYPE_CHECKING
11+
12+
if TYPE_CHECKING:
13+
from ._patch import * # pylint: disable=unused-wildcard-import
14+
15+
from ._client import MachineLearningServicesClient # type: ignore
16+
from ._version import VERSION
17+
18+
__version__ = VERSION
19+
20+
try:
21+
from ._patch import __all__ as _patch_all
22+
from ._patch import *
23+
except ImportError:
24+
_patch_all = []
25+
from ._patch import patch_sdk as _patch_sdk
26+
27+
# Export GeneratedRAIClient as alias of MachineLearningServicesClient for backward compatibility
28+
29+
__all__ = [
30+
"MachineLearningServicesClient",
31+
]
32+
__all__.extend([p for p in _patch_all if p not in __all__]) # pyright: ignore
33+
34+
_patch_sdk()

‎sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/autogen/raiclient/_client.py‎

Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
# pylint: disable=line-too-long,useless-suppression
2+
# coding=utf-8
3+
# --------------------------------------------------------------------------
4+
# Copyright (c) Microsoft Corporation. All rights reserved.
5+
# Licensed under the MIT License. See License.txt in the project root for license information.
6+
# Code generated by Microsoft (R) Python Code Generator.
7+
# Changes may cause incorrect behavior and will be lost if the code is regenerated.
8+
# --------------------------------------------------------------------------
9+
10+
from copy import deepcopy
11+
from typing import Any, TYPE_CHECKING
12+
from typing_extensions import Self
13+
14+
from azure.core import PipelineClient
15+
from azure.core.pipeline import policies
16+
from azure.core.rest import HttpRequest, HttpResponse
17+
18+
from ._configuration import MachineLearningServicesClientConfiguration
19+
from ._serialization import Deserializer, Serializer
20+
from .operations import RAISvcOperations
21+
22+
if TYPE_CHECKING:
23+
from azure.core.credentials import TokenCredential
24+
25+
26+
class MachineLearningServicesClient:
27+
"""MachineLearningServicesClient.
28+
29+
:ivar rai_svc: RAISvcOperations operations
30+
:vartype rai_svc: raiclient.operations.RAISvcOperations
31+
:param endpoint: Supported Azure-AI endpoints. Required.
32+
:type endpoint: str
33+
:param subscription_id: The ID of the target subscription. Required.
34+
:type subscription_id: str
35+
:param resource_group_name: The name of the Resource Group. Required.
36+
:type resource_group_name: str
37+
:param workspace_name: The name of the AzureML workspace or AI project. Required.
38+
:type workspace_name: str
39+
:param credential: Credential used to authenticate requests to the service. Required.
40+
:type credential: ~azure.core.credentials.TokenCredential
41+
:keyword api_version: The API version to use for this operation. Default value is
42+
"2022-11-01-preview". Note that overriding this default value may result in unsupported
43+
behavior.
44+
:paramtype api_version: str
45+
"""
46+
47+
def __init__(
48+
self,
49+
endpoint: str,
50+
subscription_id: str,
51+
resource_group_name: str,
52+
workspace_name: str,
53+
credential: "TokenCredential",
54+
**kwargs: Any
55+
) -> None:
56+
_endpoint = "{endpoint}/raisvc/v1.0/subscriptions/{subscriptionId}/resourceGroups/{resourceGroupName}/providers/Microsoft.MachineLearningServices/workspaces/{workspaceName}"
57+
self._config = MachineLearningServicesClientConfiguration(
58+
endpoint=endpoint,
59+
subscription_id=subscription_id,
60+
resource_group_name=resource_group_name,
61+
workspace_name=workspace_name,
62+
credential=credential,
63+
**kwargs
64+
)
65+
_policies = kwargs.pop("policies", None)
66+
if _policies is None:
67+
_policies = [
68+
policies.RequestIdPolicy(**kwargs),
69+
self._config.headers_policy,
70+
self._config.user_agent_policy,
71+
self._config.proxy_policy,
72+
policies.ContentDecodePolicy(**kwargs),
73+
self._config.redirect_policy,
74+
self._config.retry_policy,
75+
self._config.authentication_policy,
76+
self._config.custom_hook_policy,
77+
self._config.logging_policy,
78+
policies.DistributedTracingPolicy(**kwargs),
79+
policies.SensitiveHeaderCleanupPolicy(**kwargs) if self._config.redirect_policy else None,
80+
self._config.http_logging_policy,
81+
]
82+
self._client: PipelineClient = PipelineClient(base_url=_endpoint, policies=_policies, **kwargs)
83+
84+
self._serialize = Serializer()
85+
self._deserialize = Deserializer()
86+
self._serialize.client_side_validation = False
87+
self.rai_svc = RAISvcOperations(self._client, self._config, self._serialize, self._deserialize)
88+
89+
def send_request(self, request: HttpRequest, *, stream: bool = False, **kwargs: Any) -> HttpResponse:
90+
"""Runs the network request through the client's chained policies.
91+
92+
>>> from azure.core.rest import HttpRequest
93+
>>> request = HttpRequest("GET", "https://www.example.org/")
94+
<HttpRequest [GET], url: 'https://www.example.org/'>
95+
>>> response = client.send_request(request)
96+
<HttpResponse: 200 OK>
97+
98+
For more information on this code flow, see https://aka.ms/azsdk/dpcodegen/python/send_request
99+
100+
:param request: The network request you want to make. Required.
101+
:type request: ~azure.core.rest.HttpRequest
102+
:keyword bool stream: Whether the response payload will be streamed. Defaults to False.
103+
:return: The response of your network call. Does not do error handling on your response.
104+
:rtype: ~azure.core.rest.HttpResponse
105+
"""
106+
107+
request_copy = deepcopy(request)
108+
path_format_arguments = {
109+
"endpoint": self._serialize.url("self._config.endpoint", self._config.endpoint, "str", skip_quote=True),
110+
"subscriptionId": self._serialize.url("self._config.subscription_id", self._config.subscription_id, "str"),
111+
"resourceGroupName": self._serialize.url(
112+
"self._config.resource_group_name", self._config.resource_group_name, "str"
113+
),
114+
"workspaceName": self._serialize.url("self._config.workspace_name", self._config.workspace_name, "str"),
115+
}
116+
117+
request_copy.url = self._client.format_url(request_copy.url, **path_format_arguments)
118+
return self._client.send_request(request_copy, stream=stream, **kwargs) # type: ignore
119+
120+
def close(self) -> None:
121+
self._client.close()
122+
123+
def __enter__(self) -> Self:
124+
self._client.__enter__()
125+
return self
126+
127+
def __exit__(self, *exc_details: Any) -> None:
128+
self._client.__exit__(*exc_details)

‎sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/autogen/raiclient/_configuration.py‎

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
# coding=utf-8
2+
# --------------------------------------------------------------------------
3+
# Copyright (c) Microsoft Corporation. All rights reserved.
4+
# Licensed under the MIT License. See License.txt in the project root for license information.
5+
# Code generated by Microsoft (R) Python Code Generator.
6+
# Changes may cause incorrect behavior and will be lost if the code is regenerated.
7+
# --------------------------------------------------------------------------
8+
9+
from typing import Any, TYPE_CHECKING
10+
11+
from azure.core.pipeline import policies
12+
13+
from ._version import VERSION
14+
15+
if TYPE_CHECKING:
16+
from azure.core.credentials import TokenCredential
17+
18+
19+
class MachineLearningServicesClientConfiguration: # pylint: disable=too-many-instance-attributes,name-too-long
20+
"""Configuration for MachineLearningServicesClient.
21+
22+
Note that all parameters used to create this instance are saved as instance
23+
attributes.
24+
25+
:param endpoint: Supported Azure-AI endpoints. Required.
26+
:type endpoint: str
27+
:param subscription_id: The ID of the target subscription. Required.
28+
:type subscription_id: str
29+
:param resource_group_name: The name of the Resource Group. Required.
30+
:type resource_group_name: str
31+
:param workspace_name: The name of the AzureML workspace or AI project. Required.
32+
:type workspace_name: str
33+
:param credential: Credential used to authenticate requests to the service. Required.
34+
:type credential: ~azure.core.credentials.TokenCredential
35+
:keyword api_version: The API version to use for this operation. Default value is
36+
"2022-11-01-preview". Note that overriding this default value may result in unsupported
37+
behavior.
38+
:paramtype api_version: str
39+
"""
40+
41+
def __init__(
42+
self,
43+
endpoint: str,
44+
subscription_id: str,
45+
resource_group_name: str,
46+
workspace_name: str,
47+
credential: "TokenCredential",
48+
**kwargs: Any
49+
) -> None:
50+
api_version: str = kwargs.pop("api_version", "2022-11-01-preview")
51+
52+
if endpoint is None:
53+
raise ValueError("Parameter 'endpoint' must not be None.")
54+
if subscription_id is None:
55+
raise ValueError("Parameter 'subscription_id' must not be None.")
56+
if resource_group_name is None:
57+
raise ValueError("Parameter 'resource_group_name' must not be None.")
58+
if workspace_name is None:
59+
raise ValueError("Parameter 'workspace_name' must not be None.")
60+
if credential is None:
61+
raise ValueError("Parameter 'credential' must not be None.")
62+
63+
self.endpoint = endpoint
64+
self.subscription_id = subscription_id
65+
self.resource_group_name = resource_group_name
66+
self.workspace_name = workspace_name
67+
self.credential = credential
68+
self.api_version = api_version
69+
self.credential_scopes = kwargs.pop("credential_scopes", ["https://ml.azure.com/.default"])
70+
kwargs.setdefault("sdk_moniker", "rai_client/{}".format(VERSION))
71+
self.polling_interval = kwargs.get("polling_interval", 30)
72+
self._configure(**kwargs)
73+
74+
def _configure(self, **kwargs: Any) -> None:
75+
self.user_agent_policy = kwargs.get("user_agent_policy") or policies.UserAgentPolicy(**kwargs)
76+
self.headers_policy = kwargs.get("headers_policy") or policies.HeadersPolicy(**kwargs)
77+
self.proxy_policy = kwargs.get("proxy_policy") or policies.ProxyPolicy(**kwargs)
78+
self.logging_policy = kwargs.get("logging_policy") or policies.NetworkTraceLoggingPolicy(**kwargs)
79+
self.http_logging_policy = kwargs.get("http_logging_policy") or policies.HttpLoggingPolicy(**kwargs)
80+
self.custom_hook_policy = kwargs.get("custom_hook_policy") or policies.CustomHookPolicy(**kwargs)
81+
self.redirect_policy = kwargs.get("redirect_policy") or policies.RedirectPolicy(**kwargs)
82+
self.retry_policy = kwargs.get("retry_policy") or policies.RetryPolicy(**kwargs)
83+
self.authentication_policy = kwargs.get("authentication_policy")
84+
if self.credential and not self.authentication_policy:
85+
self.authentication_policy = policies.BearerTokenCredentialPolicy(
86+
self.credential, *self.credential_scopes, **kwargs
87+
)

0 commit comments

Comments
 (0)