Data only red team mode (Azure#40531)

* Update task_query_response.prompty

remove required keys

* Update task_simulate.prompty

* Update task_query_response.prompty

* Update task_simulate.prompty

* Fix the api_key needed

* Update for release

* Black fix for file

* Add original text in global context

* Update test

* Update the indirect attack simulator

* Black suggested fixes

* Update simulator prompty

* Update adversarial scenario enum to exclude XPIA

* Update changelog

* Black fixes

* Remove duplicate import

* Fix the mypy error

* Mypy please be happy

* Updates to non adv simulator

* accept context from assistant messages, exclude them when using them for conversation

* update changelog

* pylint fixes

* pylint fixes

* remove redundant quotes

* Fix typo

* pylint fix

* Update broken tests

* Include the grounding json in the manifest

* Fix typo

* Come on package

* Release 1.0.0b5

* Notice from Chang

* Remove adv_conv template parameters from the outputs

* Update chanagelog

* Experimental tags on adv scenarios

* Readme fix onbreaking change

* Add the category and both user and assistant context to the response of qr_json_lines

* Update changelog

* Rename _kwargs to _options

* _options as prefix

* update troubleshooting for simulator

* Rename according to suggestions

* Clean up readme

* more links

* Bugfix: zip_longest created null parameters

* Updated changelog

* zip does the job

* remove ununsed import

* Fix changelog merge

* Remove print statements

* call evaluate_with_rai_service instead of evaluate

* clean up

* Data only works e2e, todo handle skip_evals

* Data only mode

* bug fixes

* aggregated binary threshold results for evaluators in metrics

* handle when the right columns do not exist

* Marking mypy, pylint, black as false

* add retry logic to service calls and send_prompt_async

* small fixes

* Add to constatns

* typo

* update retry to handle asyncio timeout

* fix attack success

* _skip_evals and docstrings

* close file handlers

* fix red_team

* update changelog

* end run

* Update sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/red_team/_red_team.py

Co-authored-by: Ankit Singhal <[email protected]>

* skip evals as a param

* Using the same artifact name

---------

Co-authored-by: Nagkumar Arkalgud <[email protected]>
Co-authored-by: Nagkumar Arkalgud <[email protected]>
Co-authored-by: Nagkumar Arkalgud <[email protected]>
Co-authored-by: Sydney Lister <[email protected]>
Co-authored-by: Ankit Singhal <[email protected]>

Author: 6 people

sdk/evaluation/azure-ai-evaluation/CHANGELOG.md

@@ -11,11 +11,12 @@
     - AzureOpenAITextSimilarityGrader
 
 ### Breaking Changes
+- In the experimental RedTeam's scan method, the `data_only` param has been replaced with `skip_evals` and if you do not want data to be uploaded, use the `skip_upload` flag.
 
 ### Bugs Fixed
 - Fixed error in `evaluate` where data fields could not contain numeric characters. Previously, a data file with schema:
     ```
-    "query1": "some query", "response: "some response"
+    "query1": "some query", "response": "some response"
     ```
     throws error when passed into `evaluator_config` as `{"evaluator_name": {"column_mapping": {"query": "${data.query1}", "response": "${data.response}"}},}`.
     Now, users may import data containing fields with numeric characters.