Users/anksing/chat built in flow fix (#34603)

* Adding tests to capture groundedness with expected values

* Fixing built in task type chat

* Reverting logger change

* Fixing built in task type chat

* Reverting logger change

Author: singankit

sdk/ai/azure-ai-generative/azure/ai/generative/evaluate/pf_templates/built_in_metrics/chat/flow.dag.yaml

@@ -132,7 +132,7 @@ nodes:
     type: code
     path: evaluate_chat_rag.py
   inputs:
-    connection: Default_AzureOpenAI
+    connection: openai_connection
     chat: ${inputs.messages}
     deployment_name: ${inputs.deployment_name}
     selected_metrics: ${select_metrics.output}
@@ -146,7 +146,7 @@ nodes:
     type: code
     path: evaluate_coherence_fluency.py
   inputs:
-    connection: Default_AzureOpenAI
+    connection: openai_connection
     deployment_name: ${inputs.deployment_name}
     parsed_qa: ${parse_chat.output}
     selected_metrics: ${select_metrics.output}

sdk/ai/azure-ai-generative/tests/evaluate/data/rag_conversation_data.jsonl

@@ -0,0 +1,2 @@
+{"messages":[{"content":"will my compass work in patagonia","role":"user","context":{"customer_info":"## customer_info\n  \nname: Jane Doe   \nage: 28    \nphone_number: 555-987-6543     item_number: 7 \n \n# chat history: \n \n# product context: \n\n"}},{"content":"Yes, the Pathfinder Pro-1 Adventure Compass can be used in Patagonia. It is designed for use in both the northern and southern hemispheres and has a built-in adjustable declination correction for precise navigation.","role":"assistant","session_state":{"id":"1234-5678-9012-3456"},"context":{"citations":[{"id":"data\/3-product-info\/product_info_66.md","content":"# Information about product item_number: 66\nPathfinder Pro-1 Adventure Compass, priced: $39.99.\nExperience accurate navigation and superior outdoor performance\n\nFeatures:\n\nRobust, lightweight ABS plastic construction for durability and easy carry\nPrecision\nTechnical Specs:\n\nBest Use: Hiking, Orienteering, Backpacking"},{"id":"data\/3-product-info\/product_info_17.md","content":"# Information about product item_number: 17\nRainGuard Hiking Jacket, price $110,\n\n## Brand\nMountainStyle\n\n## Category\nHiking Clothing\n## Features\nHere is a feature list for the RainGuard Hiking Jacket:\n## Features\n- Waterproof and Breathable: The RainGuard Hiking Jacket is designed to keep you dry and comfortable in wet conditions, thanks to its waterproof and breathable fabric.\n\n## Technical Specs\n- **Best Use:** Hiking"},{"id":"customer_info_and_recent_purchases","content":"# current date_time: 2023-10-13 15:00:00\n\n## Customer Details:\n\nName: Jane Doe\nAge: 28\nPhone Number: 555-987-6543\n\n## Recent Purchases:\n\nOrder Number: 5\nDate: 2023-05-01\nItem:\n- Description: TrailMaster X4 Tent\n  Quantity: 1\n  Price: $250\n  Item Number: 1"}]}},{"content":"and what is the temperature rating of my sleeping bag?","role":"user","session_state":{"id":"1234-5678-9012-3456"},"context":{"customer_info":"## customer_info\n  \nname: Jane Doe   \nage: 28 "}},{"content":"The CozyNights Sleeping Bag has a temperature rating of 20\u00b0F to 60\u00b0F (-6\u00b0C to 15\u00b0C). It is suitable for use in spring, summer, and fall seasons","role":"assistant","context":{"citations":[{"id":"data\/3-product-info\/product_info_7.md","content":"# Information about product item_number: 7\nCozyNights Sleeping Bag, price $100,\n\n## Brand\nCozyNights\n\n## Category\nSleeping Bags\n\n## Features\n- **Lightweight**: Designed to be lightweight for easy carrying during outdoor adventures.\n- **3-Season**: Suitable for use in spring, summer, and fall seasons.\n\n## Technical Specifications\n- **Material**: Polyester\n- **Color**: Red\n\n## FAQ\n31) What is the temperature rating of the CozyNights Sleeping Bag?\nThe CozyNights Sleeping Bag is rated for 3-season use and has a temperature rating of 20 F to 60 F (-6 C to 15 C)."},{"id":"data\/3-product-info\/product_info_1.md","content":"# Information about product item_number: 1\nTrailMaster X4 Tent, price $250,\n\n## Brand\nOutdoorLiving\n\n## Category\nTents\n\n## Features\n- Polyester material for durability\n- Spacious interior to accommodate multiple people\n\n## Technical Specs\n**Best Use**: Camping\n**Capacity**: 4-person"},{"id":"customer_info_and_recent_purchases","content":"# current date_time: 2023-10-13 15:01:03\n\n## Customer Details:\n\nName: Jane Doe\nAge: 28\nPhone Number: 555-987-6543\nEmail: [email protected]\nAddress: 789 Broadway St, Seattle, WA 98101\n\nLoyalty Program: True\nLoyalty Program Level: Bronze\n\n## Recent Purchases:\n\nOrder Number: 5\nDate: 2023-05-01\nItem:\n- Description: TrailMaster X4 Tent\n  Quantity: 1\n  Price: $250\n  Item Number: 1"}]}},{"content":"Awesome, thanks!","role":"user","session_state":{"id":"1234-5678-9012-3456"},"context":{"customer_info":"## customer_info\n  \nname: Jane Doe   \nage: 28    \nphone_number: 555-987-6543 "}},{"content":"You are welcome, Jane!","role":"assistant","context":{"citations":[{"id":"customer_info_and_recent_purchases","content":"# current date_time: 2023-10-13 15:01:43\n\n## Customer Details:\n\nName: Jane Doe\nAge: 28\nPhone Number: 555-987-6543\nEmail: [email protected]\nAddress: 789 Broadway St, Seattle, WA 98101\n\nLoyalty Program: True\nLoyalty Program Level: Bronze\n\n## Recent Purchases:\n\nOrder Number: 5\nDate: 2023-05-01\nItem:\n- Description: TrailMaster X4 Tent\n  Quantity: 1\n  Price: $250\n  Item Number: 1"}]}}]}
+{"messages":[{"content":"will my compass work in patagonia","role":"user","context":{"customer_info":"## customer_info\n  \nname: Jane Doe   \nage: 28    \nphone_number: 555-987-6543     item_number: 7 \n \n# chat history: \n \n# product context: \n\n"}},{"content":"Yes, the Pathfinder Pro-1 Adventure Compass can be used in Patagonia. It is designed for use in both the northern and southern hemispheres and has a built-in adjustable declination correction for precise navigation.","role":"assistant","session_state":{"id":"1234-5678-9012-3456"},"context":{"citations":[{"id":"data\/3-product-info\/product_info_66.md","content":"# Information about product item_number: 66\nPathfinder Pro-1 Adventure Compass, priced: $39.99.\nExperience accurate navigation and superior outdoor performance\n\nFeatures:\n\nRobust, lightweight ABS plastic construction for durability and easy carry\nPrecision\nTechnical Specs:\n\nBest Use: Hiking, Orienteering, Backpacking"},{"id":"data\/3-product-info\/product_info_17.md","content":"# Information about product item_number: 17\nRainGuard Hiking Jacket, price $110,\n\n## Brand\nMountainStyle\n\n## Category\nHiking Clothing\n## Features\nHere is a feature list for the RainGuard Hiking Jacket:\n## Features\n- Waterproof and Breathable: The RainGuard Hiking Jacket is designed to keep you dry and comfortable in wet conditions, thanks to its waterproof and breathable fabric.\n\n## Technical Specs\n- **Best Use:** Hiking"},{"id":"customer_info_and_recent_purchases","content":"# current date_time: 2023-10-13 15:00:00\n\n## Customer Details:\n\nName: Jane Doe\nAge: 28\nPhone Number: 555-987-6543\n\n## Recent Purchases:\n\nOrder Number: 5\nDate: 2023-05-01\nItem:\n- Description: TrailMaster X4 Tent\n  Quantity: 1\n  Price: $250\n  Item Number: 1"}]}},{"content":"and what is the temperature rating of my sleeping bag?","role":"user","session_state":{"id":"1234-5678-9012-3456"},"context":{"customer_info":"## customer_info\n  \nname: Jane Doe   \nage: 28 "}},{"content":"The CozyNights Sleeping Bag has a temperature rating of 20\u00b0F to 60\u00b0F (-6\u00b0C to 15\u00b0C). It is suitable for use in spring, summer, and fall seasons","role":"assistant","context":{"citations":[{"id":"data\/3-product-info\/product_info_7.md","content":"# Information about product item_number: 7\nCozyNights Sleeping Bag, price $100,\n\n## Brand\nCozyNights\n\n## Category\nSleeping Bags\n\n## Features\n- **Lightweight**: Designed to be lightweight for easy carrying during outdoor adventures.\n- **3-Season**: Suitable for use in spring, summer, and fall seasons.\n\n## Technical Specifications\n- **Material**: Polyester\n- **Color**: Red\n\n## FAQ\n31) What is the temperature rating of the CozyNights Sleeping Bag?\nThe CozyNights Sleeping Bag is rated for 3-season use and has a temperature rating of 20 F to 60 F (-6 C to 15 C)."},{"id":"data\/3-product-info\/product_info_1.md","content":"# Information about product item_number: 1\nTrailMaster X4 Tent, price $250,\n\n## Brand\nOutdoorLiving\n\n## Category\nTents\n\n## Features\n- Polyester material for durability\n- Spacious interior to accommodate multiple people\n\n## Technical Specs\n**Best Use**: Camping\n**Capacity**: 4-person"},{"id":"customer_info_and_recent_purchases","content":"# current date_time: 2023-10-13 15:01:03\n\n## Customer Details:\n\nName: Jane Doe\nAge: 28\nPhone Number: 555-987-6543\nEmail: [email protected]\nAddress: 789 Broadway St, Seattle, WA 98101\n\nLoyalty Program: True\nLoyalty Program Level: Bronze\n\n## Recent Purchases:\n\nOrder Number: 5\nDate: 2023-05-01\nItem:\n- Description: TrailMaster X4 Tent\n  Quantity: 1\n  Price: $250\n  Item Number: 1"}]}},{"content":"Awesome, thanks!","role":"user","session_state":{"id":"1234-5678-9012-3456"},"context":{"customer_info":"## customer_info\n  \nname: Jane Doe   \nage: 28    \nphone_number: 555-987-6543 "}},{"content":"You are welcome, Jane!","role":"assistant","context":{"citations":[{"id":"customer_info_and_recent_purchases","content":"# current date_time: 2023-10-13 15:01:43\n\n## Customer Details:\n\nName: Jane Doe\nAge: 28\nPhone Number: 555-987-6543\nEmail: [email protected]\nAddress: 789 Broadway St, Seattle, WA 98101\n\nLoyalty Program: True\nLoyalty Program Level: Bronze\n\n## Recent Purchases:\n\nOrder Number: 5\nDate: 2023-05-01\nItem:\n- Description: TrailMaster X4 Tent\n  Quantity: 1\n  Price: $250\n  Item Number: 1"}]}}]}

sdk/ai/azure-ai-generative/tests/evaluate/e2etests/test_evaluate_e2e.py

@@ -163,14 +163,15 @@ async def answer_length(*, data, **kwargs):
         assert "answer_length_random" in columns_in_tabular_data
 
     def test_task_type_chat(self, e2e_openai_api_base, e2e_openai_api_key, e2e_openai_completion_deployment_name, tmpdir):
+        data_path = os.path.join(pathlib.Path(__file__).parent.parent.resolve(), "data")
+        data_file = os.path.join(data_path, "rag_conversation_data.jsonl")
 
-        data_path = os.getcwd() + "/rag_conversation_data.jsonl"
         with tmpdir.as_cwd():
             output_path = tmpdir + "/evaluation_output"
 
             result = evaluate(  # This will log metric/artifacts using mlflow
                 evaluation_name="rag-chat-1",
-                data=data_path,
+                data=data_file,
                 task_type="chat",
                 model_config={
                     "api_version": "2023-07-01-preview",