Switch to gpt-41-mini as default chat model (#2557)

* Switch to gpt-41-mini with evaluations

* Update model used in tests

* Change search agent to 4.1-mini as well

* Update model version

Author: pamelafox

README.md

@@ -51,7 +51,7 @@ This template, the application code and configuration it contains, has been buil
 
 [📺 Watch a video overview of the app.](https://youtu.be/3acB0OWmLvM)
 
-This sample demonstrates a few approaches for creating ChatGPT-like experiences over your own data using the Retrieval Augmented Generation pattern. It uses Azure OpenAI Service to access a GPT model (gpt-4o-mini), and Azure AI Search for data indexing and retrieval.
+This sample demonstrates a few approaches for creating ChatGPT-like experiences over your own data using the Retrieval Augmented Generation pattern. It uses Azure OpenAI Service to access a GPT model (gpt-4.1-mini), and Azure AI Search for data indexing and retrieval.
 
 The repo includes sample data so it's ready to try end to end. In this sample application we use a fictitious company called Contoso Electronics, and the experience allows its employees to ask questions about the benefits, internal policies, as well as job descriptions and roles.
 

docs/agentic_retrieval.md

@@ -10,7 +10,7 @@ See the agentic retrieval documentation.
 
 ### Prerequisites
 
-* A deployment of any of the supported agentic retrieval models in the [supported regions](https://learn.microsoft.com/azure/ai-services/openai/concepts/models#standard-deployment-model-availability). If you're not sure, try to create a gpt-4o-mini deployment from your Azure OpenAI deployments page.
+* A deployment of any of the supported agentic retrieval models in the [supported regions](https://learn.microsoft.com/azure/ai-services/openai/concepts/models#standard-deployment-model-availability). If you're not sure, try to create a gpt-4.1-mini deployment from your Azure OpenAI deployments page.
 
 ### Deployment
 
@@ -24,14 +24,14 @@ See the agentic retrieval documentation.
 
 2. **(Optional) Set the agentic retrieval model**
 
-   You can configure which model agentic retrieval uses. By default, gpt-4o-mini is used
+   You can configure which model agentic retrieval uses. By default, gpt-4.1-mini is used.
 
-   For gpt-4o:
+   To change the model, set the following environment variables appropriately:
 
    ```shell
    azd env set AZURE_OPENAI_SEARCHAGENT_DEPLOYMENT searchagent
-   azd env set AZURE_OPENAI_SEARCHAGENT_MODEL gpt-4o
-   azd env set AZURE_OPENAI_SEARCHAGENT_MODEL_VERSION 2024-11-20
+   azd env set AZURE_OPENAI_SEARCHAGENT_MODEL gpt-4.1-mini
+   azd env set AZURE_OPENAI_SEARCHAGENT_MODEL_VERSION 2025-04-14
    ```
 
 3. **Update the infrastructure and application:**

docs/deploy_existing.md

@@ -26,8 +26,8 @@ You should set these values before running `azd up`. Once you've set them, retur
 1. Run `azd env set AZURE_OPENAI_SERVICE {Name of existing OpenAI service}`
 1. Run `azd env set AZURE_OPENAI_RESOURCE_GROUP {Name of existing resource group that OpenAI service is provisioned to}`
 1. Run `azd env set AZURE_OPENAI_LOCATION {Location of existing OpenAI service}`
-1. Run `azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT {Name of existing chat deployment}`. Only needed if your chat deployment name is not the default 'gpt-4o-mini'.
-1. Run `azd env set AZURE_OPENAI_CHATGPT_MODEL {Model name of existing chat deployment}`. Only needed if your chat model is not the default 'gpt-4o-mini'.
+1. Run `azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT {Name of existing chat deployment}`. Only needed if your chat deployment name is not the default 'gpt-4.1-mini'.
+1. Run `azd env set AZURE_OPENAI_CHATGPT_MODEL {Model name of existing chat deployment}`. Only needed if your chat model is not the default 'gpt-4.1-mini'.
 1. Run `azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT_VERSION {Version string for existing chat deployment}`. Only needed if your chat deployment model version is not the default '2024-07-18'. You definitely need to change this if you changed the model.
 1. Run `azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT_SKU {Name of SKU for existing chat deployment}`. Only needed if your chat deployment SKU is not the default 'Standard', like if it is 'GlobalStandard' instead.
 1. Run `azd env set AZURE_OPENAI_EMB_DEPLOYMENT {Name of existing embedding deployment}`. Only needed if your embeddings deployment is not the default 'embedding'.

docs/deploy_features.md

@@ -24,7 +24,7 @@ You should typically enable these features before running `azd up`. Once you've
 
 ## Using different chat completion models
 
-As of late March 2025, the default chat completion model is `gpt-4o-mini`. If you deployed this sample before that date, the default model is `gpt-3.5-turbo`. You can change the chat completion model to any Azure OpenAI chat model that's available in your Azure OpenAI resource region by following these steps:
+As of early June 2025, the default chat completion model is `gpt-4.1-mini`. If you deployed this sample before that date, the default model is `gpt-3.5-turbo` or `gpt-4o-mini`. You can change the chat completion model to any Azure OpenAI chat model that's available in your Azure OpenAI resource region by following these steps:
 
 1. To set the name of the deployment, run this command with a unique name in your Azure OpenAI account. You can use any deployment name, as long as it's unique in your Azure OpenAI account. For convenience, many developers use the same deployment name as the model name, but this is not required.
 
@@ -40,24 +40,30 @@ As of late March 2025, the default chat completion model is `gpt-4o-mini`. If yo
 
 1. To set the GPT model to a different [available model](https://learn.microsoft.com/azure/ai-services/openai/concepts/models), run this command with the appropriate model name.
 
-    For GPT-4:
+    For gpt-4.1-mini:
 
     ```bash
-    azd env set AZURE_OPENAI_CHATGPT_MODEL gpt-4
+    azd env set AZURE_OPENAI_CHATGPT_MODEL gpt-4.1-mini
     ```
 
-    For GPT-4o:
+    For gpt-4o:
 
     ```bash
     azd env set AZURE_OPENAI_CHATGPT_MODEL gpt-4o
     ```
 
-    For GPT-4o mini:
+    For gpt-4o mini:
 
     ```bash
     azd env set AZURE_OPENAI_CHATGPT_MODEL gpt-4o-mini
     ```
 
+    For gpt-4:
+
+    ```bash
+    azd env set AZURE_OPENAI_CHATGPT_MODEL gpt-4
+    ```
+
     For gpt-3.5-turbo:
 
     ```bash
@@ -66,24 +72,30 @@ As of late March 2025, the default chat completion model is `gpt-4o-mini`. If yo
 
 1. To set the Azure OpenAI model version from the [available versions](https://learn.microsoft.com/azure/ai-services/openai/concepts/models), run this command with the appropriate version string.
 
-    For GPT-4:
+    For gpt-4.1-mini:
 
     ```bash
-    azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT_VERSION turbo-2024-04-09
+    azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT_VERSION 2025-04-14
     ```
 
-    For GPT-4o:
+    For gpt-4o:
 
     ```bash
     azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT_VERSION 2024-05-13
     ```
 
-    For GPT-4o mini:
+    For gpt-4o mini:
 
     ```bash
     azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT_VERSION 2024-07-18
     ```
 
+    For gpt-4:
+
+    ```bash
+    azd env set AZURE_OPENAI_CHATGPT_DEPLOYMENT_VERSION turbo-2024-04-09
+    ```
+
     For gpt-3.5-turbo:
 
     ```bash

docs/gpt4v.md

@@ -23,7 +23,7 @@ For more details on how this feature works, read [this blog post](https://techco
 * The ability to deploy a gpt-4o model in the [supported regions](https://learn.microsoft.com/azure/ai-services/openai/concepts/models#standard-deployment-model-availability). If you're not sure, try to create a gpt-4o deployment from your Azure OpenAI deployments page.
 * Ensure that you can deploy the Azure OpenAI resource group in [a region and deployment SKU where all required components are available](https://learn.microsoft.com/azure/cognitive-services/openai/concepts/models#model-summary-table-and-region-availability):
   * Azure OpenAI models
-    * gpt-4o-mini
+    * gpt-4.1-mini
     * text-embedding-3-large
     * gpt-4o (for vision/evaluation features)
   * [Azure AI Vision](https://learn.microsoft.com/azure/ai-services/computer-vision/)

evals/results/baseline/README.md

@@ -1,6 +1,6 @@
 {
     "evaluation_gpt_model": "gpt-4o",
-    "evaluation_timestamp": 1746818372,
+    "evaluation_timestamp": 1748932460,
     "testdata_path": "/Users/pamelafox/azure-search-openai-demo/evals/ground_truth.jsonl",
     "target_url": "http://localhost:50505/chat",
     "target_parameters": {

evals/results/baseline/eval_results.jsonl

@@ -1,31 +1,31 @@
 {
     "gpt_groundedness": {
-        "pass_count": 43,
-        "pass_rate": 0.86,
-        "mean_rating": 4.5
+        "pass_count": 47,
+        "pass_rate": 0.94,
+        "mean_rating": 4.76
     },
     "gpt_relevance": {
-        "pass_count": 42,
-        "pass_rate": 0.84,
-        "mean_rating": 4.22
+        "pass_count": 47,
+        "pass_rate": 0.94,
+        "mean_rating": 4.42
     },
     "answer_length": {
-        "mean": 919.26,
-        "max": 1647,
-        "min": 193
+        "mean": 829.06,
+        "max": 1380,
+        "min": 275
     },
     "latency": {
-        "mean": 4.46,
-        "max": 15.129978,
-        "min": 2.465542
+        "mean": 2.89,
+        "max": 5.183753,
+        "min": -1.0
     },
     "citations_matched": {
-        "total": 24,
-        "rate": 0.49
+        "total": 26,
+        "rate": 0.52
     },
     "any_citation": {
-        "total": 50,
-        "rate": 1.0
+        "total": 49,
+        "rate": 0.98
     },
     "num_questions": {
         "total": 50

evals/results/baseline/evaluate_parameters.json

@@ -0,0 +1,28 @@
+{
+    "testdata_path": "ground_truth.jsonl",
+    "results_dir": "results/experiment<TIMESTAMP>",
+    "requested_metrics": ["gpt_groundedness", "gpt_relevance", "answer_length", "latency", "citations_matched", "any_citation"],
+    "target_url": "http://localhost:50505/chat",
+    "target_parameters": {
+        "overrides": {
+            "top": 3,
+            "temperature": 0.3,
+            "minimum_reranker_score": 0,
+            "minimum_search_score": 0,
+            "retrieval_mode": "hybrid",
+            "semantic_ranker": true,
+            "semantic_captions": false,
+            "suggest_followup_questions": false,
+            "use_oid_security_filter": false,
+            "use_groups_security_filter": false,
+            "vector_fields": [
+                "embedding"
+            ],
+            "use_gpt4v": false,
+            "gpt4v_input": "textAndImages",
+            "seed": 1
+        }
+    },
+    "target_response_answer_jmespath": "message.content",
+    "target_response_context_jmespath": "context.data_points.text"
+}