Merge pull request #271345 from mrbullwinkle/mrb_04_05_2024_chatml

[Azure OpenAI] ChatML

Author: v-dirichards

articles/ai-services/openai/includes/chat-markup-language.md renamed to articles/ai-services/openai/how-to/chat-markup-language.md

@@ -6,17 +6,20 @@ author: mrbullwinkle #dereklegenzoff
 ms.author: mbullwin #delegenz
 ms.service: azure-ai-openai
 ms.topic: include
-ms.date: 05/15/2023
+ms.date: 04/05/2024
 manager: nitinme
 keywords: ChatGPT
 ---
 
-## Working with the Chat models
+# Chat Markup Language ChatML (Preview)
 
 > [!IMPORTANT]
-> Using GPT-35-Turbo models with the completion endpoint as described in this article remains in preview and is only possible with gpt-35-turbo verision (0301). We strongly recommend using the GA Chat Completion API/endpoint. The Chat Completion API is the recommended method of interacting with the GPT-35-Turbo models. The Chat Completion API is also the only way to access the GPT-4 models.
+> Using GPT-3.5-Turbo models with the completion endpoint as described in this article remains in preview and is only possible with `gpt-35-turbo` version (0301) which is [slated for retirement as early as June 13th, 2024](../concepts/model-retirements.md#current-models). We strongly recommend using the [GA Chat Completion API/endpoint](./chatgpt.md). The Chat Completion API is the recommended method of interacting with the GPT-3.5-Turbo models. The Chat Completion API is also the only way to access the GPT-4 models.
 
-The following code snippet shows the most basic way to use the GPT-35-Turbo models with ChatML. If this is your first time using these models programmatically we recommend starting with our [GPT-35-Turbo & GPT-4 Quickstart](../chatgpt-quickstart.md).
+The following code snippet shows the most basic way to use the GPT-3.5-Turbo models with ChatML. If this is your first time using these models programmatically we recommend starting with our [GPT-35-Turbo & GPT-4 Quickstart](../chatgpt-quickstart.md).
+
+> [!NOTE]  
+> In the Azure OpenAI documentation we refer to GPT-3.5-Turbo, and GPT-35-Turbo interchangeably. The official name of the model on OpenAI is `gpt-3.5-turbo`, but for Azure OpenAI due to Azure specific character constraints the underlying model name is `gpt-35-turbo`.
 
 ```python
 import os
@@ -40,7 +43,7 @@ print(response['choices'][0]['text'])
 > [!NOTE]  
 > The following parameters aren't available with the gpt-35-turbo model: `logprobs`, `best_of`, and `echo`. If you set any of these parameters, you'll get an error.
 
-The `<|im_end|>` token indicates the end of a message. We recommend including `<|im_end|>` token as a stop sequence to ensure that the model stops generating text when it reaches the end of the message. 
+The `<|im_end|>` token indicates the end of a message. When using ChatML it is recommended to include `<|im_end|>` token as a stop sequence to ensure that the model stops generating text when it reaches the end of the message. 
 
 Consider setting `max_tokens` to a slightly higher value than normal such as 300 or 500. This ensures that the model doesn't stop generating text before it reaches the end of the message.
 
@@ -53,8 +56,6 @@ Unlike previous GPT-3 and GPT-3.5 models, the `gpt-35-turbo` model as well as th
 
 You can find the model retirement dates for these models on our [models](../concepts/models.md) page.
 
-<a id="chatml"></a>
-
 ## Working with Chat Markup Language (ChatML)
 
 > [!NOTE]  

articles/ai-services/openai/how-to/chatgpt.md

@@ -7,35 +7,17 @@ ms.author: mbullwin #delegenz
 ms.service: azure-ai-openai
 ms.custom: build-2023, build-2023-dataai
 ms.topic: how-to
-ms.date: 03/29/2024
+ms.date: 04/05/2024
 manager: nitinme
 keywords: ChatGPT
 zone_pivot_groups: openai-chat
 ---
 
-# Learn how to work with the GPT-35-Turbo and GPT-4 models
+# Learn how to work with the GPT-3.5-Turbo and GPT-4 models
 
-The GPT-35-Turbo and GPT-4 models are language models that are optimized for conversational interfaces. The models behave differently than the older GPT-3 models. Previous models were text-in and text-out, meaning they accepted a prompt string and returned a completion to append to the prompt. However, the GPT-35-Turbo and GPT-4 models are conversation-in and message-out. The models expect input formatted in a specific chat-like transcript format, and return a completion that represents a model-written message in the chat. While this format was designed specifically for multi-turn conversations, you'll find it can also work well for non-chat scenarios too.
+The GPT-3.5-Turbo and GPT-4 models are language models that are optimized for conversational interfaces. The models behave differently than the older GPT-3 models. Previous models were text-in and text-out, meaning they accepted a prompt string and returned a completion to append to the prompt. However, the GPT-3.5-Turbo and GPT-4 models are conversation-in and message-out. The models expect input formatted in a specific chat-like transcript format, and return a completion that represents a model-written message in the chat. While this format was designed specifically for multi-turn conversations, you'll find it can also work well for non-chat scenarios too.
 
-In Azure OpenAI there are two different options for interacting with these type of models:
+This article walks you through getting started with the GPT-3.5-Turbo and GPT-4 models. It's important to use the techniques described here to get the best results. If you try to interact with the models the same way you did with the older model series, the models will often be verbose and provide less useful responses.
 
-- Chat Completion API.
-- Completion API with Chat Markup Language (ChatML).
+[!INCLUDE [Chat Completions](../includes/chat-completion.md)]
 
-The Chat Completion API is a new dedicated API for interacting with the GPT-35-Turbo and GPT-4 models. This API is the preferred method for accessing these models. **It is also the only way to access the new GPT-4 models**.
-
-ChatML uses the same [completion API](../reference.md#completions) that you use for other models like text-davinci-002, it requires a unique token based prompt format known as Chat Markup Language (ChatML). This provides lower level access than the dedicated Chat Completion API, but also requires additional input validation, only supports gpt-35-turbo models, and **the underlying format is more likely to change over time**.
-
-This article walks you through getting started with the GPT-35-Turbo and GPT-4 models. It's important to use the techniques described here to get the best results. If you try to interact with the models the same way you did with the older model series, the models will often be verbose and provide less useful responses.
-
-::: zone pivot="programming-language-chat-completions"
-
-[!INCLUDE [Studio quickstart](../includes/chat-completion.md)]
-
-::: zone-end
-
-::: zone pivot="programming-language-chat-ml"
-
-[!INCLUDE [Python SDK quickstart](../includes/chat-markup-language.md)]
-
-::: zone-end

articles/ai-services/openai/includes/chat-completion.md

@@ -6,7 +6,7 @@ author: mrbullwinkle #dereklegenzoff
 ms.author: mbullwin #delegenz
 ms.service: azure-ai-openai
 ms.topic: include
-ms.date: 11/02/2023
+ms.date: 04/05/2024
 manager: nitinme
 keywords: ChatGPT
 
@@ -16,6 +16,9 @@ keywords: ChatGPT
 
 The following code snippet shows the most basic way to use the GPT-3.5-Turbo and GPT-4 models with the Chat Completion API. If this is your first time using these models programmatically, we recommend starting with our [GPT-3.5-Turbo & GPT-4 Quickstart](../chatgpt-quickstart.md).
 
+> [!NOTE]  
+> In the Azure OpenAI documentation we refer to GPT-3.5-Turbo, and GPT-35-Turbo interchangeably. The official name of the model on OpenAI is `gpt-3.5-turbo`, but for Azure OpenAI due to Azure specific character constraints the underlying model name is `gpt-35-turbo`.
+
 # [OpenAI Python 1.x](#tab/python-new)
 
 ```python
@@ -173,7 +176,7 @@ Every response includes a `finish_reason`. The possible values for `finish_reaso
 * **stop**: API returned complete model output.
 * **length**: Incomplete model output due to max_tokens parameter or token limit.
 * **content_filter**: Omitted content due to a flag from our content filters.
-* **null**:API response still in progress or incomplete.
+* **null**: API response still in progress or incomplete.
 
 Consider setting `max_tokens` to a slightly higher value than normal such as 300 or 500. This ensures that the model doesn't stop generating text before it reaches the end of the message.
 
@@ -361,7 +364,7 @@ while True:
 
 ---
 
-When you run the code above you will get a blank console window. Enter your first question in the window and then hit enter. Once the response is returned, you can repeat the process and keep asking questions.
+When you run the code above you'll get a blank console window. Enter your first question in the window and then hit enter. Once the response is returned, you can repeat the process and keep asking questions.
 
 ## Managing conversations
 
@@ -542,6 +545,16 @@ An alternative approach is to limit the conversation duration to the max token l
 
 The token counting portion of the code demonstrated previously is a simplified version of one of [OpenAI's cookbook examples](https://github.com/openai/openai-cookbook/blob/main/examples/How_to_format_inputs_to_ChatGPT_models.ipynb).
 
+## Troubleshooting
+
+### Don't use ChatML syntax with the Chat Completions endpoint
+
+We have found that some customers will try using the [legacy ChatML syntax](../how-to/chat-markup-language.md) with the chat completion endpoints and newer models. ChatML was a preview capability that only worked with the legacy completions endpoint with the `gpt-35-turbo` version 0301 model which is [slated for retirement](../concepts/model-retirements.md). Attempting to use ChatML syntax with newer models and the chat completions endpoint can result errors as well as unexpected model response behavior, and is not recommended.  
+
+| Error |Cause | Solution |
+|---|---|---|
+| 400 - *Failed to generate output due to special tokens in the input.*  | Your prompt contains legacy ChatML tokens not recognized or supported by the model/endpoint. | Ensure that your prompt/messages array does not contain any legacy ChatML tokens. If you are upgrading from a legacy model, please exclude all special tokens before submitting an API request to the model.|
+
 ## Next steps
 
 * [Learn more about Azure OpenAI](../overview.md).

articles/ai-services/openai/toc.yml

@@ -111,6 +111,10 @@ items:
       href: ./how-to/integrate-synapseml.md
     - name: Deploy and use web apps
       href: ./how-to/use-web-app.md
+    - name: Legacy
+      items:
+      - name: Chat Markup Language (ChatML)
+        href: ./how-to/chat-markup-language.md 
   - name: Content filtering
     href: ./how-to/content-filters.md
   - name: Use blocklists