Merge pull request #273822 from ChenJieting/jieting/llm-tool-update

update the llm tool doc for build

Author: ttorble

articles/ai-studio/how-to/prompt-flow-tools/llm-tool.md

@@ -17,14 +17,51 @@ author: lgayhardt
 
 [!INCLUDE [Azure AI Studio preview](../../includes/preview-ai-studio.md)]
 
-To use large language models (LLMs) for natural language processing, you use the prompt flow LLM tool.
+The large language model (LLM) tool in prompt flow enables you to take advantage of widely used large language models like [OpenAI](https://platform.openai.com/), [Azure OpenAI Service](../../../ai-services/openai/overview.md), and models in [Azure AI Studio model catalog](../model-catalog.md) for natural language processing.
+> [!NOTE]
+> The previous version of the LLM tool is now being deprecated. Please upgrade to latest [promptflow-tools](https://pypi.org/project/promptflow-tools/) package to consume new llm tools.
+
+Prompt flow provides a few different large language model APIs:
+
+- [Completion](https://platform.openai.com/docs/api-reference/completions): OpenAI's completion models generate text based on provided prompts.
+- [Chat](https://platform.openai.com/docs/api-reference/chat): OpenAI's chat models facilitate interactive conversations with text-based inputs and responses.
 
 > [!NOTE]
-> For embeddings to convert text into dense vector representations for various natural language processing tasks, see [Embedding tool](embedding-tool.md).
+> Don't use non-ascii characters in resource group name of Azure OpenAI resource, prompt flow didn't support this case.
 
 ## Prerequisites
 
-Prepare a prompt as described in the [Prompt tool](prompt-tool.md#prerequisites) documentation. The LLM tool and Prompt tool both support [Jinja](https://jinja.palletsprojects.com/en/3.1.x/) templates. For more information and best practices, see [Prompt engineering techniques](../../../ai-services/openai/concepts/advanced-prompt-engineering.md).
+Create OpenAI resources, Azure OpenAI resources, or MaaS deployment with the LLM models (for example: llama2, mistral, cohere etc.) in Azure AI Studio model catalog:
+
+- **OpenAI**:
+
+    - Sign up your account on the [OpenAI website](https://openai.com/).
+    - Sign in and [find your personal API key](https://platform.openai.com/account/api-keys).
+
+- **Azure OpenAI**:
+
+    - [Create Azure OpenAI resources](../../../ai-services/openai/how-to/create-resource.md).
+
+- **MaaS deployment**:
+
+    [Create MaaS deployment for models in Azure AI Studio model catalog](../../concepts/deployments-overview.md#deploy-models-with-model-as-a-service).
+
+    You can create serverless connection to use this MaaS deployment.
+
+## Connections
+
+Set up connections to provisioned resources in prompt flow.
+
+| Type        | Name     | API key  | API base | API type | API version |
+|-------------|----------|----------|----------|----------|-------------|
+| OpenAI      | Required | Required | -        | -        | -           |
+| Azure OpenAI| Required | Required | Required | Required | Required    |
+| Serverless  | Required | Required | Required | -        | -           |
+
+  > [!TIP]
+  > - To use Microsoft Entra ID auth type for Azure OpenAI connection, you need assign either the `Cognitive Services OpenAI User` or `Cognitive Services OpenAI Contributor role` to user or user assigned managed identity.
+  > - Learn more about [how to specify to use user identity to submit flow run](../create-manage-runtime.md#create-an-automatic-runtime-on-a-flow-page).
+  > - Learn more about [How to configure Azure OpenAI Service with managed identities](../../../ai-services/openai/how-to/managed-identity.md).
 
 ## Build with the LLM tool
 
@@ -35,7 +72,7 @@ Prepare a prompt as described in the [Prompt tool](prompt-tool.md#prerequisites)
 
 1. Select the connection to one of your provisioned resources. For example, select **Default_AzureOpenAI**.
 1. From the **Api** dropdown list, select **chat** or **completion**.
-1. Enter values for the LLM tool input parameters described in the [Text completion inputs table](#inputs). If you selected the **chat** API, see the [Chat inputs table](#chat-inputs). If you selected the **completion** API, see the [Text completion inputs table](#text-completion-inputs). For information about how to prepare the prompt input, see [Prerequisites](#prerequisites).
+1. Enter values for the LLM tool input parameters described in the [Text completion inputs table](#inputs). If you selected the **chat** API, see the [Chat inputs table](#chat-inputs). If you selected the **completion** API, see the [Text completion inputs table](#text-completion-inputs). For information about how to prepare the prompt input, see [How to write a prompt](#how-to-write-a-prompt).
 1. Add more tools to your flow, as needed. Or select **Run** to run the flow.
 1. The outputs are described in the [Outputs table](#outputs).
 
@@ -74,15 +111,146 @@ The following input parameters are available.
 | presence\_penalty      | float       | The value that controls the model's behavior regarding repeating phrases. Default is 0.      | No       |
 | frequency\_penalty     | float       | The value that controls the model's behavior regarding generating rare phrases. Default is 0. | No       |
 | logit\_bias            | dictionary  | The logit bias for the language model. Default is empty dictionary.                            | No       |
+| tool\_choice           | object      | Value that controls which tool is called by the model. Default is null.                        | No       |
+| tools                  | list        | A list of tools the model may generate JSON inputs for. Default is null.                       | No       |
+| response_format        | object      | An object specifying the format that the model must output. Default is null.                   | No       |
 
 ## Outputs
 
 The output varies depending on the API you selected for inputs.
 
-| API        | Return type | Description                              |
-|------------|-------------|------------------------------------------|
-| Completion | string      | The text of one predicted completion.     |
-| Chat       | string      | The text of one response of conversation. |
+| Return type | Description                              |
+|-------------|------------------------------------------|
+| string      | Text of one predicted completion or response of conversation |
+
+## How to write a prompt?
+
+Prepare a prompt as described in the [Prompt tool](prompt-tool.md#prerequisites) documentation. The LLM tool and Prompt tool both support [Jinja](https://jinja.palletsprojects.com/en/3.1.x/) templates. For more information and best practices, see [Prompt engineering techniques](../../../ai-services/openai/concepts/advanced-prompt-engineering.md).
+
+For example, for a chat prompt we offer a method to distinguish between different roles in a chat prompt, such as "system", "user", "assistant" and "tool". The "system", "user", "assistant" roles can have "name" and "content" properties. The "tool" role, however, should have "tool_call_id" and "content" properties. For an example of a tool chat prompt, please refer to [Sample 3](#sample-3).
+
+### Sample 1
+```jinja
+# system:
+You are a helpful assistant.
+
+{% for item in chat_history %}
+# user:
+{{item.inputs.question}}
+# assistant:
+{{item.outputs.answer}}
+{% endfor %}
+
+# user:
+{{question}}
+```
+
+In LLM tool, the prompt is transformed to match the [OpenAI messages](https://platform.openai.com/docs/api-reference/chat/create#chat-create-messages) structure before sending to OpenAI chat API.
+
+```
+[
+    {
+        "role": "system",
+        "content": "You are a helpful assistant."
+    },
+    {
+        "role": "user",
+        "content": "<question-of-chat-history-round-1>"
+    },
+    {
+        "role": "assistant",
+        "content": "<answer-of-chat-history-round-1>"
+    },
+    ...
+    {
+        "role": "user",
+        "content": "<question>"
+    }
+]
+```
+
+### Sample 2
+```jinja
+# system:
+{# For role naming customization, the following syntax is used #}
+## name:
+Alice
+## content:
+You are a bot can tell good jokes.
+```
+
+In LLM tool, the prompt is transformed to match the [OpenAI messages](https://platform.openai.com/docs/api-reference/chat/create#chat-create-messages) structure before sending to OpenAI chat API.
+
+```
+[
+    {
+        "role": "system",
+        "name": "Alice",
+        "content": "You are a bot can tell good jokes."
+    }
+]
+```
+
+### Sample 3
+This sample illustrates how to write a tool chat prompt.
+```jinja
+# system:
+You are a helpful assistant.
+# user:
+What is the current weather like in Boston?
+# assistant:
+{# The assistant message with 'tool_calls' must be followed by messages with role 'tool'. #}
+## tool_calls:
+{{llm_output.tool_calls}}
+# tool:
+{#
+Messages with role 'tool' must be a response to a preceding message with 'tool_calls'.
+Additionally, 'tool_call_id's should match ids of assistant message 'tool_calls'.
+#}
+## tool_call_id:
+{{llm_output.tool_calls[0].id}}
+## content:
+{{tool-answer-of-last-question}}
+# user:
+{{question}}
+```
+
+In LLM tool, the prompt is transformed to match the [OpenAI messages](https://platform.openai.com/docs/api-reference/chat/create#chat-create-messages) structure before sending to OpenAI chat API.
+
+```
+[
+    {
+        "role": "system",
+        "content": "You are a helpful assistant."
+    },
+    {
+        "role": "user",
+        "content": "What is the current weather like in Boston?"
+    },
+    {
+        "role": "assistant",
+        "content": null,
+        "function_call": null,
+        "tool_calls": [
+            {
+                "id": "<tool-call-id-of-last-question>",
+                "type": "function",
+                "function": "<function-to-call-of-last-question>"
+            }
+        ]
+    },
+    {
+        "role": "tool",
+        "tool_call_id": "<tool-call-id-of-last-question>",
+        "content": "<tool-answer-of-last-question>"
+    }
+    ...
+    {
+        "role": "user",
+        "content": "<question>"
+    }
+]
+```
 
 ## Next steps
 

articles/ai-studio/how-to/prompt-flow-tools/prompt-flow-tools-overview.md

@@ -19,7 +19,7 @@ The following table provides an index of tools in prompt flow.
 
 | Tool name | Description | Package name |
 |------|-----------|-------------|--------------|
-| [LLM](./llm-tool.md) | Use large language models (LLM) with the Azure OpenAI Service for tasks such as text completion or chat. | [promptflow-tools](https://pypi.org/project/promptflow-tools/) |
+| [LLM](./llm-tool.md) | Use large language models (LLM) for tasks such as text completion or chat. | [promptflow-tools](https://pypi.org/project/promptflow-tools/) |
 | [Prompt](./prompt-tool.md) | Craft a prompt by using Jinja as the templating language. | [promptflow-tools](https://pypi.org/project/promptflow-tools/) |
 | [Python](./python-tool.md) | Run Python code. | [promptflow-tools](https://pypi.org/project/promptflow-tools/) |
 | [Azure OpenAI GPT-4 Turbo with Vision](./azure-open-ai-gpt-4v-tool.md) | Use an Azure OpenAI GPT-4 Turbo with Vision model deployment to analyze images and provide textual responses to questions about them. | [promptflow-tools](https://pypi.org/project/promptflow-tools/) |