Merge pull request #279874 from MicrosoftDocs/main

Merge main to live, 4 AM

Author: v-ccolin

articles/ai-studio/reference/reference-model-inference-api.md

@@ -49,6 +49,13 @@ Models deployed to [serverless API endpoints](../how-to/deploy-models-serverless
 > * [Mistral-Large](../how-to/deploy-models-mistral.md)
 > * [Phi-3](../how-to/deploy-models-phi-3.md) family of models
 
+Models deployed to [managed inference](../concepts/deployments-overview.md):
+
+> [!div class="checklist"]
+> * [Meta Llama 3 instruct](../how-to/deploy-models-llama.md) family of models
+> * [Phi-3](../how-to/deploy-models-phi-3.md) family of models
+> * Mixtral famility of models
+
 The API is compatible with Azure OpenAI model deployments.
 
 ## Capabilities
@@ -66,6 +73,65 @@ The API indicates how developers can consume predictions for the following modal
 * [Image embeddings](reference-model-inference-images-embeddings.md): Creates an embedding vector representing the input text and image.
 
 
+### Inference SDK support
+
+You can use streamlined inference clients in the language of your choice to consume predictions from models running the Azure AI model inference API.
+
+# [Python](#tab/python)
+
+Install the package `azure-ai-inference` using your package manager, like pip:
+
+```bash
+pip install azure-ai-inference
+```
+
+Then, you can use the package to consume the model. The following example shows how to create a client to consume chat completions:
+
+```python
+import os
+from azure.ai.inference import ChatCompletionsClient
+from azure.core.credentials import AzureKeyCredential
+
+model = ChatCompletionsClient(
+    endpoint=os.environ["AZUREAI_ENDPOINT_URL"],
+    credential=AzureKeyCredential(os.environ["AZUREAI_ENDPOINT_KEY"]),
+)
+```
+
+# [JavaScript](#tab/javascript)
+
+Install the package `@azure-rest/ai-inference` using npm:
+
+```bash
+npm install @azure-rest/ai-inference
+```
+
+Then, you can use the package to consume the model. The following example shows how to create a client to consume chat completions:
+
+```javascript
+import ModelClient from "@azure-rest/ai-inference";
+import { isUnexpected } from "@azure-rest/ai-inference";
+import { AzureKeyCredential } from "@azure/core-auth";
+
+const client = new ModelClient(
+    process.env.AZUREAI_ENDPOINT_URL, 
+    new AzureKeyCredential(process.env.AZUREAI_ENDPOINT_KEY)
+);
+```
+
+# [REST](#tab/rest)
+
+Use the reference section to explore the API design and which parameters are available. For example, the reference section for [Chat completions](reference-model-inference-chat-completions.md) details how to use the route `/chat/completions` to generate predictions based on chat-formatted instructions:
+
+__Request__
+
+```HTTP/1.1
+POST /chat/completions?api-version=2024-04-01-preview
+Authorization: Bearer <bearer-token>
+Content-Type: application/json
+```
+---
+
 ### Extensibility
 
 The Azure AI Model Inference API specifies a set of modalities and parameters that models can subscribe to. However, some models may have further capabilities that the ones the API indicates. On those cases, the API allows the developer to pass them as extra parameters in the payload.
@@ -74,6 +140,38 @@ By setting a header `extra-parameters: allow`, the API will attempt to pass any
 
 The following example shows a request passing the parameter `safe_prompt` supported by Mistral-Large, which isn't specified in the Azure AI Model Inference API:
 
+# [Python](#tab/python)
+
+```python
+response = model.complete(
+    messages=[
+        SystemMessage(content="You are a helpful assistant."),
+        UserMessage(content="How many languages are in the world?"),
+    ],
+    model_extras={
+        "safe_mode": True
+    }
+)
+```
+
+# [JavaScript](#tab/javascript)
+
+```javascript
+var messages = [
+    { role: "system", content: "You are a helpful assistant" },
+    { role: "user", content: "How many languages are in the world?" },
+];
+
+var response = await client.path("/chat/completions").post({
+    body: {
+        messages: messages,
+        safe_mode: true
+    }
+});
+```
+
+# [REST](#tab/rest)
+
 __Request__
 
 ```HTTP/1.1
@@ -102,6 +200,8 @@ extra-parameters: allow
 }
 ```
 
+---
+
 > [!TIP]
 > Alternatively, you can set `extra-parameters: drop` to drop any unknown parameter in the request. Use this capability in case you happen to be sending requests with extra parameters that you know the model won't support but you want the request to completes anyway. A typical example of this is indicating `seed` parameter.
 
@@ -111,6 +211,71 @@ The Azure AI Model Inference API indicates a general set of capabilities but eac
 
 The following example shows the response for a chat completion request indicating the parameter `reponse_format` and asking for a reply in `JSON` format. In the example, since the model doesn't support such capability an error 422 is returned to the user. 
 
+# [Python](#tab/python)
+
+```python
+from azure.ai.inference.models import ChatCompletionsResponseFormat
+from azure.core.exceptions import HttpResponseError
+import json
+
+try:
+    response = model.complete(
+        messages=[
+            SystemMessage(content="You are a helpful assistant."),
+            UserMessage(content="How many languages are in the world?"),
+        ],
+        response_format={ "type": ChatCompletionsResponseFormat.JSON_OBJECT }
+    )
+except HttpResponseError as ex:
+    if ex.status_code == 422:
+        response = json.loads(ex.response._content.decode('utf-8'))
+        if isinstance(response, dict) and "detail" in response:
+            for offending in response["detail"]:
+                param = ".".join(offending["loc"])
+                value = offending["input"]
+                print(
+                    f"Looks like the model doesn't support the parameter '{param}' with value '{value}'"
+                )
+    else:
+        raise ex
+```
+
+# [JavaScript](#tab/javascript)
+
+```javascript
+try {
+    var messages = [
+        { role: "system", content: "You are a helpful assistant" },
+        { role: "user", content: "How many languages are in the world?" },
+    ];
+    
+    var response = await client.path("/chat/completions").post({
+        body: {
+            messages: messages,
+            response_format: { type: "json_object" }
+        }
+    });
+}
+catch (error) {
+    if (error.status_code == 422) {
+        var response = JSON.parse(error.response._content)
+        if (response.detail) {
+            for (const offending of response.detail) {
+                var param = offending.loc.join(".")
+                var value = offending.input
+                console.log(`Looks like the model doesn't support the parameter '${param}' with value '${value}'`)
+            }
+        }
+    }
+    else 
+    {
+        throw error
+    }
+}
+```
+
+# [REST](#tab/rest)
+
 __Request__
 
 ```HTTP/1.1
@@ -150,6 +315,7 @@ __Response__
     "message": "One of the parameters contain invalid values."
 }
 ```
+---
 
 > [!TIP]
 > You can inspect the property `details.loc` to understand the location of the offending parameter and `details.input` to see the value that was passed in the request.
@@ -160,6 +326,65 @@ The Azure AI model inference API supports [Azure AI Content Safety](../concepts/
 
 The following example shows the response for a chat completion request that has triggered content safety. 
 
+# [Python](#tab/python)
+
+```python
+from azure.ai.inference.models import AssistantMessage, UserMessage, SystemMessage
+
+try:
+    response = model.complete(
+        messages=[
+            SystemMessage(content="You are an AI assistant that helps people find information."),
+            UserMessage(content="Chopping tomatoes and cutting them into cubes or wedges are great ways to practice your knife skills."),
+        ]
+    )
+
+    print(response.choices[0].message.content)
+
+except HttpResponseError as ex:
+    if ex.status_code == 400:
+        response = json.loads(ex.response._content.decode('utf-8'))
+        if isinstance(response, dict) and "error" in response:
+            print(f"Your request triggered an {response['error']['code']} error:\n\t {response['error']['message']}")
+        else:
+            raise ex
+    else:
+        raise ex
+```
+
+# [JavaScript](#tab/javascript)
+
+```javascript
+try {
+    var messages = [
+        { role: "system", content: "You are an AI assistant that helps people find information." },
+        { role: "user", content: "Chopping tomatoes and cutting them into cubes or wedges are great ways to practice your knife skills." },
+    ]
+
+    var response = await client.path("/chat/completions").post({
+        body: {
+            messages: messages,
+        }
+    });
+    
+    console.log(response.body.choices[0].message.content)
+}
+catch (error) {
+    if (error.status_code == 400) {
+        var response = JSON.parse(error.response._content)
+        if (response.error) {
+            console.log(`Your request triggered an ${response.error.code} error:\n\t ${response.error.message}`)
+        }
+        else
+        {
+            throw error
+        }
+    }
+}
+```
+
+# [REST](#tab/rest)
+
 __Request__
 
 ```HTTP/1.1
@@ -196,95 +421,8 @@ __Response__
     "type": null
 }
 ```
+---
 
 ## Getting started
 
-The Azure AI Model Inference API is currently supported in models deployed as [Serverless API endpoints](../how-to/deploy-models-serverless.md). Deploy any of the [supported models](#availability) to a new [Serverless API endpoints](../how-to/deploy-models-serverless.md) to get started. Then you can consume the API in the following ways: 
-
-# [Studio](#tab/azure-studio)
-
-You can use the Azure AI Model Inference API to run evaluations or while building with *Prompt flow*. Create a [Serverless Model connection](../how-to/deploy-models-serverless-connect.md) to a *Serverless API endpoint* and consume its predictions. The Azure AI Model Inference API is used under the hood.
-
-# [Python](#tab/python)
-
-Since the API is OpenAI-compatible, you can use any supported SDK that already supports Azure OpenAI. In the following example, we show how you can use LiteLLM with the common API:
-
-```python
-import litellm
-
-client = litellm.LiteLLM(
-    base_url="https://<endpoint-name>.<region>.inference.ai.azure.com",
-    api_key="<key>",
-)
-
-response = client.chat.completions.create(
-    messages=[
-        {
-            "content": "Who is the most renowned French painter?", 
-            "role": "user"
-        }
-    ],
-    model="azureai",
-    custom_llm_provider="custom_openai",
-)
-
-print(response.choices[0].message.content)
-```
-
-# [REST](#tab/rest)
-
-Models deployed in Azure Machine Learning and Azure AI studio in Serverless API endpoints support the Azure AI Model Inference API. Each endpoint exposes the OpenAPI specification for the modalities the model support. Use the **Endpoint URI** and the **Key** to download the OpenAPI definition for the model. In the following example, we download it from a bash console. Replace `<TOKEN>` by the **Key** and `<ENDPOINT_URI>` for the **Endpoint URI**.
-
-```bash
-wget -d --header="Authorization: Bearer <TOKEN>" <ENDPOINT_URI>/swagger.json
-```
-
-Use the **Endpoint URI** and the **Key** to submit requests. The following example sends a request to a Cohere embedding model:
-
-```HTTP/1.1
-POST /embeddings?api-version=2024-04-01-preview
-Authorization: Bearer <bearer-token>
-Content-Type: application/json
-```
-
-```JSON
-{
-      "input": [
-        "Explain the theory of strings"
-      ],
-      "input_type": "query",
-      "encoding_format": "float",
-      "dimensions": 1024
-}
-```
-
-__Response__
-
-```json
-{
-    "id": "ab1c2d34-5678-9efg-hi01-0123456789ea",
-    "object": "list",
-    "data": [
-        {
-            "index": 0,
-            "object": "embedding",
-            "embedding": [
-                0.001912117,
-                0.048706055,
-                -0.06359863,
-                //...
-                -0.00044369698
-            ]
-        }
-    ],
-    "model": "",
-    "usage": {
-        "prompt_tokens": 7,
-        "completion_tokens": 0,
-        "total_tokens": 7
-    }
-}
-```
-
-
----
+The Azure AI Model Inference API is currently supported in certain models deployed as [Serverless API endpoints](../how-to/deploy-models-serverless.md) and Managed Online Endpoints. Deploy any of the [supported models](#availability) and use the exact same code to consume their predictions.

articles/app-service/overview-vnet-integration.md

@@ -70,9 +70,9 @@ The virtual network integration feature supports two virtual interfaces per work
 
 Virtual network integration depends on a dedicated subnet. When you create a subnet, the Azure subnet consumes five IPs from the start. One address is used from the integration subnet for each App Service plan instance. If you scale your app to four instances, then four addresses are used.
 
-When you scale up/down in instance size, the amount of IP addresses used by the App Service plan is temporarily doubled while the scale operation completes. The new instances need to be fully operational before the existing instances are deprovisioned. The scale operation affects the real, available supported instances for a given subnet size. Platform upgrades need free IP addresses to ensure upgrades can happen without interruptions to outbound traffic. Finally, after scale up, down, or in operations complete, there might be a short period of time before IP addresses are released. In rare cases, this operation can be up to 12 hours.
+When you scale up/down in instance size, the amount of IP addresses used by the App Service plan is temporarily doubled while the scale operation completes. The new instances need to be fully operational before the existing instances are deprovisioned. The scale operation affects the real, available supported instances for a given subnet size. Platform upgrades need free IP addresses to ensure upgrades can happen without interruptions to outbound traffic. Finally, after scale up, down, or in operations complete, there might be a short period of time before IP addresses are released. In rare cases, this operation can be up to 12 hours and if you rapidly scaling in/out or up/down, you need more IPs than the maximum scale.
 
-Because subnet size can't be changed after assignment, use a subnet that's large enough to accommodate whatever scale your app might reach. You should also reserve IP addresses for platform upgrades. To avoid any issues with subnet capacity, use a `/26` with 64 addresses. When you're creating subnets in Azure portal as part of integrating with the virtual network, a minimum size of `/27` is required. If the subnet already exists before integrating through the portal, you can use a `/28` subnet.
+Because subnet size can't be changed after assignment, use a subnet that's large enough to accommodate whatever scale your app might reach. You should also reserve IP addresses for platform upgrades. To avoid any issues with subnet capacity, we recommand allocating double the IPs of your planned maximum scale. A `/26` with 64 addresses cover the maximum scale of a single multitenant App Service plan. When you're creating subnets in Azure portal as part of integrating with the virtual network, a minimum size of `/27` is required. If the subnet already exists before integrating through the portal, you can use a `/28` subnet.
 
 With multi plan subnet join (MPSJ), you can join multiple App Service plans in to the same subnet. All App Service plans must be in the same subscription but the virtual network/subnet can be in a different subscription. Each instance from each App Service plan requires an IP address from the subnet and to use MPSJ a minimum size of `/26` subnet is required. If you plan to join many and/or large scale plans, you should plan for larger subnet ranges.