Update reference-model-inference-api.md

santiagxf · web-flow · commit 486d99905d3d · 2024-07-02T00:22:37.000-04:00
diff --git a/articles/ai-studio/reference/reference-model-inference-api.md b/articles/ai-studio/reference/reference-model-inference-api.md
@@ -49,6 +49,13 @@ Models deployed to [serverless API endpoints](../how-to/deploy-models-serverless
 > * [Mistral-Large](../how-to/deploy-models-mistral.md)
 > * [Phi-3](../how-to/deploy-models-phi-3.md) family of models
 
+Models deployed to [managed inference](../concepts/deployments-overview.md):
+
+> [!div class="checklist"]
+> * [Meta Llama 3 instruct](../how-to/deploy-models-llama.md) family of models
+> * [Phi-3](../how-to/deploy-models-phi-3.md) family of models
+> * Mixtral famility of models
+
 The API is compatible with Azure OpenAI model deployments.
 
 ## Capabilities
@@ -66,6 +73,50 @@ The API indicates how developers can consume predictions for the following modal
 * [Image embeddings](reference-model-inference-images-embeddings.md): Creates an embedding vector representing the input text and image.
 
 
+### Inference SDK support
+
+You can use streamlined inference clients in the language of your choice to consume predictions from models running the API.
+
+# [REST](#tab/python)
+
+```python
+import os
+from azure.ai.inference import ChatCompletionsClient
+from azure.core.credentials import AzureKeyCredential
+
+model = ChatCompletionsClient(
+    endpoint=os.environ["AZUREAI_ENDPOINT_URL"],
+    credential=AzureKeyCredential(os.environ["AZUREAI_ENDPOINT_KEY"]),
+)
+```
+
+# [REST](#tab/javascript)
+
+```javascript
+import ModelClient from "@azure-rest/ai-inference";
+import { isUnexpected } from "@azure-rest/ai-inference";
+import { AzureKeyCredential } from "@azure/core-auth";
+
+const client = new ModelClient(
+    process.env.AZUREAI_ENDPOINT_URL, 
+    new AzureKeyCredential(process.env.AZUREAI_ENDPOINT_KEY)
+);
+```
+
+# [REST](#tab/rest)
+
+Use the reference section to explore the API design and which parameters are available. For example, the reference section for [Chat completions](reference-model-inference-chat-completions.md) details how to use the route `/chat/completions` to generate predictions based on chat-formatted instructions:
+
+__Request__
+
+```HTTP/1.1
+POST /chat/completions?api-version=2024-04-01-preview
+Authorization: Bearer <bearer-token>
+Content-Type: application/json
+```
+
+---
+
 ### Extensibility
 
 The Azure AI Model Inference API specifies a set of modalities and parameters that models can subscribe to. However, some models may have further capabilities that the ones the API indicates. On those cases, the API allows the developer to pass them as extra parameters in the payload.
@@ -74,6 +125,38 @@ By setting a header `extra-parameters: allow`, the API will attempt to pass any
 
 The following example shows a request passing the parameter `safe_prompt` supported by Mistral-Large, which isn't specified in the Azure AI Model Inference API:
 
+# [REST](#tab/python)
+
+```python
+response = model.complete(
+    messages=[
+        SystemMessage(content="You are a helpful assistant."),
+        UserMessage(content="How many languages are in the world?"),
+    ],
+    model_extras={
+        "safe_mode": True
+    }
+)
+```
+
+# [REST](#tab/javascript)
+
+```javascript
+var messages = [
+    { role: "system", content: "You are a helpful assistant" },
+    { role: "user", content: "How many languages are in the world?" },
+];
+
+var response = await client.path("/chat/completions").post({
+    body: {
+        messages: messages,
+        safe_mode: true
+    }
+});
+```
+
+# [REST](#tab/rest)
+
 __Request__
 
 ```HTTP/1.1
@@ -102,6 +185,8 @@ extra-parameters: allow
 }
 ```
 
+---
+
 > [!TIP]
 > Alternatively, you can set `extra-parameters: drop` to drop any unknown parameter in the request. Use this capability in case you happen to be sending requests with extra parameters that you know the model won't support but you want the request to completes anyway. A typical example of this is indicating `seed` parameter.
 
@@ -111,6 +196,71 @@ The Azure AI Model Inference API indicates a general set of capabilities but eac
 
 The following example shows the response for a chat completion request indicating the parameter `reponse_format` and asking for a reply in `JSON` format. In the example, since the model doesn't support such capability an error 422 is returned to the user. 
 
+# [REST](#tab/python)
+
+```python
+from azure.ai.inference.models import ChatCompletionsResponseFormat
+from azure.core.exceptions import HttpResponseError
+import json
+
+try:
+    response = model.complete(
+        messages=[
+            SystemMessage(content="You are a helpful assistant."),
+            UserMessage(content="How many languages are in the world?"),
+        ],
+        response_format={ "type": ChatCompletionsResponseFormat.JSON_OBJECT }
+    )
+except HttpResponseError as ex:
+    if ex.status_code == 422:
+        response = json.loads(ex.response._content.decode('utf-8'))
+        if isinstance(response, dict) and "detail" in response:
+            for offending in response["detail"]:
+                param = ".".join(offending["loc"])
+                value = offending["input"]
+                print(
+                    f"Looks like the model doesn't support the parameter '{param}' with value '{value}'"
+                )
+    else:
+        raise ex
+```
+
+# [REST](#tab/python)
+
+```javascript
+try {
+    var messages = [
+        { role: "system", content: "You are a helpful assistant" },
+        { role: "user", content: "How many languages are in the world?" },
+    ];
+    
+    var response = await client.path("/chat/completions").post({
+        body: {
+            messages: messages,
+            response_format: { type: "json_object" }
+        }
+    });
+}
+catch (error) {
+    if (error.status_code == 422) {
+        var response = JSON.parse(error.response._content)
+        if (response.detail) {
+            for (const offending of response.detail) {
+                var param = offending.loc.join(".")
+                var value = offending.input
+                console.log(`Looks like the model doesn't support the parameter '${param}' with value '${value}'`)
+            }
+        }
+    }
+    else 
+    {
+        throw error
+    }
+}
+```
+
+# [REST](#tab/rest)
+
 __Request__
 
 ```HTTP/1.1
@@ -150,6 +300,7 @@ __Response__
     "message": "One of the parameters contain invalid values."
 }
 ```
+---
 
 > [!TIP]
 > You can inspect the property `details.loc` to understand the location of the offending parameter and `details.input` to see the value that was passed in the request.
@@ -160,6 +311,65 @@ The Azure AI model inference API supports [Azure AI Content Safety](../concepts/
 
 The following example shows the response for a chat completion request that has triggered content safety. 
 
+# [REST](#tab/python)
+
+```python
+from azure.ai.inference.models import AssistantMessage, UserMessage, SystemMessage
+
+try:
+    response = model.complete(
+        messages=[
+            SystemMessage(content="You are an AI assistant that helps people find information."),
+            UserMessage(content="Chopping tomatoes and cutting them into cubes or wedges are great ways to practice your knife skills."),
+        ]
+    )
+
+    print(response.choices[0].message.content)
+
+except HttpResponseError as ex:
+    if ex.status_code == 400:
+        response = json.loads(ex.response._content.decode('utf-8'))
+        if isinstance(response, dict) and "error" in response:
+            print(f"Your request triggered an {response['error']['code']} error:\n\t {response['error']['message']}")
+        else:
+            raise ex
+    else:
+        raise ex
+```
+
+# [REST](#tab/javascript)
+
+```javascript
+try {
+    var messages = [
+        { role: "system", content: "You are an AI assistant that helps people find information." },
+        { role: "user", content: "Chopping tomatoes and cutting them into cubes or wedges are great ways to practice your knife skills." },
+    ]
+
+    var response = await client.path("/chat/completions").post({
+        body: {
+            messages: messages,
+        }
+    });
+    
+    console.log(response.body.choices[0].message.content)
+}
+catch (error) {
+    if (error.status_code == 400) {
+        var response = JSON.parse(error.response._content)
+        if (response.error) {
+            console.log(`Your request triggered an ${response.error.code} error:\n\t ${response.error.message}`)
+        }
+        else
+        {
+            throw error
+        }
+    }
+}
+```
+
+# [REST](#tab/rest)
+
 __Request__
 
 ```HTTP/1.1
@@ -196,95 +406,4 @@ __Response__
     "type": null
 }
 ```
-
-## Getting started
-
-The Azure AI Model Inference API is currently supported in models deployed as [Serverless API endpoints](../how-to/deploy-models-serverless.md). Deploy any of the [supported models](#availability) to a new [Serverless API endpoints](../how-to/deploy-models-serverless.md) to get started. Then you can consume the API in the following ways: 
-
-# [Studio](#tab/azure-studio)
-
-You can use the Azure AI Model Inference API to run evaluations or while building with *Prompt flow*. Create a [Serverless Model connection](../how-to/deploy-models-serverless-connect.md) to a *Serverless API endpoint* and consume its predictions. The Azure AI Model Inference API is used under the hood.
-
-# [Python](#tab/python)
-
-Since the API is OpenAI-compatible, you can use any supported SDK that already supports Azure OpenAI. In the following example, we show how you can use LiteLLM with the common API:
-
-```python
-import litellm
-
-client = litellm.LiteLLM(
-    base_url="https://<endpoint-name>.<region>.inference.ai.azure.com",
-    api_key="<key>",
-)
-
-response = client.chat.completions.create(
-    messages=[
-        {
-            "content": "Who is the most renowned French painter?", 
-            "role": "user"
-        }
-    ],
-    model="azureai",
-    custom_llm_provider="custom_openai",
-)
-
-print(response.choices[0].message.content)
-```
-
-# [REST](#tab/rest)
-
-Models deployed in Azure Machine Learning and Azure AI studio in Serverless API endpoints support the Azure AI Model Inference API. Each endpoint exposes the OpenAPI specification for the modalities the model support. Use the **Endpoint URI** and the **Key** to download the OpenAPI definition for the model. In the following example, we download it from a bash console. Replace `<TOKEN>` by the **Key** and `<ENDPOINT_URI>` for the **Endpoint URI**.
-
-```bash
-wget -d --header="Authorization: Bearer <TOKEN>" <ENDPOINT_URI>/swagger.json
-```
-
-Use the **Endpoint URI** and the **Key** to submit requests. The following example sends a request to a Cohere embedding model:
-
-```HTTP/1.1
-POST /embeddings?api-version=2024-04-01-preview
-Authorization: Bearer <bearer-token>
-Content-Type: application/json
-```
-
-```JSON
-{
-      "input": [
-        "Explain the theory of strings"
-      ],
-      "input_type": "query",
-      "encoding_format": "float",
-      "dimensions": 1024
-}
-```
-
-__Response__
-
-```json
-{
-    "id": "ab1c2d34-5678-9efg-hi01-0123456789ea",
-    "object": "list",
-    "data": [
-        {
-            "index": 0,
-            "object": "embedding",
-            "embedding": [
-                0.001912117,
-                0.048706055,
-                -0.06359863,
-                //...
-                -0.00044369698
-            ]
-        }
-    ],
-    "model": "",
-    "usage": {
-        "prompt_tokens": 7,
-        "completion_tokens": 0,
-        "total_tokens": 7
-    }
-}
-```
-
-
 ---
