You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
# How to deploy Mistral models with Azure Machine Learning studio
19
19
Mistral AI offers two categories of models in Azure Machine Learning studio:
20
20
21
-
- Premium models: Mistral-large. These models are available with pay-as-you-go token based billing with Models as a Service in the studio model catalog.
21
+
- Premium models: Mistral Large. These models are available with pay-as-you-go token based billing with Models as a Service in the studio model catalog.
22
22
- Open models: Mixtral-8x7B-Instruct-v01, Mixtral-8x7B-v01, Mistral-7B-Instruct-v01, and Mistral-7B-v01. These models are also available in the Azure Machine Learning studio model catalog and can be deployed to dedicated VM instances in your own Azure subscription with managed online endpoints.
23
23
24
24
You can browse the Mistral family of models in the model catalog by filtering on the Mistral collection.
25
25
26
-
## Mistral-large
26
+
## Mistral Large
27
27
28
-
In this article, you learn how to use Azure Machine Learning studio to deploy the Mistral-large model as a service with pay-as you go billing.
28
+
In this article, you learn how to use Azure Machine Learning studio to deploy the Mistral Large model as a service with pay-as you go billing.
29
29
30
-
Mistral-large is Mistral AI's most advanced Large Language Model (LLM). It can be used on any language-based task thanks to its state-of-the-art reasoning and knowledge capabilities.
30
+
Mistral Large is Mistral AI's most advanced Large Language Model (LLM). It can be used on any language-based task thanks to its state-of-the-art reasoning and knowledge capabilities.
31
31
32
32
Additionally, mistral-large is:
33
33
34
-
- Straight-to-the-point. Purposely trained to eliminate unnecessary verbosity and generate concise outputs.
35
-
- Specialized in RAG. Crucial information is not lost in the middle of long context windows (up to 32K tokens).
36
-
- Strong in coding. Code generation, review, and comments. Can output results as JSON and do function calling.
34
+
- Specialized in RAG. Crucial information isn't lost in the middle of long context windows (up to 32 K tokens).
35
+
- Strong in coding. Code generation, review, and comments. Supports all mainstream coding languages.
37
36
- Multi-lingual by design. Best-in-class performance in French, German, Spanish, and Italian - in addition to English. Dozens of other languages are supported.
37
+
- Responsible AI. Efficient guardrails baked in the model, with additional safety layer with safe_mode option.
Certain models in the model catalog can be deployed as a service with pay-as-you-go, providing a way to consume them as an API without hosting them on your subscription, while keeping the enterprise security and compliance organizations need. This deployment option doesn't require quota from your subscription.
44
44
45
-
Mistral-large can be deployed as a service with pay-as-you-go, and is offered by Mistral AI through the Microsoft Azure Marketplace. Please note that Mistral AI can change or update the terms of use and pricing of this model.
45
+
Mistral Large can be deployed as a service with pay-as-you-go, and is offered by Mistral AI through the Microsoft Azure Marketplace. Please note that Mistral AI can change or update the terms of use and pricing of this model.
46
46
47
47
### Azure Marketplace model offerings
48
48
49
49
The following models are available in Azure Marketplace for Mistral AI when deployed as a service with pay-as-you-go:
50
50
51
-
* Mistral-large (preview)
51
+
* Mistral Large (preview)
52
52
53
53
### Prerequisites
54
54
55
55
- An Azure subscription with a valid payment method. Free or trial Azure subscriptions won't work. If you don't have an Azure subscription, create a [paid Azure account](https://azure.microsoft.com/pricing/purchase-options/pay-as-you-go) to begin.
56
-
- An Azure Machine Learning workspace and a compute instance. If you don't have these, use the steps in the [Quickstart: Create workspace resources](quickstart-create-resources.md) article to create them.
56
+
- An Azure Machine Learning workspace. If you don't have these, use the steps in the [Quickstart: Create workspace resources](quickstart-create-resources.md) article to create them.
57
57
58
58
> [!IMPORTANT]
59
59
> Pay-as-you-go model deployment offering is only available in workspaces created in **East US 2** and **France Central** regions.
60
60
61
-
- Azure role-based access controls (Azure RBAC) are used to grant access to operations in Azure Machine Learning. To perform the steps in this article, your user account must be assigned the __owner__ or __contributor__ role for the Azure subscription. Alternatively, your account can be assigned a custom role that has the following permissions:
62
-
63
-
- On the Azure subscription—to subscribe the workspace to the Azure Marketplace offering, once for each workspace, per offering:
- Azure role-based access controls (Azure RBAC) are used to grant access to operations. To perform the steps in this article, your user account must be assigned the __Azure AI Developer role__ on the Resouce Group.
77
62
78
63
For more information on permissions, see [Manage access to an Azure Machine Learning workspace](how-to-assign-roles.md).
79
64
@@ -93,14 +78,11 @@ To create a deployment:
93
78
94
79
1. In the deployment wizard, select the link to **Azure Marketplace Terms** to learn more about the terms of use.
95
80
1. You can also select the **Marketplace offer details** tab to learn about pricing for the selected model.
96
-
1. If this is your first time deploying the model in the workspace, you have to subscribe your workspace for the particular offering (for example, Mistral-large). This step requires that your account has the Azure subscription permissions and resource group permissions listed in the prerequisites. Each workspace has its own subscription to the particular Azure Marketplace offering, which allows you to control and monitor spending. Select **Subscribe and Deploy**. Currently you can have only one deployment for each model within a workspace.
97
-
98
-
> [!NOTE]
99
-
> Subscribing a workspace to a particular Azure Marketplace offering (in this case, Mistral-large) requires that your account has **Contributor** or **Owner** access at the subscription level where the project is created. Alternatively, your user account can be assigned a custom role that has the Azure subscription permissions and resource group permissions listed in the [prerequisites](#prerequisites).
81
+
1. If this is your first time deploying the model in the workspace, you have to subscribe your workspace for the particular offering (for example, Mistral-large). This step requires that your account has the **Azure AI Developer role** permissions on the Resource Group, as listed in the prerequisites. Each workspace has its own subscription to the particular Azure Marketplace offering, which allows you to control and monitor spending. Select **Subscribe and Deploy**. Currently you can have only one deployment for each model within a workspace.
100
82
101
83
:::image type="content" source="media/how-to-deploy-models-mistral/mistral-deploy-marketplace-terms.png" alt-text="A screenshot showing the terms and conditions of a given model." lightbox="media/how-to-deploy-models-mistral/mistral-deploy-marketplace-terms.png":::
102
84
103
-
1. Once you subscribe the workspace for the particular Azure Marketplace offering, subsequent deployments of the _same_ offering in the _same_ workspace don't require subscribing again. Therefore, you don't need to have the subscription-level permissions for subsequent deployments. If this scenario applies to you, you will see a **Continue to deploy** option to select.
85
+
1. Once you subscribe the workspace for the particular Azure Marketplace offering, subsequent deployments of the _same_ offering in the _same_ workspace don't require subscribing again. If this scenario applies to you, you will see a **Continue to deploy** option to select.
104
86
105
87
:::image type="content" source="media/how-to-deploy-models-mistral/mistral-deploy-pay-as-you-go-project.png" alt-text="A screenshot showing a project that is already subscribed to the offering." lightbox="media/how-to-deploy-models-mistral/mistral-deploy-pay-as-you-go-project.png":::
106
88
@@ -110,15 +92,15 @@ To create a deployment:
110
92
111
93
1. Select **Deploy**. Wait until the deployment is finished and you're redirected to the serverless endpoints page.
112
94
1. Select the endpoint to open its Details page.
113
-
1. Select the **Test** tab to start interacting with the model.
114
-
1. You can also take note of the **Target** URL and the **Secret Key** to call the deployment and generate chat completions.
95
+
1. Select the **Test** tab to start interacting with the model.
115
96
1. You can always find the endpoint's details, URL, and access keys by navigating to **Workspace** > **Endpoints** > **Serverless endpoints**.
97
+
1. Take note of the **Target** URL and the **Secret Key** to call the deployment and generate chat completions using the [`<target_url>/v1/chat/completions`](#chat-api) API.
116
98
117
99
To learn about billing for Mistral models deployed with pay-as-you-go, see [Cost and quota considerations for Mistral models deployed as a service](#cost-and-quota-considerations-for-mistral-large-deployed-as-a-service).
118
100
119
-
### Consume the Mistral-large model as a service
101
+
### Consume the Mistral Large model as a service
120
102
121
-
Mistral-large can be consumed using the chat API.
103
+
Mistral Large can be consumed using the chat API.
122
104
123
105
1. In the **workspace**, select **Endpoints** > **Serverless endpoints**.
124
106
1. Find and select the deployment you created.
@@ -150,7 +132,7 @@ Payload is a JSON formatted string containing the following parameters:
150
132
|-----|-----|-----|-----|
151
133
|`messages`|`string`| No default. This value must be specified. | The message or history of messages to use to prompt the model. |
152
134
|`stream`|`boolean`|`False`| Streaming allows the generated tokens to be sent as data-only server-sent events whenever they become available. |
153
-
|`max_tokens`|`integer`|`16`| The maximum number of tokens to generate in the completion. The token count of your prompt plus `max_tokens` can't exceed the model's context length. |
135
+
|`max_tokens`|`integer`|`8192`| The maximum number of tokens to generate in the completion. The token count of your prompt plus `max_tokens` can't exceed the model's context length. |
154
136
|`top_p`|`float`|`1`| An alternative to sampling with temperature, called nucleus sampling, where the model considers the results of the tokens with `top_p` probability mass. So 0.1 means only the tokens comprising the top 10% probability mass are considered. We generally recommend altering `top_p` or `temperature`, but not both. |
155
137
|`temperature`|`float`|`1`| The sampling temperature to use, between 0 and 2. Higher values mean the model samples more broadly the distribution of tokens. Zero means greedy sampling. We recommend altering this or `top_p`, but not both. |
156
138
|`ignore_eos`|`boolean`|`False`| Whether to ignore the EOS token and continue generating tokens after the EOS token is generated. |
@@ -262,15 +244,15 @@ The following is an example response:
### Cost and quota considerations for Mistral-large deployed as a service
255
+
### Cost and quota considerations for Mistral Large deployed as a service
274
256
275
257
Mistral models deployed as a service are offered by Mistral AI through Azure Marketplace and integrated with Azure Machine Learning studio for use. You can find Azure Marketplace pricing when deploying the models.
276
258
@@ -280,10 +262,6 @@ For more information on how to track costs, see [Monitor costs for models offere
280
262
281
263
Quota is managed per deployment. Each deployment has a rate limit of 200,000 tokens per minute and 1,000 API requests per minute. However, we currently limit one deployment per model per project. Contact Microsoft Azure Support if the current rate limits aren't sufficient for your scenarios.
282
264
283
-
## Data and policy
284
-
285
-
No data from the user using models deployed as a service with pay-as-you-go is sent to the model provider (in this case Mistral AI).
286
-
287
265
## Content filtering
288
266
289
267
Models deployed as a service with pay-as-you-go are protected by Azure AI content safety. With Azure AI content safety enabled, both the prompt and completion pass through an ensemble of classification models aimed at detecting and preventing the output of harmful content. The content filtering system detects and takes action on specific categories of potentially harmful content in both input prompts and output completions. Learn more about [Azure AI Content Safety](/azure/ai-services/content-safety/overview).
![prmerger-automator[bot]](https://avatars.githubusercontent.com/u/22479449?v=4&size=40){kind=avatar}
0 commit comments