You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
|[Mistral-OCR-2503](https://aka.ms/aistudio/landing/mistral-ocr-2503)| image to text | - **Input:** image or PDF pages (1,000 pages, max 50MB PDF file) <br> - **Output:** text <br /> - **Tool calling:** No <br /> - **Response formats:** Text, JSON, Markdown |
265
+
|[Mistral-OCR-2503](https://aka.ms/aistudio/landing/mistral-ocr-2503)|[image to text](../how-to/use-image-models.md)| - **Input:** image or PDF pages (1,000 pages, max 50MB PDF file) <br> - **Output:** text <br /> - **Tool calling:** No <br /> - **Response formats:** Text, JSON, Markdown |
266
266
|[Mistral-small-2503](https://aka.ms/aistudio/landing/mistral-small-2503)|[chat-completion (with images)](../model-inference/how-to/use-chat-multi-modal.md?context=/azure/ai-foundry/context/context)| - **Input:** text and images (131,072 tokens), <br> image-based tokens are 16px x 16px <br> blocks of the original images <br /> - **Output:** text (4,096 tokens) <br /> - **Tool calling:** Yes <br /> - **Response formats:** Text, JSON |
title: How to use image-to-text models in the model catalog
3
+
titleSuffix: Azure AI Foundry
4
+
description: Learn how to use image-to-text models from the AI Foundry model catalog.
5
+
manager: scottpolly
6
+
author: msakande
7
+
reviewer: frogglew
8
+
ms.service: azure-ai-model-inference
9
+
ms.topic: how-to
10
+
ms.date: 05/02/2025
11
+
ms.author: mopeakande
12
+
ms.reviewer: frogglew
13
+
ms.custom: references_regions, tool_generated
14
+
---
15
+
16
+
# How to use image-to-text models in the model catalog
17
+
18
+
This article explains how to use _image-to-text_ models in the AI Foundry model catalog.
19
+
20
+
Image-to-text models are designed to analyze images and generate descriptive text based on what they see. Think of them as a combination of a camera and a writer. You provide an image as an input to the model, and the model looks at the image and identifies different elements within it, like objects, people, scenes, and even text. Based on its analysis, the model then generates a written description of the image, summarizing what it sees.
21
+
22
+
Image-to-text models excel at various use cases such as accessibility features, content organization (tagging), creating product and educational visual descriptions, and digitizing content via Optical Character Recognition (OCR). One might say image-to-text models bridge the gap between visual content and written language, making information more accessible and easier to process in various contexts.
23
+
24
+
## Prerequisites
25
+
26
+
To use image models in your application, you need:
27
+
28
+
- An Azure subscription with a valid payment method. Free or trial Azure subscriptions won't work. If you don't have an Azure subscription, create a [paid Azure account](https://azure.microsoft.com/pricing/purchase-options/pay-as-you-go) to begin.
29
+
30
+
- An [Azure AI Foundry project](create-projects.md).
31
+
32
+
- An image model deployment on Azure AI Foundry.
33
+
34
+
- This article uses a __Mistral OCR__ model deployment.
35
+
36
+
- The endpoint URL and key.
37
+
38
+
## Use image-to-text model
39
+
40
+
1. Authenticate using an API key. First, deploy the model to generate the endpoint URL and an API key to authenticate against the service. In this example, the endpoint and key are strings holding the endpoint URL and the API key. The API endpoint URL and API key can be found on the **Deployments + Endpoint** page once the model is deployed.
41
+
42
+
If you're using Bash:
43
+
44
+
```bash
45
+
export AZURE_API_KEY = "<your-api-key>"
46
+
```
47
+
48
+
If you're in PowerShell:
49
+
50
+
```powershell
51
+
$Env:AZURE_API_KEY = "<your-api-key>"
52
+
```
53
+
54
+
If you're using Windows command prompt:
55
+
56
+
```
57
+
export AZURE_API_KEY = "<your-api-key>"
58
+
```
59
+
60
+
1. Run a basic code sample. Different image models accept different data formats. In this example, _Mistral OCR 25.03_ supports only base64 encoded data; document url or image url isn't supported. Paste the following code into a shell.
61
+
62
+
```http
63
+
curl --request POST \
64
+
--url https://<your_serverless_endpoint>/v1/ocr \
65
+
--header 'Authorization: <api_key>' \
66
+
--header 'Content-Type: Application/json' \
67
+
--data '{
68
+
"model": "mistral-ocr-2503",
69
+
"document": {
70
+
"type": "document_url",
71
+
"document_name": "test",
72
+
"document_url": "data:application/pdf;base64,JVBER... <replace with your base64 encoded image data>"
Some image-to-text models only support specific data formats. Mistral OCR 25.03, for example, requires `base64 encoded image data` for their `document_url` parameter. The following table lists the supported and unsupported data formats for image models in the model catalog.
Copy file name to clipboardExpand all lines: articles/ai-services/openai/azure-government.md
+4-85Lines changed: 4 additions & 85 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -6,7 +6,7 @@ author: challenp
6
6
ms.service: azure-ai-openai
7
7
ms.topic: how-to
8
8
ms.custom: references_regions, azuregovernment
9
-
ms.date: 4/7/2025
9
+
ms.date: 5/1/2025
10
10
recommendations: false
11
11
---
12
12
@@ -33,9 +33,7 @@ The following sections show model availability by region and deployment type. Mo
33
33
* Data stored at rest remains in the designated Azure region of the resource.
34
34
* Data may be processed for inferencing in either of the two Azure Government regions.
35
35
36
-
SKU name in code: DataZoneStandard
37
-
38
-
Data zone standard deployments are available in the same Azure OpenAI resource as all other Azure OpenAI deployment types but allow you to leverage Azure global infrastructure to dynamically route traffic to the data center within the Microsoft defined data zone with the best availability for each request. Data zone standard provides higher default quotas than our Azure geography-based deployment types.
36
+
Data zone standard deployments are available in the same Azure OpenAI resource as all other Azure OpenAI deployment types but allow you to leverage Azure Government infrastructure to dynamically route traffic to the data center within the USGov data zone with the best availability for each request.
39
37
40
38
To request quota increases for these models, submit a request at [https://aka.ms/AOAIGovQuota](https://aka.ms/AOAIGovQuota). Note the following maximum quota limits allowed via that form:
41
39
@@ -51,8 +49,8 @@ To request quota increases for these models, submit a request at [https://aka.ms
51
49
| usgovarizona | ✅ | - | - | ✅ | - |
52
50
| usgovvirginia | ✅ | - | - | ✅ | - |
53
51
54
-
[NOTE]
55
-
> Provisioned Throughput Units (PTUs) are different from standard quota in Azure OpenAI and are not available by default in Azure Government. To learn more about this offering contact your Microsoft Account Team.
52
+
> [!IMPORTANT]
53
+
> Reserved Instance Provisioned Deployments are now available in Azure Government as of May 2025. Refer to [Provisioned Managed Offering in Azure Government](./concepts/gov-provisioned.md) for more details.
56
54
57
55
<br>
58
56
@@ -62,8 +60,6 @@ The following feature differences exist when comparing Azure OpenAI in Azure Gov
62
60
63
61
|Feature|Description|
64
62
|--------|--------|
65
-
| Structured Outputs | Not currently supported. |
66
-
| Reservation Based Purchases | Not currently supported. |
67
63
| Batch Deployments | Not currently supported. |
68
64
| Connect your data | Virtual network and private links are supported. Deployment to a web app or a copilot in Copilot Studio is not supported. |
69
65
| Abuse Monitoring | Not all features of Abuse Monitoring are enabled for Azure OpenAI in Azure Government. You are responsible for implementing reasonable technical and operational measures to detect and mitigate any use of the service in violation of the Product Terms. [Automated Content Classification and Filtering](./concepts/content-filter.md) remains enabled by default for Azure Government. If modified content filters are required, apply at [https://aka.ms/AOAIGovModifyContentFilter](https://aka.ms/AOAIGovModifyContentFilter)|
@@ -72,80 +68,3 @@ The following feature differences exist when comparing Azure OpenAI in Azure Gov
The following guide walks you through setting up a provisioned deployment with your Azure OpenAI Service resource in Azure Government.
80
-
81
-
### Prerequisites
82
-
83
-
- An Azure Government subscription
84
-
- An Azure OpenAI resource
85
-
- An approved quota for a provisioned deployment and purchased a commitment
86
-
87
-
### Managing provisioned throughput commitments
88
-
89
-
For Azure OpenAI in Azure Government, provisioned throughput deployments require prepurchased commitments created and managed from the **Manage Commitments** view in Azure OpenAI Studio. You can navigate to this view by selecting **Manage Commitments** from the Quota pane.
90
-
91
-
From the Manage Commitments view, you can do several things:
92
-
* Purchase new commitments or edit existing commitments.
93
-
* Monitor all commitments in your subscription.
94
-
* Identify and take action on commitments that might cause unexpected billing.
95
-
96
-
| Setting | Notes |
97
-
|---------|-------|
98
-
|**Select a resource**| Choose the resource where you create the provisioned deployment. Once you have purchased the commitment, you are unable to use the quota on another resource until the current commitment expires. |
99
-
|**Select a commitment type**| Select Provisioned. (Provisioned is equivalent to Provisioned Managed) |
100
-
|**Current uncommitted provisioned quota**| The number of PTUs currently available for you to commit to this resource. |
101
-
|**Amount to commit (PTU)**| Choose the number of PTUs you're committing to. **This number can be increased during the commitment term, but can't be decreased**. Enter values in increments of 50 for the commitment type Provisioned. |
102
-
|**Commitment tier for current period**| The commitment period is set to one month. |
103
-
|**Renewal settings**| Autorenew at current PTUs <br> Autorenew at lower PTUs <br> Do not autorenew |
104
-
105
-
> [!IMPORTANT]
106
-
> A new commitment is billed up-front for the entire term. If the renewal settings are set to auto-renew, then you will be billed again on each renewal date based on the renewal settings.
107
-
108
-
> [!IMPORTANT]
109
-
> When you add PTUs to a commitment, they will be billed immediately, at a pro-rated amount from the current date to the end of the existing commitment term. Adding PTUs does not reset the commitment term.
110
-
111
-
### Changing renewal settings
112
-
113
-
Commitment renewal settings can be changed at any time before the expiration date of your commitment.
114
-
115
-
> [!IMPORTANT]
116
-
> If you allow a commitment to expire or decrease in size such that the deployments under the resource require more PTUs than you have in your resource commitment, you will receive hourly overage charges for any excess PTUs. For example, a resource that has deployments that total 500 PTUs and a commitment for 300 PTUs will generate hourly overage charges for 200 PTUs.
117
-
118
-
### Common commitment management scenarios
119
-
120
-
**Discontinue use of provisioned throughput**
121
-
122
-
To end use of provisioned throughput and prevent hourly overage charges after commitment expiration, two steps must be taken:
123
-
124
-
1. Set the renewal policy on all commitments to *Don't autorenew*.
125
-
2. Delete the provisioned deployments using the quota.
126
-
127
-
**Move a commitment/deployment to a new resource in the same subscription/region**
128
-
129
-
It isn't possible in Azure OpenAI Studio to directly *move* a deployment or a commitment to a new resource. Instead, a new deployment needs to be created on the target resource and traffic moved to it. This process requires a new commitment purchase on the new resource. Because commitments are charged up-front for a 30-day period, it's necessary to time this move with the expiration of the original commitment to minimize overlap with the new commitment and “double-billing” during the overlap.
130
-
131
-
There are two approaches that can be taken to implement this transition.
132
-
133
-
**Option 1: No-Overlap Switchover**
134
-
135
-
This option requires some downtime, but requires no extra quota and generates no extra costs.
136
-
137
-
| Steps | Notes |
138
-
|-------|-------|
139
-
|Set the renewal policy on the existing commitment to expire| This action prevents the commitment from renewing and generating further charges |
140
-
|Before expiration of the existing commitment, delete its deployment | Downtime starts at this point and will last until the new deployment is created and traffic is moved. You can minimize the duration by timing the deletion to happen as close to the expiration date/time as possible.|
141
-
|After expiration of the existing commitment, create the commitment on the new resource|Minimize downtime by executing this step and the next step as soon after expiration as possible.|
142
-
|Create the deployment on the new resource and move traffic to it||
143
-
144
-
**Option 2: Overlapped Switchover**
145
-
146
-
This option has no downtime by having both existing and new deployments live at the same time. This method also requires having quota available to create the new deployment and generates extra costs during the overlapped deployments.
147
-
148
-
| Steps | Notes |
149
-
|-------|-------|
150
-
|Set the renewal policy on the existing commitment to expire| Doing so prevents the commitment from renewing and generating further charges.|
151
-
|Before expiration of the existing commitment:<br>1. Create the commitment on the new resource.<br>2. Create the new deployment.<br>3. Switch traffic<br>4. Delete existing deployment| Ensure you leave enough time for all steps before the existing commitment expires, otherwise overage charges will be generated (see next section) for options. |
|[Mistral-OCR-2503](https://aka.ms/aistudio/landing/mistral-ocr-2503)| image to text | - **Input:** image or PDF pages (1,000 pages, max 50MB PDF file) <br> - **Output:** text <br /> - **Tool calling:** No <br /> - **Response formats:** Text, JSON, Markdown |
269
+
|[Mistral-OCR-2503](https://aka.ms/aistudio/landing/mistral-ocr-2503)|[image to text](../ai-foundry/how-to/use-image-models.md?context=/azure/machine-learning/context/context)| - **Input:** image or PDF pages (1,000 pages, max 50MB PDF file) <br> - **Output:** text <br /> - **Tool calling:** No <br /> - **Response formats:** Text, JSON, Markdown |
270
270
|[Mistral-small-2503](https://aka.ms/aistudio/landing/mistral-small-2503)|[chat-completion (with images)](../ai-foundry/model-inference/how-to/use-chat-multi-modal.md?context=/azure/machine-learning/context/context)| - **Input:** text and images (131,072 tokens), <br> image-based tokens are 16px x 16px <br> blocks of the original images <br /> - **Output:** text (4,096 tokens) <br /> - **Tool calling:** Yes <br /> - **Response formats:** Text, JSON |
![learn-build-service-prod[bot]](https://avatars.githubusercontent.com/in/237442?v=4&size=40){kind=avatar}
0 commit comments