Skip to content

Commit 865d1a8

Browse files
santiagxfsantiagxf
committed
feat: image embeddings
1 parent 59fd620 commit 865d1a8

File tree

8 files changed

+518
-0
lines changed

8 files changed

+518
-0
lines changed

‎articles/ai-foundry/model-inference/how-to/use-image-embeddings.md

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
---
2+
title: How to generate embeddings with Azure AI model inference service
3+
titleSuffix: Azure AI Foundry
4+
description: Learn how to generate embeddings with Azure AI model inference
5+
manager: scottpolly
6+
author: msakande
7+
reviewer: santiagxf
8+
ms.service: azure-ai-model-inference
9+
ms.topic: how-to
10+
ms.date: 12/23/2024
11+
ms.author: mopeakande
12+
ms.reviewer: fasantia
13+
ms.custom: generated
14+
zone_pivot_groups: azure-ai-inference-samples
15+
---
16+
17+
# How to generate image embeddings with Azure AI model inference
18+
19+
20+
::: zone pivot="programming-language-python"
21+
22+
[!INCLUDE [python](../includes/use-image-embeddings/python.md)]
23+
::: zone-end
24+
25+
26+
::: zone pivot="programming-language-javascript"
27+
28+
[!INCLUDE [javascript](../includes/use-image-embeddings/javascript.md)]
29+
::: zone-end
30+
31+
32+
::: zone pivot="programming-language-java"
33+
34+
[!INCLUDE [java](../includes/use-image-embeddings/java.md)]
35+
::: zone-end
36+
37+
38+
::: zone pivot="programming-language-csharp"
39+
40+
[!INCLUDE [csharp](../includes/use-image-embeddings/csharp.md)]
41+
::: zone-end
42+
43+
44+
::: zone pivot="programming-language-rest"
45+
46+
[!INCLUDE [rest](../includes/use-image-embeddings/rest.md)]
47+
::: zone-end
48+
49+
## Related content
50+
51+
* [Use embeddings models](use-embeddings.md)
52+
* [Azure AI Model Inference API](../../../ai-studio/reference/reference-model-inference-api.md)

‎articles/ai-foundry/model-inference/includes/use-image-embeddings/csharp.md

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
---
2+
title: How to generate image embeddings with Azure AI model inference
3+
titleSuffix: Azure AI Foundry
4+
description: Learn how to generate embeddings with Azure AI model inference
5+
manager: scottpolly
6+
author: msakande
7+
reviewer: santiagxf
8+
ms.service: azure-ai-model-inference
9+
ms.topic: how-to
10+
ms.date: 01/22/2025
11+
ms.author: mopeakande
12+
ms.reviewer: fasantia
13+
ms.custom: generated
14+
zone_pivot_groups: azure-ai-inference-samples
15+
---
16+
17+
> [!NOTE]
18+
> Using image embeddings is only supported using Python, JavaScript, or REST requests.

‎articles/ai-foundry/model-inference/includes/use-image-embeddings/java.md

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
---
2+
title: How to generate image embeddings with Azure AI model inference
3+
titleSuffix: Azure AI Foundry
4+
description: Learn how to generate embeddings with Azure AI model inference
5+
manager: scottpolly
6+
author: msakande
7+
reviewer: santiagxf
8+
ms.service: azure-ai-model-inference
9+
ms.topic: how-to
10+
ms.date: 01/22/2025
11+
ms.author: mopeakande
12+
ms.reviewer: fasantia
13+
ms.custom: generated
14+
zone_pivot_groups: azure-ai-inference-samples
15+
---
16+
17+
> [!NOTE]
18+
> Using image embeddings is only supported using Python, JavaScript, or REST requests.

‎articles/ai-foundry/model-inference/includes/use-image-embeddings/javascript.md

Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
---
2+
title: How to generate image embeddings with Azure AI model inference
3+
titleSuffix: Azure AI Foundry
4+
description: Learn how to generate embeddings with Azure AI model inference
5+
manager: scottpolly
6+
author: msakande
7+
reviewer: santiagxf
8+
ms.service: azure-ai-model-inference
9+
ms.topic: how-to
10+
ms.date: 01/22/2025
11+
ms.author: mopeakande
12+
ms.reviewer: fasantia
13+
ms.custom: generated
14+
zone_pivot_groups: azure-ai-inference-samples
15+
---
16+
17+
[!INCLUDE [Feature preview](~/reusable-content/ce-skilling/azure/includes/ai-studio/includes/feature-preview.md)]
18+
19+
This article explains how to use image embeddings API with models deployed to Azure AI model inference in Azure AI Foundry.
20+
21+
## Prerequisites
22+
23+
To use embedding models in you application, you need:
24+
25+
[!INCLUDE [how-to-prerequisites](../how-to-prerequisites.md)]
26+
27+
* An image embeddings model deployment. If you don't have one read [Add and configure models to Azure AI services](../../how-to/create-model-deployments.md) to add an embeddings model to your resource.
28+
29+
* This example uses `Cohere-embed-v3-english` from Cohere.
30+
31+
* Install the Azure Inference library for JavaScript with the following command:
32+
33+
```bash
34+
npm install @azure-rest/ai-inference
35+
```
36+
37+
> [!TIP]
38+
> Read more about the [Azure AI inference package and reference](https://aka.ms/azsdk/azure-ai-inference/javascript/reference).
39+
40+
## Use embeddings
41+
42+
First, create the client to consume the model. The following code uses an endpoint URL and key that are stored in environment variables.
43+
44+
45+
```javascript
46+
import ModelClient from "@azure-rest/ai-inference";
47+
import { isUnexpected } from "@azure-rest/ai-inference";
48+
import { AzureKeyCredential } from "@azure/core-auth";
49+
50+
const client = new ModelClient(
51+
process.env.AZURE_INFERENCE_ENDPOINT,
52+
new AzureKeyCredential(process.env.AZURE_INFERENCE_CREDENTIAL),
53+
"text-embedding-3-small"
54+
);
55+
```
56+
57+
If you have configured the resource to with **Microsoft Entra ID** support, you can use the following code snippet to create a client.
58+
59+
### Create embeddings
60+
61+
To create image embeddings, you need to pass the image data as part of your request. Image data should be in PNG format and encoded as base64.
62+
63+
```javascript
64+
var image_path = "sample1.png";
65+
var image_data = fs.readFileSync(image_path);
66+
var image_data_base64 = Buffer.from(image_data).toString("base64");
67+
68+
var response = await client.path("/images/embeddings").post({
69+
body: {
70+
input: [ { image: image_data_base64 } ],
71+
}
72+
});
73+
```
74+
75+
> [!TIP]
76+
> When creating a request, take into account the token's input limit for the model. If you need to embed larger portions of text, you would need a chunking strategy.
77+
78+
The response is as follows, where you can see the model's usage statistics:
79+
80+
81+
```javascript
82+
if (isUnexpected(response)) {
83+
throw response.body.error;
84+
}
85+
86+
console.log(response.embedding);
87+
console.log(response.body.model);
88+
console.log(response.body.usage);
89+
```
90+
91+
> [!IMPORTANT]
92+
> Computing embeddings in batches may not be supported for all the models. This is the case of the `cohere-embed-v3` model. In this case, you need to send one image at a time.
93+
94+
#### Embedding images and text pairs
95+
96+
Some models can generate embeddings from images and text pairs. In this case, you can use the `image` and `text` fields in the request to pass the image and text to the model. The following example shows how to create embeddings for images and text pairs:
97+
98+
99+
```javascript
100+
var image_path = "sample1.png";
101+
var image_data = fs.readFileSync(image_path);
102+
var image_data_base64 = Buffer.from(image_data).toString("base64");
103+
104+
var response = await client.path("images/embeddings").post({
105+
body: {
106+
input: [
107+
{
108+
text: "A cute baby sea otter",
109+
image: image_data_base64
110+
}
111+
]
112+
}
113+
});
114+
```
115+
116+
#### Create different types of embeddings
117+
118+
Some models can generate multiple embeddings for the same input depending on how you plan to use them. This capability allows you to retrieve more accurate embeddings for RAG patterns.
119+
120+
The following example shows how to create embeddings that are used to create an embedding for a document that will be stored in a vector database:
121+
122+
123+
```javascript
124+
var response = await client.path("/embeddings").post({
125+
body: {
126+
input: [ { image: image_data_base64 } ],
127+
input_type: "document",
128+
}
129+
});
130+
```
131+
132+
When you work on a query to retrieve such a document, you can use the following code snippet to create the embeddings for the query and maximize the retrieval performance.
133+
134+
135+
```javascript
136+
var response = await client.path("/embeddings").post({
137+
body: {
138+
input: [ { image: image_data_base64 } ],
139+
input_type: "query",
140+
}
141+
});
142+
```
143+
144+
Notice that not all the embedding models support indicating the input type in the request and on those cases a 422 error is returned.

‎articles/ai-foundry/model-inference/includes/use-image-embeddings/python.md

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,132 @@
1+
---
2+
title: How to generate image embeddings with Azure AI model inference
3+
titleSuffix: Azure AI Foundry
4+
description: Learn how to generate embeddings with Azure AI model inference
5+
manager: scottpolly
6+
author: msakande
7+
reviewer: santiagxf
8+
ms.service: azure-ai-model-inference
9+
ms.topic: how-to
10+
ms.date: 01/22/2025
11+
ms.author: mopeakande
12+
ms.reviewer: fasantia
13+
ms.custom: generated
14+
zone_pivot_groups: azure-ai-inference-samples
15+
---
16+
17+
[!INCLUDE [Feature preview](~/reusable-content/ce-skilling/azure/includes/ai-studio/includes/feature-preview.md)]
18+
19+
This article explains how to use image embeddings API with models deployed to Azure AI model inference in Azure AI Foundry.
20+
21+
## Prerequisites
22+
23+
To use embedding models in you application, you need:
24+
25+
[!INCLUDE [how-to-prerequisites](../how-to-prerequisites.md)]
26+
27+
* An image embeddings model deployment. If you don't have one read [Add and configure models to Azure AI services](../../how-to/create-model-deployments.md) to add an embeddings model to your resource.
28+
29+
* This example uses `Cohere-embed-v3-english` from Cohere.
30+
31+
* Install the Azure AI inference package with the following command:
32+
33+
```bash
34+
pip install -U azure-ai-inference
35+
```
36+
37+
> [!TIP]
38+
> Read more about the [Azure AI inference package and reference](https://aka.ms/azsdk/azure-ai-inference/python/reference).
39+
40+
## Use embeddings
41+
42+
First, create the client to consume the model. The following code uses an endpoint URL and key that are stored in environment variables.
43+
44+
45+
```python
46+
import os
47+
from azure.ai.inference import ImageEmbeddingsClient
48+
from azure.core.credentials import AzureKeyCredential
49+
50+
model = ImageEmbeddingsClient(
51+
endpoint=os.environ["AZURE_INFERENCE_ENDPOINT"],
52+
credential=AzureKeyCredential(os.environ["AZURE_INFERENCE_CREDENTIAL"]),
53+
model="text-embedding-3-small"
54+
)
55+
```
56+
57+
If you have configured the resource to with **Microsoft Entra ID** support, you can use the following code snippet to create a client.
58+
59+
### Create embeddings
60+
61+
To create image embeddings, you need to pass the image data as part of your request. Image data should be in PNG format and encoded as base64.
62+
63+
```python
64+
from azure.ai.inference.models import ImageEmbeddingInput
65+
66+
image_input= ImageEmbeddingInput.load(image_file="sample1.png", image_format="png")
67+
response = model.embed(
68+
input=[ image_input ],
69+
)
70+
```
71+
72+
> [!TIP]
73+
> When creating a request, take into account the token's input limit for the model. If you need to embed larger portions of text, you would need a chunking strategy.
74+
75+
The response is as follows, where you can see the model's usage statistics:
76+
77+
78+
```python
79+
import numpy as np
80+
81+
for embed in response.data:
82+
print("Embeding of size:", np.asarray(embed.embedding).shape)
83+
84+
print("Model:", response.model)
85+
print("Usage:", response.usage)
86+
```
87+
88+
> [!IMPORTANT]
89+
> Computing embeddings in batches may not be supported for all the models. This is the case of the `cohere-embed-v3` model. In this case, you need to send one image at a time.
90+
91+
#### Embedding images and text pairs
92+
93+
Some models can generate embeddings from images and text pairs. In this case, you can use the `image` and `text` fields in the request to pass the image and text to the model. The following example shows how to create embeddings for images and text pairs:
94+
95+
96+
```python
97+
text_image_input= ImageEmbeddingInput.load(image_file="sample1.png", image_format="png")
98+
text_image_input.text = "A cute baby sea otter"
99+
response = model.embed(
100+
input=[ text_image_input ],
101+
)
102+
```
103+
104+
#### Create different types of embeddings
105+
106+
Some models can generate multiple embeddings for the same input depending on how you plan to use them. This capability allows you to retrieve more accurate embeddings for RAG patterns.
107+
108+
The following example shows how to create embeddings that are used to create an embedding for a document that will be stored in a vector database:
109+
110+
111+
```python
112+
from azure.ai.inference.models import EmbeddingInputType
113+
114+
response = model.embed(
115+
input=[ image_input ],
116+
input_type=EmbeddingInputType.DOCUMENT,
117+
)
118+
```
119+
120+
When you work on a query to retrieve such a document, you can use the following code snippet to create the embeddings for the query and maximize the retrieval performance.
121+
122+
123+
```python
124+
from azure.ai.inference.models import EmbeddingInputType
125+
126+
response = model.embed(
127+
input=[ image_input ],
128+
input_type=EmbeddingInputType.QUERY,
129+
)
130+
```
131+
132+
Notice that not all the embedding models support indicating the input type in the request and on those cases a 422 error is returned.

0 commit comments

Comments
 (0)