|
| 1 | +--- |
| 2 | +title: How to generate image embeddings with Azure AI model inference |
| 3 | +titleSuffix: Azure AI Foundry |
| 4 | +description: Learn how to generate embeddings with Azure AI model inference |
| 5 | +manager: scottpolly |
| 6 | +author: msakande |
| 7 | +reviewer: santiagxf |
| 8 | +ms.service: azure-ai-model-inference |
| 9 | +ms.topic: how-to |
| 10 | +ms.date: 01/22/2025 |
| 11 | +ms.author: mopeakande |
| 12 | +ms.reviewer: fasantia |
| 13 | +ms.custom: generated |
| 14 | +zone_pivot_groups: azure-ai-inference-samples |
| 15 | +--- |
| 16 | + |
| 17 | +[!INCLUDE [Feature preview](~/reusable-content/ce-skilling/azure/includes/ai-studio/includes/feature-preview.md)] |
| 18 | + |
| 19 | +This article explains how to use image embeddings API with models deployed to Azure AI model inference in Azure AI Foundry. |
| 20 | + |
| 21 | +## Prerequisites |
| 22 | + |
| 23 | +To use embedding models in your application, you need: |
| 24 | + |
| 25 | +[!INCLUDE [how-to-prerequisites](../how-to-prerequisites.md)] |
| 26 | + |
| 27 | +* An image embeddings model deployment. If you don't have one read [Add and configure models to Azure AI services](../../how-to/create-model-deployments.md) to add an embeddings model to your resource. |
| 28 | + |
| 29 | + * This example uses `Cohere-embed-v3-english` from Cohere. |
| 30 | + |
| 31 | +* Install the Azure Inference library for JavaScript with the following command: |
| 32 | + |
| 33 | + ```bash |
| 34 | + npm install @azure-rest/ai-inference |
| 35 | + ``` |
| 36 | + |
| 37 | + > [!TIP] |
| 38 | + > Read more about the [Azure AI inference package and reference](https://aka.ms/azsdk/azure-ai-inference/javascript/reference). |
| 39 | +
|
| 40 | +## Use embeddings |
| 41 | + |
| 42 | +First, create the client to consume the model. The following code uses an endpoint URL and key that are stored in environment variables. |
| 43 | + |
| 44 | + |
| 45 | +```javascript |
| 46 | +import ModelClient from "@azure-rest/ai-inference"; |
| 47 | +import { isUnexpected } from "@azure-rest/ai-inference"; |
| 48 | +import { AzureKeyCredential } from "@azure/core-auth"; |
| 49 | + |
| 50 | +const client = new ModelClient( |
| 51 | + process.env.AZURE_INFERENCE_ENDPOINT, |
| 52 | + new AzureKeyCredential(process.env.AZURE_INFERENCE_CREDENTIAL), |
| 53 | + "Cohere-embed-v3-english" |
| 54 | +); |
| 55 | +``` |
| 56 | + |
| 57 | +If you configured the resource to with **Microsoft Entra ID** support, you can use the following code snippet to create a client. |
| 58 | + |
| 59 | +```javascript |
| 60 | +import ModelClient from "@azure-rest/ai-inference"; |
| 61 | +import { isUnexpected } from "@azure-rest/ai-inference"; |
| 62 | +import { DefaultAzureCredential } from "@azure/identity"; |
| 63 | + |
| 64 | +const client = new ModelClient( |
| 65 | + process.env.AZURE_INFERENCE_ENDPOINT, |
| 66 | + new DefaultAzureCredential(), |
| 67 | + "Cohere-embed-v3-english" |
| 68 | +); |
| 69 | +``` |
| 70 | + |
| 71 | +### Create embeddings |
| 72 | + |
| 73 | +To create image embeddings, you need to pass the image data as part of your request. Image data should be in PNG format and encoded as base64. |
| 74 | + |
| 75 | +```javascript |
| 76 | +var image_path = "sample1.png"; |
| 77 | +var image_data = fs.readFileSync(image_path); |
| 78 | +var image_data_base64 = Buffer.from(image_data).toString("base64"); |
| 79 | + |
| 80 | +var response = await client.path("/images/embeddings").post({ |
| 81 | + body: { |
| 82 | + input: [ { image: image_data_base64 } ], |
| 83 | + } |
| 84 | +}); |
| 85 | +``` |
| 86 | + |
| 87 | +> [!TIP] |
| 88 | +> When creating a request, take into account the token's input limit for the model. If you need to embed larger portions of text, you would need a chunking strategy. |
| 89 | +
|
| 90 | +The response is as follows, where you can see the model's usage statistics: |
| 91 | + |
| 92 | + |
| 93 | +```javascript |
| 94 | +if (isUnexpected(response)) { |
| 95 | + throw response.body.error; |
| 96 | +} |
| 97 | + |
| 98 | +console.log(response.embedding); |
| 99 | +console.log(response.body.model); |
| 100 | +console.log(response.body.usage); |
| 101 | +``` |
| 102 | + |
| 103 | +> [!IMPORTANT] |
| 104 | +> Computing embeddings in batches may not be supported for all the models. For example, for `cohere-embed-v3` model, you need to send one image at a time. |
| 105 | +
|
| 106 | +#### Embedding images and text pairs |
| 107 | + |
| 108 | +Some models can generate embeddings from images and text pairs. In this case, you can use the `image` and `text` fields in the request to pass the image and text to the model. The following example shows how to create embeddings for images and text pairs: |
| 109 | + |
| 110 | + |
| 111 | +```javascript |
| 112 | +var image_path = "sample1.png"; |
| 113 | +var image_data = fs.readFileSync(image_path); |
| 114 | +var image_data_base64 = Buffer.from(image_data).toString("base64"); |
| 115 | + |
| 116 | +var response = await client.path("images/embeddings").post({ |
| 117 | + body: { |
| 118 | + input: [ |
| 119 | + { |
| 120 | + text: "A cute baby sea otter", |
| 121 | + image: image_data_base64 |
| 122 | + } |
| 123 | + ] |
| 124 | + } |
| 125 | +}); |
| 126 | +``` |
| 127 | + |
| 128 | +#### Create different types of embeddings |
| 129 | + |
| 130 | +Some models can generate multiple embeddings for the same input depending on how you plan to use them. This capability allows you to retrieve more accurate embeddings for RAG patterns. |
| 131 | + |
| 132 | +The following example shows how to create embeddings that are used to create an embedding for a document that will be stored in a vector database: |
| 133 | + |
| 134 | + |
| 135 | +```javascript |
| 136 | +var response = await client.path("/embeddings").post({ |
| 137 | + body: { |
| 138 | + input: [ { image: image_data_base64 } ], |
| 139 | + input_type: "document", |
| 140 | + } |
| 141 | +}); |
| 142 | +``` |
| 143 | + |
| 144 | +When you work on a query to retrieve such a document, you can use the following code snippet to create the embeddings for the query and maximize the retrieval performance. |
| 145 | + |
| 146 | + |
| 147 | +```javascript |
| 148 | +var response = await client.path("/embeddings").post({ |
| 149 | + body: { |
| 150 | + input: [ { image: image_data_base64 } ], |
| 151 | + input_type: "query", |
| 152 | + } |
| 153 | +}); |
| 154 | +``` |
| 155 | + |
| 156 | +Notice that not all the embedding models support indicating the input type in the request and on those cases a 422 error is returned. |
0 commit comments