Skip to content

Commit 3332ac7

Browse files
committed
remove work with images section from chat completion article
1 parent b1c11c0 commit 3332ac7

File tree

5 files changed

+0
-316
lines changed

5 files changed

+0
-316
lines changed

articles/ai-foundry/model-inference/includes/use-chat-completions/csharp.md

Lines changed: 0 additions & 66 deletions
Original file line numberDiff line numberDiff line change
@@ -408,69 +408,3 @@ catch (RequestFailedException ex)
408408
409409
> [!TIP]
410410
> To learn more about how you can configure and control Azure AI content safety settings, check the [Azure AI content safety documentation](https://aka.ms/azureaicontentsafety).
411-
412-
## Use chat completions with images
413-
414-
Some models can reason across text and images and generate text completions based on both kinds of input. In this section, you explore the capabilities of Some models for vision in a chat fashion:
415-
416-
> [!IMPORTANT]
417-
> Some models support only one image for each turn in the chat conversation and only the last image is retained in context. If you add multiple images, it results in an error.
418-
419-
To see this capability, download an image and encode the information as `base64` string. The resulting data should be inside of a [data URL](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URLs):
420-
421-
422-
```csharp
423-
string imageUrl = "https://news.microsoft.com/source/wp-content/uploads/2024/04/The-Phi-3-small-language-models-with-big-potential-1-1900x1069.jpg";
424-
string imageFormat = "jpeg";
425-
HttpClient httpClient = new HttpClient();
426-
httpClient.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0");
427-
byte[] imageBytes = httpClient.GetByteArrayAsync(imageUrl).Result;
428-
string imageBase64 = Convert.ToBase64String(imageBytes);
429-
string dataUrl = $"data:image/{imageFormat};base64,{imageBase64}";
430-
```
431-
432-
Visualize the image:
433-
434-
:::image type="content" source="../../../../ai-foundry/media/how-to/sdks/small-language-models-chart-example.jpg" alt-text="A chart displaying the relative capabilities between large language models and small language models." lightbox="../../../../ai-foundry/media/how-to/sdks/small-language-models-chart-example.jpg":::
435-
436-
Now, create a chat completion request with the image:
437-
438-
439-
```csharp
440-
ChatCompletionsOptions requestOptions = new ChatCompletionsOptions()
441-
{
442-
Messages = {
443-
new ChatRequestSystemMessage("You are an AI assistant that helps people find information."),
444-
new ChatRequestUserMessage([
445-
new ChatMessageTextContentItem("Which conclusion can be extracted from the following chart?"),
446-
new ChatMessageImageContentItem(new Uri(dataUrl))
447-
]),
448-
},
449-
MaxTokens=2048,
450-
Model = "phi-3.5-vision-instruct",
451-
};
452-
453-
var response = client.Complete(requestOptions);
454-
Console.WriteLine(response.Value.Content);
455-
```
456-
457-
The response is as follows, where you can see the model's usage statistics:
458-
459-
460-
```csharp
461-
Console.WriteLine($"{response.Value.Role}: {response.Value.Content}");
462-
Console.WriteLine($"Model: {response.Value.Model}");
463-
Console.WriteLine("Usage:");
464-
Console.WriteLine($"\tPrompt tokens: {response.Value.Usage.PromptTokens}");
465-
Console.WriteLine($"\tTotal tokens: {response.Value.Usage.TotalTokens}");
466-
Console.WriteLine($"\tCompletion tokens: {response.Value.Usage.CompletionTokens}");
467-
```
468-
469-
```console
470-
ASSISTANT: The chart illustrates that larger models tend to perform better in quality, as indicated by their size in billions of parameters. However, there are exceptions to this trend, such as Phi-3-medium and Phi-3-small, which outperform smaller models in quality. This suggests that while larger models generally have an advantage, there might be other factors at play that influence a model's performance.
471-
Model: phi-3.5-vision-instruct
472-
Usage:
473-
Prompt tokens: 2380
474-
Completion tokens: 126
475-
Total tokens: 2506
476-
```

articles/ai-foundry/model-inference/includes/use-chat-completions/java.md

Lines changed: 0 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -142,28 +142,4 @@ The following example shows how to handle events when the model detects harmful
142142
> [!TIP]
143143
> To learn more about how you can configure and control Azure AI content safety settings, check the [Azure AI content safety documentation](https://aka.ms/azureaicontentsafety).
144144
145-
## Use chat completions with images
146145

147-
Some models can reason across text and images and generate text completions based on both kinds of input. In this section, you explore the capabilities of Some models for vision in a chat fashion:
148-
149-
> [!IMPORTANT]
150-
> Some models support only one image for each turn in the chat conversation and only the last image is retained in context. If you add multiple images, it results in an error.
151-
152-
To see this capability, download an image and encode the information as `base64` string. The resulting data should be inside of a [data URL](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URLs):
153-
154-
Visualize the image:
155-
156-
:::image type="content" source="../../../../ai-foundry/media/how-to/sdks/small-language-models-chart-example.jpg" alt-text="A chart displaying the relative capabilities between large language models and small language models." lightbox="../../../../ai-foundry/media/how-to/sdks/small-language-models-chart-example.jpg":::
157-
158-
Now, create a chat completion request with the image:
159-
160-
The response is as follows, where you can see the model's usage statistics:
161-
162-
```console
163-
ASSISTANT: The chart illustrates that larger models tend to perform better in quality, as indicated by their size in billions of parameters. However, there are exceptions to this trend, such as Phi-3-medium and Phi-3-small, which outperform smaller models in quality. This suggests that while larger models generally have an advantage, there might be other factors at play that influence a model's performance.
164-
Model: mistral-large-2407
165-
Usage:
166-
Prompt tokens: 2380
167-
Completion tokens: 126
168-
Total tokens: 2506
169-
```

articles/ai-foundry/model-inference/includes/use-chat-completions/javascript.md

Lines changed: 0 additions & 79 deletions
Original file line numberDiff line numberDiff line change
@@ -388,82 +388,3 @@ catch (error) {
388388
> [!TIP]
389389
> To learn more about how you can configure and control Azure AI content safety settings, check the [Azure AI content safety documentation](https://aka.ms/azureaicontentsafety).
390390
391-
## Use chat completions with images
392-
393-
Some models can reason across text and images and generate text completions based on both kinds of input. In this section, you explore the capabilities of Some models for vision in a chat fashion:
394-
395-
> [!IMPORTANT]
396-
> Some models support only one image for each turn in the chat conversation and only the last image is retained in context. If you add multiple images, it results in an error.
397-
398-
To see this capability, download an image and encode the information as `base64` string. The resulting data should be inside of a [data URL](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URLs):
399-
400-
401-
```javascript
402-
const image_url = "https://news.microsoft.com/source/wp-content/uploads/2024/04/The-Phi-3-small-language-models-with-big-potential-1-1900x1069.jpg";
403-
const image_format = "jpeg";
404-
405-
const response = await fetch(image_url, { headers: { "User-Agent": "Mozilla/5.0" } });
406-
const image_data = await response.arrayBuffer();
407-
const image_data_base64 = Buffer.from(image_data).toString("base64");
408-
const data_url = `data:image/${image_format};base64,${image_data_base64}`;
409-
```
410-
411-
Visualize the image:
412-
413-
414-
```javascript
415-
const img = document.createElement("img");
416-
img.src = data_url;
417-
document.body.appendChild(img);
418-
```
419-
420-
:::image type="content" source="../../../../ai-foundry/media/how-to/sdks/small-language-models-chart-example.jpg" alt-text="A chart displaying the relative capabilities between large language models and small language models." lightbox="../../../../ai-foundry/media/how-to/sdks/small-language-models-chart-example.jpg":::
421-
422-
Now, create a chat completion request with the image:
423-
424-
425-
```javascript
426-
var messages = [
427-
{ role: "system", content: "You are a helpful assistant that can generate responses based on images." },
428-
{ role: "user", content:
429-
[
430-
{ type: "text", text: "Which conclusion can be extracted from the following chart?" },
431-
{ type: "image_url", image:
432-
{
433-
url: data_url
434-
}
435-
}
436-
]
437-
}
438-
];
439-
440-
var response = await client.path("/chat/completions").post({
441-
body: {
442-
messages: messages,
443-
temperature: 0,
444-
top_p: 1,
445-
max_tokens: 2048,
446-
}
447-
});
448-
```
449-
450-
The response is as follows, where you can see the model's usage statistics:
451-
452-
453-
```javascript
454-
console.log(response.body.choices[0].message.role + ": " + response.body.choices[0].message.content);
455-
console.log("Model:", response.body.model);
456-
console.log("Usage:");
457-
console.log("\tPrompt tokens:", response.body.usage.prompt_tokens);
458-
console.log("\tCompletion tokens:", response.body.usage.completion_tokens);
459-
console.log("\tTotal tokens:", response.body.usage.total_tokens);
460-
```
461-
462-
```console
463-
ASSISTANT: The chart illustrates that larger models tend to perform better in quality, as indicated by their size in billions of parameters. However, there are exceptions to this trend, such as Phi-3-medium and Phi-3-small, which outperform smaller models in quality. This suggests that while larger models generally have an advantage, there might be other factors at play that influence a model's performance.
464-
Model: mistral-large-2407
465-
Usage:
466-
Prompt tokens: 2380
467-
Completion tokens: 126
468-
Total tokens: 2506
469-
```

articles/ai-foundry/model-inference/includes/use-chat-completions/python.md

Lines changed: 0 additions & 73 deletions
Original file line numberDiff line numberDiff line change
@@ -371,76 +371,3 @@ except HttpResponseError as ex:
371371
> [!TIP]
372372
> To learn more about how you can configure and control Azure AI content safety settings, check the [Azure AI content safety documentation](https://aka.ms/azureaicontentsafety).
373373
374-
## Use chat completions with images
375-
376-
Some models can reason across text and images and generate text completions based on both kinds of input. In this section, you explore the capabilities of Some models for vision in a chat fashion:
377-
378-
> [!IMPORTANT]
379-
> Some models support only one image for each turn in the chat conversation and only the last image is retained in context. If you add multiple images, it results in an error.
380-
381-
To see this capability, download an image and encode the information as `base64` string. The resulting data should be inside of a [data URL](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URLs):
382-
383-
384-
```python
385-
from urllib.request import urlopen, Request
386-
import base64
387-
388-
image_url = "https://news.microsoft.com/source/wp-content/uploads/2024/04/The-Phi-3-small-language-models-with-big-potential-1-1900x1069.jpg"
389-
image_format = "jpeg"
390-
391-
request = Request(image_url, headers={"User-Agent": "Mozilla/5.0"})
392-
image_data = base64.b64encode(urlopen(request).read()).decode("utf-8")
393-
data_url = f"data:image/{image_format};base64,{image_data}"
394-
```
395-
396-
Visualize the image:
397-
398-
399-
```python
400-
import requests
401-
import IPython.display as Disp
402-
403-
Disp.Image(requests.get(image_url).content)
404-
```
405-
406-
:::image type="content" source="../../../../ai-foundry/media/how-to/sdks/small-language-models-chart-example.jpg" alt-text="A chart displaying the relative capabilities between large language models and small language models." lightbox="../../../../ai-foundry/media/how-to/sdks/small-language-models-chart-example.jpg":::
407-
408-
Now, create a chat completion request with the image:
409-
410-
411-
```python
412-
from azure.ai.inference.models import TextContentItem, ImageContentItem, ImageUrl
413-
response = client.complete(
414-
messages=[
415-
SystemMessage("You are a helpful assistant that can generate responses based on images."),
416-
UserMessage(content=[
417-
TextContentItem(text="Which conclusion can be extracted from the following chart?"),
418-
ImageContentItem(image=ImageUrl(url=data_url))
419-
]),
420-
],
421-
temperature=0,
422-
top_p=1,
423-
max_tokens=2048,
424-
)
425-
```
426-
427-
The response is as follows, where you can see the model's usage statistics:
428-
429-
430-
```python
431-
print(f"{response.choices[0].message.role}:\n\t{response.choices[0].message.content}\n")
432-
print("Model:", response.model)
433-
print("Usage:")
434-
print("\tPrompt tokens:", response.usage.prompt_tokens)
435-
print("\tCompletion tokens:", response.usage.completion_tokens)
436-
print("\tTotal tokens:", response.usage.total_tokens)
437-
```
438-
439-
```console
440-
ASSISTANT: The chart illustrates that larger models tend to perform better in quality, as indicated by their size in billions of parameters. However, there are exceptions to this trend, such as Phi-3-medium and Phi-3-small, which outperform smaller models in quality. This suggests that while larger models generally have an advantage, there might be other factors at play that influence a model's performance.
441-
Model: mistral-large-2407
442-
Usage:
443-
Prompt tokens: 2380
444-
Completion tokens: 126
445-
Total tokens: 2506
446-
```

articles/ai-foundry/model-inference/includes/use-chat-completions/rest.md

Lines changed: 0 additions & 74 deletions
Original file line numberDiff line numberDiff line change
@@ -542,77 +542,3 @@ The following example shows how to handle events when the model detects harmful
542542

543543
> [!TIP]
544544
> To learn more about how you can configure and control Azure AI content safety settings, check the [Azure AI content safety documentation](https://aka.ms/azureaicontentsafety).
545-
546-
## Use chat completions with images
547-
548-
Some models can reason across text and images and generate text completions based on both kinds of input. In this section, you explore the capabilities of Some models for vision in a chat fashion:
549-
550-
> [!IMPORTANT]
551-
> Some models support only one image for each turn in the chat conversation and only the last image is retained in context. If you add multiple images, it results in an error.
552-
553-
To see this capability, download an image and encode the information as `base64` string. The resulting data should be inside of a [data URL](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URLs):
554-
555-
> [!TIP]
556-
> You will need to construct the data URL using a scripting or programming language. This tutorial uses [this sample image](../../../../ai-foundry/media/how-to/sdks/small-language-models-chart-example.jpg) in JPEG format. A data URL has a format as follows: `data:image/jpg;base64,0xABCDFGHIJKLMNOPQRSTUVWXYZ...`.
557-
558-
Visualize the image:
559-
560-
:::image type="content" source="../../../../ai-foundry/media/how-to/sdks/small-language-models-chart-example.jpg" alt-text="A chart displaying the relative capabilities between large language models and small language models." lightbox="../../../../ai-foundry/media/how-to/sdks/small-language-models-chart-example.jpg":::
561-
562-
Now, create a chat completion request with the image:
563-
564-
565-
```json
566-
{
567-
"model": "phi-3.5-vision-instruct",
568-
"messages": [
569-
{
570-
"role": "user",
571-
"content": [
572-
{
573-
"type": "text",
574-
"text": "Which peculiar conclusion about LLMs and SLMs can be extracted from the following chart?"
575-
},
576-
{
577-
"type": "image_url",
578-
"image_url": {
579-
"url": "data:image/jpg;base64,0xABCDFGHIJKLMNOPQRSTUVWXYZ..."
580-
}
581-
}
582-
]
583-
}
584-
],
585-
"temperature": 0,
586-
"top_p": 1,
587-
"max_tokens": 2048
588-
}
589-
```
590-
591-
The response is as follows, where you can see the model's usage statistics:
592-
593-
594-
```json
595-
{
596-
"id": "0a1234b5de6789f01gh2i345j6789klm",
597-
"object": "chat.completion",
598-
"created": 1718726686,
599-
"model": "phi-3.5-vision-instruct",
600-
"choices": [
601-
{
602-
"index": 0,
603-
"message": {
604-
"role": "assistant",
605-
"content": "The chart illustrates that larger models tend to perform better in quality, as indicated by their size in billions of parameters. However, there are exceptions to this trend, such as Phi-3-medium and Phi-3-small, which outperform smaller models in quality. This suggests that while larger models generally have an advantage, there might be other factors at play that influence a model's performance.",
606-
"tool_calls": null
607-
},
608-
"finish_reason": "stop",
609-
"logprobs": null
610-
}
611-
],
612-
"usage": {
613-
"prompt_tokens": 2380,
614-
"completion_tokens": 126,
615-
"total_tokens": 2506
616-
}
617-
}
618-
```

0 commit comments

Comments
 (0)