|
| 1 | +--- |
| 2 | +title: GPT-4 Turbo with Vision concepts |
| 3 | +titleSuffix: Azure OpenAI |
| 4 | +description: Learn about vision chats enabled by GPT-4 Turbo with Vision. |
| 5 | +author: PatrickFarley |
| 6 | +ms.author: pafarley |
| 7 | +ms.service: azure-ai-openai |
| 8 | +ms.topic: conceptual |
| 9 | +ms.date: 01/02/2024 |
| 10 | +manager: nitinme |
| 11 | +keywords: |
| 12 | +--- |
| 13 | + |
| 14 | +# GPT-4 Turbo with Vision concepts |
| 15 | + |
| 16 | +GPT-4 Turbo with Vision is a large multimodal model (LMM) developed by OpenAI that can analyze images and provide textual responses to questions about them. It incorporates both natural language processing and visual understanding. This guide provides details on the capabilities and limitations of GPT-4 Turbo with Vision. |
| 17 | + |
| 18 | +To try out GPT-4 Turbo with Vision, see the [quickstart](/azure/ai-services/openai/gpt-v-quickstart). |
| 19 | + |
| 20 | +## Chats with vision |
| 21 | + |
| 22 | +The GPT-4 Turbo with Vision model answers general questions about what's present in the images or videos you upload. |
| 23 | + |
| 24 | + |
| 25 | +## Enhancements |
| 26 | + |
| 27 | +Enhancements let you incorporate other Azure AI services (such as Azure AI Vision) to add new functionality to the chat-with-vision experience. |
| 28 | + |
| 29 | +**Object grounding**: Azure AI Vision complements GPT-4 Turbo with Vision’s text response by identifying and locating salient objects in the input images. This lets the chat model give more accurate and detailed responses about the contents of the image. |
| 30 | + |
| 31 | +:::image type="content" source="../media/concepts/gpt-v/object-grounding.png" alt-text="Screenshot of an image with object grounding applied. Objects have bounding boxes with labels."::: |
| 32 | + |
| 33 | +:::image type="content" source="../media/concepts/gpt-v/object-grounding-response.png" alt-text="Screenshot of a chat response to an image prompt about an outfit. The response is an itemized list of clothing items seen in the image."::: |
| 34 | + |
| 35 | +**Optical Character Recognition (OCR)**: Azure AI Vision complements GPT-4 Turbo with Vision by providing high-quality OCR results as supplementary information to the chat model. It allows the model to produce higher quality responses for images with dense text, transformed images, and numbers-heavy financial documents, and increases the variety of languages the model can recognize in text. |
| 36 | + |
| 37 | +:::image type="content" source="../media/concepts/gpt-v/receipts.png" alt-text="Photo of several receipts."::: |
| 38 | + |
| 39 | +:::image type="content" source="../media/concepts/gpt-v/ocr-response.png" alt-text="Screenshot of the JSON response of an OCR call."::: |
| 40 | + |
| 41 | +**Video prompt**: The **video prompt** enhancement lets you use video clips as input for AI chat, enabling the model to generate summaries and answers about video content. It uses Azure AI Vision Video Retrieval to sample a set of frames from a video and create a transcript of the speech in the video. |
| 42 | + |
| 43 | +In order to use the video prompt enhancement, you need both an Azure AI Vision resource and an Azure Video Indexer resource, in addition to your Azure OpenAI resource. |
| 44 | + |
| 45 | +> [!VIDEO https://www.microsoft.com/en-us/videoplayer/embed/RW1eHRf] |
| 46 | +
|
| 47 | + |
| 48 | +## Special pricing information |
| 49 | + |
| 50 | +> [!IMPORTANT] |
| 51 | +> Pricing details are subject to change in the future. |
| 52 | +
|
| 53 | +GPT-4 Turbo with Vision accrues charges like other Azure OpenAI chat models. You pay a per-token rate for the prompts and completions, detailed on the [Pricing page](/pricing/details/cognitive-services/openai-service/). The base charges and additional features are outlined here: |
| 54 | + |
| 55 | +Base Pricing for GPT-4 Turbo with Vision is: |
| 56 | +- Input: $0.01 per 1000 tokens |
| 57 | +- Output: $0.03 per 1000 tokens |
| 58 | + |
| 59 | +See the [Tokens section of the overview](/azure/ai-services/openai/overview#tokens) for information on how text and images translate to tokens. |
| 60 | + |
| 61 | +Additionally, if you use video prompt integration with the Video Retrieval add-on, it accrues other costs: |
| 62 | +- Ingestion: $0.05 per minute of video |
| 63 | +- Transactions: $0.25 per 1000 queries of the Video Retrieval index |
| 64 | + |
| 65 | +Processing videos involves the use of extra tokens to identify key frames for analysis. The number of these additional tokens will be roughly equivalent to the sum of the tokens in the text input, plus 700 tokens. |
| 66 | + |
| 67 | +### Example price calculation |
| 68 | + |
| 69 | +> [!IMPORTANT] |
| 70 | +> The following content is an example only, and prices are subject to change in the future. |
| 71 | +
|
| 72 | +For a typical use case, take a 3-minute video with a 100-token prompt input. The video has a transcript that's 100 tokens long, and when the service processes the prompt, it generates 100 tokens of output. The pricing for this transaction would be: |
| 73 | + |
| 74 | +| Item | Detail | Total Cost | |
| 75 | +|-----------------|-----------------|--------------| |
| 76 | +| GPT-4 Turbo with Vision input tokens | 100 text tokens | $0.001 | |
| 77 | +| Additional Cost to identify frames | 100 input tokens + 700 tokens + 1 Video Retrieval transaction | $0.00825 | |
| 78 | +| Image Inputs and Transcript Input | 20 images (85 tokens each) + 100 transcript tokens | $0.018 | |
| 79 | +| Output Tokens | 100 tokens (assumed) | $0.003 | |
| 80 | +| **Total Cost** | | **$0.03025** | |
| 81 | + |
| 82 | +Additionally, there's a one-time indexing cost of $0.15 to generate the Video Retrieval index for this 3-minute video. This index can be reused across any number of Video Retrieval and GPT-4 Turbo with Vision API calls. |
| 83 | + |
| 84 | +## Limitations |
| 85 | + |
| 86 | +This section describes the limitations of GPT-4 Turbo with Vision. |
| 87 | + |
| 88 | +### Image support |
| 89 | + |
| 90 | +- **Limitation on image enhancements per chat session**: Enhancements cannot be applied to multiple images within a single chat call. |
| 91 | +- **Maximum input image size**: The maximum size for input images is restricted to 20 MB. |
| 92 | +- **Object grounding in enhancement API**: When the enhancement API is used for object grounding, and the model detects duplicates of an object, it will generate one bounding box and label for all the duplicates instead of separate ones for each. |
| 93 | +- **Low resolution accuracy**: When images are analyzed using the "low resolution" setting, it allows for faster responses and uses fewer input tokens for certain use cases. However, this could impact the accuracy of object and text recognition within the image. |
| 94 | +- **Image chat restriction**: When you upload images in Azure OpenAI Studio or the API, there is a limit of 10 images per chat call. |
| 95 | + |
| 96 | +### Video support |
| 97 | + |
| 98 | +- **Low resolution**: Video frames are analyzed using GPT-4 Turbo with Vision's "low resolution" setting, which may affect the accuracy of small object and text recognition in the video. |
| 99 | +- **Video file limits**: Both MP4 and MOV file types are supported. In Azure OpenAI Studio, videos must be less than 3 minutes long. When you use the API there is no such limitation. |
| 100 | +- **Prompt limits**: Video prompts only contain one video and no images. In Azure OpenAI Studio, you can clear the session to try another video or images. |
| 101 | +- **Limited frame selection**: The service selects 20 frames from the entire video, which might not capture all the critical moments or details. Frame selection can be approximately evenly spread through the video or focused by a specific video retrieval query, depending on the prompt. |
| 102 | +- **Language support**: The service primarily supports English for grounding with transcripts. Transcripts don't provide accurate information on lyrics in songs. |
| 103 | + |
| 104 | +## Next steps |
| 105 | + |
| 106 | +- Get started using GPT-4 Turbo with Vision by following the [quickstart](/azure/ai-services/openai/gpt-v-quickstart). |
| 107 | +- For a more in-depth look at the APIs, and to use video prompts in chat, follow the [how-to guide](../how-to/gpt-with-vision.md). |
| 108 | +- See the [completions and embeddings API reference](../reference.md) |
0 commit comments