Integrated vectorization is not compatible with GPT-vision feature

[michbalazs]

Please provide us with the following information:

This issue is for a: (mark with an x)

- [x ] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

Following the README to setup an environment.
Set azd env set USE_GPT4V true and run azd up:
AI region: swedencentral
No error messages
Goto the url given
switch to GP4_v in the developer settings
click on one of the given questions
Error: The app encountered an error processing your request.
If you are an administrator of the app, view the full error in the logs. See aka.ms/appservice-logs for more information.
Error type: <class 'azure.core.exceptions.HttpResponseError'>

Any log messages given by the failure

(InvalidRequestParameter) Unknown field 'imageEmbedding' in vector field list.
Code: InvalidRequestParameter
Message: Unknown field 'imageEmbedding' in vector field list.
Exception Details: (UnknownField) Unknown field 'imageEmbedding' in vector field list.
Code: UnknownField
Message: Unknown field 'imageEmbedding' in vector field list.

Expected/desired behavior

OS and Version?

Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?)
windows 11

azd version?

run azd version and copy paste here.
azd version 1.6.1 (commit eba2c978b5443fdb002c95add4011d9e63c2e76f)

Versions

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.

[pamelafox]

• Was this your first time running azd up? If you ran it previously before enabling GPT-4-V, you'd need to delete the index, so that it could be recreated with the imageEmbedding field.
• Is it possible that you enabled the Integrated Vectorization feature? That is not compatible with GPT-4-V, so it would need to be set to false.

[mmarahrens]

Hi @pamelafox,

We have the same problem. We had to activate the integrated vectorization function in order to successfully build the index with custom documents. Without the feature we are getting the following error:

´´´bash
openai.BadRequestError: Error code: 400 - {'error': {'message': "This model's maximum context length is 8192 tokens, however you requested 10515 tokens (10515 in your prompt; 0 for the completion). Please reduce your prompt; or completion length.", 'type': 'invalid_request_error', 'param': None, 'code': None}}
´´´

I see in your second point, that the feature is not compatible with gpt-4v. Anything else can we do to overcome this issues or do we have to wait for compatibility? :)

Best regards,
Mirco

[pamelafox]

@mmarahrens That looks like a different error. Have you checked to see what in your request is causing it to go over the token count? The prompt, the search results, the conversation history, etc? Requesting too many or too large search results can cause that error.

[mmarahrens]

Ah, I was unprecise in my description. The error I describe in my comment is an upstream issue, which occurs at deployment of the envrionment with 'azd up', to be more precise at index building, when we had an additional custom pdf-document in the data folder. We overcame this issue by enabling the preview feature integrated vectorization function.

After activation the deployment runs smoothly, but now we are facing

Error: The app encountered an error processing your request.
If you are an administrator of the app, view the full error in the logs. See aka.ms/appservice-logs for more information.
Error type: <class 'azure.core.exceptions.HttpResponseError'>

if activating gpt-4 in the web UI and querying one of the default examples. The custom document is with 6.5M larger then the other example documents.

[pamelafox]

If you are using gpt-4-vision, you may be experiencing the issues I'm working on in this post: #1355
We weren't doing chunking when vision was enabled, but I'm bringing back chunking.
You may be sending too much data to the model due to the text not being chunked. I can't tell from that error, you'll need to consult the logs for the full traceback:
https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/docs/appservice.md

[cforce]

• Integrated Vectorization

so its not possible to have
USE_FEATURE_INT_VECTORIZATION="true" && USE_GPT4V="true"

Then the option VISION - shall be greyed when "USE_FEATURE_INT_VECTORIZATION=true" in the expert settings when its like this, shoulnd't?

[pamelafox]

Those features are not currently compatible, no. We have a note in the guide about that. I think we may also have a warning when you run prepdocs. It'd be a fairly large change to grey out in the settings, so not sure we'll go that far, but I think whether there's more we can do.

[cforce]

So basically you would need to create a skillset that uses chunking on text as of now already but use another skillset which extract images to be after indexed with vision model
Currently we use "split_skill(Microsoft.Skills.Text.SplitSkill)" and "embedding_skill (Microsoft.Skills.Text.AzureOpenAIEmbeddingSkill) in prepdocs.sh

In context of i found additional skillset that looks promising to support the Vision approach as well with Integrated vectorization

1.) For extracting text and normalized images from documents we use DocumentExtractionSkill
https://learn.microsoft.com/en-us/azure/search/cognitive-search-skill-document-extraction

{
  "@odata.type": "#Microsoft.Skills.Util.DocumentExtractionSkill",
  "parsingMode": "",
  "dataToExtract": "",
  "configuration": "",
  "name": "",
  "description": "",
  "context": "",
  "inputs": [
    {
      "name": "file_data",
      "source": ""
    }
  ],
  "outputs": [
    {
      "name": "content",
      "targetName": "content"
    },
    {
      "name": "normalized_images",
      "targetName": "normalized_images"
    }
  ]
}

With Microsoft.Skills.Util.DocumentExtractionSkillskill we can extract text and images dedicated in separate buckets and process each of them in their dedicated model gtp4o or gpt4vision. Setting the imageAction parameter on your indexer definition to a value other than none creates an array of images that follows the required convention for input to this skill if passed individually (that is, /document/normalized_images/). The output of /document/normalized_images/" we feed as input for Microsoft.Skills.Vision.VectorizeSkill which will then create vectors for vision
https://learn.microsoft.com/en-us/azure/search/cognitive-search-skill-image-analysis
The Microsoft.Skills.Vision.VectorizeSkill uses Azure AI Vision's multimodal embeddings API to generate embeddings for image or text input.

Text: Microsoft.Skills.Util.DocumentExtractionSkill->Microsoft.Skills.Text.SplitSkill-> Microsoft.Skills.Text.AzureOpenAIEmbeddingSkill -> gtp4o:

• Microsoft.Skills.Text.SplitSkill.input.document.content=Microsoft.Skills.Util.DocumentExtractionSkill.DocumentExtractionSkill.ouput.document.content
• Microsoft.Skills.Text.AzureOpenAIEmbeddingSkill.document.pages= Microsoft.Skills.Text.SplitSkill.ouput.document.pages

Images: Microsoft.Skills.Util.DocumentExtractionSkill->Microsoft.Skills.Vision.VectorizeSkill -> gpt4vision:

• Microsoft.Skills.Vision.VectorizeSkill.input.document.normalized_images=Microsoft.Skills.Util.DocumentExtractionSkill.DocumentExtractionSkill.output.document.normalized_images

To modify the existing IntegratedVectorizerStrategy to incorporate both text and image extraction and vectorization using the proposed approach, we can add the following steps:

• DocumentExtractionSkill: This skill will extract both the textual content and normalized images from the input document.
  Text Processing: The text will be chunked with SplitSkill and then vectorized using AzureOpenAIEmbeddingSkill.
• Image Processing: The extracted images will be passed to the Microsoft.Skills.Vision.VectorizeSkill to generate image embeddings.
• Skillset Wiring: We'll map the outputs of the DocumentExtractionSkill to both the text vectorization path and the image vectorization path.
• Embedding Field Reuse: Both the text and image embeddings can be stored under the same InputFieldMappingEntry(name="embedding"..) if they are processed separately in the skillset and then combined into the same field for indexing.
• Handling Multiple Embeddings: The system must ensure that when text embeddings are generated, they are placed in the /document/pages//embedding field, and when image embeddings are generated, they are placed in the /document/normalized_images//embedding field. You can use the same field name (embedding) as long as they are processed separately and mapped correctly.

[pamelafox]

We have made a custom skillset that runs our own code instead, mirroring the local ingestion in Azure Functions. We will deprecate the previous integrated vectorization and recommend the new skillset approach, for maximal compatibility with the repo features. Please try it out and raise any issues you run into. Thanks!
https://github.com/Azure-Samples/azure-search-openai-demo/releases/tag/2025-11-12