VLM OCR processing fails with 500 Internal Server Error from Azure OpenAI

[kiendoantrung]

Bug Report

Description

The document processing pipeline fails during the document conversion step due to an unhandled HTTPError. The error occurs when calling the Azure OpenAI GPT-4o API (/chat/completions). The server responds with a 500 Internal Server Error.

---

Steps to Reproduce

1. Trigger the document processing pipeline with a supported input document.
2. The pipeline reaches the image tagging step using api_image_request().
3. An exception is raised from the underlying HTTP request to the Azure OpenAI endpoint.

---

Stack Trace

Traceback (most recent call last):
  File "..\document_service.py", line 92, in _process_document_background
    chunks = self.doc_processor.process(processing_file_path)
  File "..\processor.py", line 38, in process
    result = self.converter.convert(source_path)
  File ".venv\Lib\site-packages\pydantic\_internal\_validate_call.py", line 39, in wrapper_function
    return wrapper(*args, **kwargs)
  File ".venv\Lib\site-packages\pydantic\_internal\_validate_call.py", line 136, in __call__
    res = self.__pydantic_validator__.validate_python(pydantic_core.ArgsKwargs(args, kwargs))
  File ".venv\Lib\site-packages\docling\document_converter.py", line 245, in convert
    return next(all_res)
  File ".venv\Lib\site-packages\docling\document_converter.py", line 268, in convert_all
    for conv_res in conv_res_iter:
  File ".venv\Lib\site-packages\docling\document_converter.py", line 340, in _convert
    for item in map(
  File ".venv\Lib\site-packages\docling\document_converter.py", line 387, in _process_document
    conv_res = self._execute_pipeline(in_doc, raises_on_error=raises_on_error)
  File ".venv\Lib\site-packages\docling\document_converter.py", line 410, in _execute_pipeline
    conv_res = pipeline.execute(in_doc, raises_on_error=raises_on_error)
  File ".venv\Lib\site-packages\docling\pipeline\base_pipeline.py", line 80, in execute
    raise e
  File ".venv\Lib\site-packages\docling\pipeline\base_pipeline.py", line 72, in execute
    conv_res = self._build_document(conv_res)
  File ".venv\Lib\site-packages\docling\pipeline\base_pipeline.py", line 270, in _build_document
    raise e
  File ".venv\Lib\site-packages\docling\pipeline\base_pipeline.py", line 230, in _build_document
    for p in pipeline_pages:  # Must exhaust!
  File ".venv\Lib\site-packages\docling\pipeline\base_pipeline.py", line 195, in _apply_on_pages
    yield from page_batch
  File ".venv\Lib\site-packages\docling\models\api_vlm_model.py", line 101, in __call__
    yield from executor.map(_vlm_request, page_batch)
  File "\uv\python\cpython-3.12.11-windows-x86_64-none\Lib\concurrent\futures\_base.py", line 619, in result_iterator
    yield _result_or_cancel(fs.pop())
  File "\uv\python\cpython-3.12.11-windows-x86_64-none\Lib\concurrent\futures\_base.py", line 317, in _result_or_cancel
    return fut.result(timeout)
  File "\uv\python\cpython-3.12.11-windows-x86_64-none\Lib\concurrent\futures\_base.py", line 456, in result
    return self.__get_result()
  File "\uv\python\cpython-3.12.11-windows-x86_64-none\Lib\concurrent\futures\_base.py", line 401, in __get_result
    raise self._exception
  File "\uv\python\cpython-3.12.11-windows-x86_64-none\Lib\concurrent\futures\thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
  File ".venv\Lib\site-packages\docling\models\api_vlm_model.py", line 87, in _vlm_request
    page_tags = api_image_request(
  File ".venv\Lib\site-packages\docling\utils\api_image_request.py", line 59, in api_image_request
    r.raise_for_status()
  File ".venv\Lib\site-packages\requests\models.py", line 1026, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError:  
500 Server Error: Internal Server Error  
for url: `https://<redacted-openai-endpoint>.azure.com/openai/deployments/gpt-4o/chat/completions?api-version=2024-12-01-preview`

Expected Behavior

The API should respond with a valid completion or return a controlled error (e.g., 4xx) that can be caught and handled gracefully in the pipeline.

Actual Behavior

The pipeline crashes with an unhandled HTTPError due to a 500 response from the Azure OpenAI deployment.

Environment

• Docling version: v2.57.0

• Python version: 3.12.11

• Model used: gpt-4o

• API version: 2024-12-01-preview

[dosubot]

The 500 Internal Server Error from Azure OpenAI during VLM OCR processing is not currently handled in the Docling pipeline—any such error from the api_image_request call will raise an uncaught HTTPError and crash the process. This is because api_image_request relies on requests.raise_for_status() without any try/except or retry logic, and the pipeline does not catch or manage these exceptions at any stage, including in the ApiVlmModel and the pipeline execution layers. As a result, any upstream API failure (like a 500 error) will terminate the entire document conversion process rather than being handled gracefully or reported in a controlled way.

To improve robustness, consider wrapping the api_image_request call in a try/except block (either in api_image_request itself or in the calling model) to catch requests.exceptions.HTTPError, log the error details, and return a controlled error response or fallback. You might also want to implement retries for transient 5xx errors and inspect the Azure OpenAI deployment for quota, rate limits, or misconfiguration that could cause 500s. No recent merged PRs address this in v2.57.0, but there is an open PR (#2445) that proposes changes to image API response handling, though it does not yet address error handling for 500s in the released codebase.

See the current implementation and error propagation here:
api_image_request.py
api_vlm_model.py
base_pipeline.py

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[kiendoantrung]

@dosu

ERROR:docling.utils.api_image_request:Error calling the API. Response was {"error":{"message":"Error occurred while processing image(s).","type":"internal_error","param":null,"code":null}}
WARNING:docling.pipeline.base_pipeline:Encountered an error during conversion of document 5f0f09120d165d101845c2d0f48bd51939d96de90ef4ac3e3c9a27205f54c1f1:
Traceback (most recent call last):

  File "/usr/local/lib/python3.12/dist-packages/docling/pipeline/base_pipeline.py", line 230, in _build_document
    for p in pipeline_pages:  # Must exhaust!
             ^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.12/dist-packages/docling/pipeline/base_pipeline.py", line 195, in _apply_on_pages
    yield from page_batch

  File "/usr/local/lib/python3.12/dist-packages/docling/models/api_vlm_model.py", line 101, in __call__
    yield from executor.map(_vlm_request, page_batch)

  File "/usr/lib/python3.12/concurrent/futures/_base.py", line 619, in result_iterator
    yield _result_or_cancel(fs.pop())
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.12/concurrent/futures/_base.py", line 317, in _result_or_cancel
    return fut.result(timeout)
           ^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.12/concurrent/futures/_base.py", line 456, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^

  File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception

  File "/usr/lib/python3.12/concurrent/futures/thread.py", line 59, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.12/dist-packages/docling/models/api_vlm_model.py", line 87, in _vlm_request
    page_tags = api_image_request(
                ^^^^^^^^^^^^^^^^^^

  File "/usr/local/lib/python3.12/dist-packages/docling/utils/api_image_request.py", line 59, in api_image_request
    r.raise_for_status()

  File "/usr/local/lib/python3.12/dist-packages/requests/models.py", line 1026, in raise_for_status
    raise HTTPError(http_error_msg, response=self)

requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: `https://<redacted-openai-endpoint>.azure.com/openai/deployments/gpt-4o/chat/completions?api-version=2024-12-01-preview`

[dosubot]

The 500 Internal Server Error from Azure OpenAI is currently unhandled in the Docling pipeline. The error is logged in api_image_request, but the function always raises an HTTPError, which propagates up through api_vlm_model and the pipeline, causing the entire document conversion to fail rather than being handled gracefully or skipped per page. There is no retry logic or controlled error response for transient 5xx errors—any such failure will crash the process see code.

To improve robustness, wrap the api_image_request call in a try/except block (either in the function itself or in the calling model), catch requests.exceptions.HTTPError, log the error details, and return a controlled error or fallback value. You may also want to implement retries for transient 5xx errors and consider handling errors per page so that a single API failure does not terminate the entire pipeline see model usage. This approach will help prevent full pipeline crashes and allow for partial or degraded output when the API is unstable.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[dolfim-ibm]

@kiendoantrung We would need more details to understand what could cause your problem. It would be interesting to understand

1. How are you calling and configuring docling?
2. As the message says, HTTPError: 500 Server Error: Internal Server Error would usually be an error in the Azure OpenAI server. Did you try on multiple documents? Are you always getting this issue or only on a particular document? Did you try reporting it to your Azure support?

[kiendoantrung]

from docling.datamodel.pipeline_options_vlm_model import ApiVlmOptions, ResponseFormat
from docling.document_converter import DocumentConverter, PdfFormatOption
from docling.datamodel.base_models import InputFormat
from docling.pipeline.vlm_pipeline import VlmPipeline
from docling.datamodel.pipeline_options import (
    VlmPipelineOptions,
    )


def azure_openai_vlm_options(deployment_name: str = None):

    options = ApiVlmOptions(
        url=f"{AZURE_OPENAI_ENDPOINT}/openai/deployments/{deployment_name}/chat/completions?api-version={AZURE_OPENAI_API_VERSION}",
        params=dict(
            model=deployment_name,
        ),
        headers={
            "api-key": AZURE_OPENAI_API_KEY,
            "Content-Type": "application/json"
        },
        prompt=(
            "OCR the full page to Markdown."
        ),
        timeout=900,
        scale=1.0,
        max_size=1024,
        response_format=ResponseFormat.MARKDOWN,
    )
    return options

pipeline_options = VlmPipelineOptions(
    enable_remote_services=True
    )

pipeline_options.vlm_options = azure_openai_vlm_options()

converter = DocumentConverter(
            format_options={
                InputFormat.PDF: PdfFormatOption(
                    pipeline_options=pipeline_options,
                    pipeline_cls=VlmPipeline,
                )
            }
        )

converter.convert("path-to-file").document.export_to_markdown()

The error occurred consistently for a period of time, but after retrying later, the same call succeeded without any changes.

So it appears the issue may have been transient — possibly related to the Azure OpenAI backend being temporarily unavailable or overloaded.

I haven't reported it to Azure support yet since it's no longer reproducible, but if it happens again, I will collect more detailed logs and open a ticket.

Thanks again for your help!

[dolfim-ibm]

Thanks for the update. I will also close the current Docling issue.