VLM fails silently when external API calls are content filtered.

[chrismoseley]

Bug

When using PictureDescriptionApiOptions with an external API that returns finish_reason: "content_filter" (e.g., Azure OpenAI with content safety filters enabled), the picture description is silently dropped. No warning is logged about the content filtering, and an empty string is stored as the description text.

api_image_request() in docling/utils/api_image_request.py only logs VlmStopReason.LENGTH and VlmStopReason.END_OF_SEQUENCE. Suggest adding VlmStopReason.FILTERED.

Steps to reproduce

Happens even with entirely innocent images. Attached example. Prompt: "
Describe the image in no more than three concise sentences. For graphs, mention the ranges of each axis. If the image is decorative, like a natural landscape or abstract patterns, just say 'decorative image'. "If there are people in the image, describe them and include their demographics like age and gender where possible. Do not invent text or data."

Docling version

2.73

Python version

3.12

[dosubot]

This is a confirmed bug: when the external API returns finish_reason: "content_filter", api_image_request does not log or handle it, resulting in a silent drop and empty description. The fix is to add FILTERED to the VlmStopReason enum and update api_image_request to detect "content_filter", log a warning, and return VlmStopReason.FILTERED so downstream consumers can handle it explicitly. This approach matches Docling's standard error reporting pattern of structured logging and explicit stop-reason metadata in prediction models. There is currently no broader handling or notification for filtered content elsewhere in the codebase, so surfacing this stop reason and log is the right move.

Relevant code references:

• VlmStopReason enum definition
• api_image_request implementation

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[dolfim-ibm]

@chrismoseley Some clarification question.

• I think your use case is mostly documented here, right?
  • https://learn.microsoft.com/en-us/azure/ai-foundry/foundry-models/concepts/content-filter?view=foundry-classic&tabs=user-prompt#content-filtering-scenarios
• "the picture description is silently dropped": my understanding is that indeed there should be no content for an image which triggered the filtering, right?
  • can you please elaborate a bit more on what you see happening and what you would expect happening?

[chrismoseley]

Yes that's the scenario. The advice from MS is to handle "finish_reason": "content_filter" in the calling app, which would be fine but when using VLM with external api in Docling, there is no indication it has happened so no way to handle downstream.

/utils/api_image_request.py

            api_resp = OpenAiApiResponse.model_validate_json(r.text)   # t.text is ''
            generated_text = api_resp.choices[0].message.content.strip() # generated_text is ''
            num_tokens = api_resp.usage.total_tokens
            stop_reason = (
                VlmStopReason.LENGTH
                if api_resp.choices[0].finish_reason == "length"
                else VlmStopReason.END_OF_SEQUENCE
            )

            # stop_reason is VlmStopReason.END_OF_SEQUENCE at this point, which is not correct.

            return generated_text, num_tokens, stop_reason

Finish reason of "content_filter" is never flagged or recorded so impossible to handle. I would expect stop_reason = VlmStopReason.CONTENT_FILTERED.

[cau-git]

@chrismoseley We can easily add VlmStopReason.CONTENT_FILTERED. Note however that no trace of any VlmStopReason will make it into the output DoclingDocument. You will only be able to figure out what happened from the auxiliary data structures in ConversionResult.pages

[chrismoseley]

Ah, yeh I see... and in the ConversionResult.pages it looks like it wouldn't be possible to link in to a specific picture.

Feels like the full fix would be to extend PictureMeta but that would be a job.

How about a CONTENT_FILTERED_SENTINEL constant (defaults to "", added option to API configs) then:

if finish_reason == "content_filter": return CONTENT_FILTERED_SENTINEL, num_tokens, VlmStopReason.CONTENT_FILTERED

?

[Br1an67]

I'd like to work on this. Add CONTENT_FILTERED to VlmStopReason enum and handle finish_reason == "content_filter" in api_image_request() with a warning log.

PR: #3051

[NicolasHoferer]

I do struggle with this as well. Do you know if there is a way to make it more reliable? Like adapting the prompt in a way that does trigger the CONTENT_FILTERED so often?

[chrismoseley]

I do struggle with this as well. Do you know if there is a way to make it more reliable? Like adapting the prompt in a way that does trigger the CONTENT_FILTERED so often?

With Azure AI Foundry you can adjust the filter to be more permissive (also, it still can cause issues).

Key thing is to find out if it's the prompt or the response that's being filtered. If it's the prompt, you won't get any response, if it's the response you get CONTENT_FILTERED.

You can mitigate it usually with the prompt. My prompt in the example above was something like "describe this image, if it contains people describe their clothing and demographics".

Images with children in them were causing the response to be filtered. Simply appending "If the image contains children, do not describe them any further" fixed the issue.