VLM Custom Pipeline and Custom Presets

[here4dadata]

In the documentation, there is a line about custom VLM configurations, only available if admin allows it...
How is this made available? via config file? env variable?
Is there way to override the preset (with only env variables or config files)?

string (Vlm Pipeline Custom Config)
Custom VLM configuration including model spec and engine options. Only available if admin allows it. Must include 'model_spec' and 'engine_options'

If we go with non-deprecated params, you end up with this error

ERROR:docling_jobkit.orchestrators.local.worker:Worker 0 failed to process job 81d9b50a-0099-4e64-9a83-97ea5aca45f8: Custom VLM configuration is not allowed. Please use a preset or contact your administrator.

[dosubot]

Hi @here4dadata! I'm Dosu and I’m helping the docling-serve team.

The allow_custom_vlm_config flag exists in docling-jobkit's DoclingConverterManagerConfig, and it defaults to False — which is why you're seeing that error.

The issue: This flag is not currently exposed in docling-serve's configuration system. When docling-serve instantiates the converter manager config, it doesn't pass through the allow_custom_vlm_config parameter, so there's currently no env variable or config file option to enable it.

Workaround options:

1. Use presets instead — vlm_pipeline_preset and picture_description_preset work without needing admin permission
2. Request this feature — docling-serve would need to add an environment variable (e.g., DOCLING_SERVE_ALLOW_CUSTOM_VLM_CONFIG) and pass it through to the DoclingConverterManagerConfig initialization

This looks like a gap in the feature integration between docling-jobkit and docling-serve. You may want to open a feature request to expose this admin setting.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[here4dadata]

How do we override the presets? Is this something that can be accomplished with an env variable/config file?

[dosubot]

Currently, docling-serve doesn't expose env variables or config files for overriding presets — the capabilities exist in docling-jobkit but aren't wired through to docling-serve.

In docling-jobkit's DoclingConverterManagerConfig, there are admin controls for this:

• custom_vlm_presets: Dictionary to define your own presets
• default_vlm_preset: Override the default preset
• allowed_vlm_presets: Whitelist specific presets

docling-jobkit also supports YAML config files for defining custom presets when using the CLI.

However, docling-serve doesn't currently pass these configuration options through when initializing the converter manager — so these admin controls aren't accessible. This seems like another integration gap that would need a feature request to expose these settings via env variables or a config file in docling-serve.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[dolfim-ibm]

Yes, we want to enable setting the presets config via a config file or ENV.

[here4dadata]

@dolfim-ibm Thanks for the update. Do you recommend just sticking with the deprecated VLM params for now?

[awschult002]

on a similar note... how would i go about setting up a deployment with remote VLM models? I have a remote Ollama instance and I want to offload the VLM inference to it. However, I don't see an examples that explicitly inform me on how to do that. When I do set "remote api = true", then how do i set that remote URL? is there an ENV VAR for it? or when i send my convert POST message, do i need to provide it in there? does that require custom model information passed in the POST?

[dolfim-ibm]

You can provide the url in the POST payload, when using the custom config.
Soon we will also enable to configure a custom preset on the server (with a name), then you can expose it to users of the API just by "name".

[awschult002]

Oh fantastic! I have been watching the IBM models recently and am currently waiting for the new embedding and reranking models to get merged in.

I think I am just going to wait and upgrade docling and ollama all at once. That way, I can get granite-docling vlm, plus embeddings and reranking. That would be an insane upgrade.

[hdnh2006]

I have done it using a config file like this, specifically for Open WebUI, but it works only for pdf. Due to how the pipeline is built by docling, It is not working for docx or xlsx files:

{
  "pipeline": "vlm",
  "do_ocr": false,
  "to_formats": [
    "md"
  ],
  "vlm_pipeline_model_api": "{\"url\": \"https://your_openai_compatible_endpoint/api/chat/completions\", \"headers\": {\"Authorization\": \"Bearer sk-api-key\"}, \"params\": {\"model\": \"your_vlm_model\", \"max_tokens\": 8192}, \"prompt\": \"Convert this page to markdown. Do not miss any text and only output the bare markdown! Your response will be used as a RAG by another model, it is important you don't make any mistake. DO YOUR BEST\", \"response_format\": \"markdown\", \"timeout\": 180.0, \"concurrency\": 16}"
}