Inference takes forever when using pipeline=vlm

[lfoppiano]

Hi all,
I'm trying to deploy the docling-serve 1.6.0 with the docling-granite VLLM on a A10 on modal.com. The standard pipeline works fine (30s/document - is this runtime reasonable?). when I run with pipeline=vlm the process goes in timeout. I tried with a L40S GPU too but it seems that the GPU is not the issue. A < 1B params model should fit into any of those.

What GPU would you recommend using?

Any suggestion for deployment for using the pipeline=vlm?

[dosubot]

For pipeline=vlm, both A10 and L40S GPUs should be sufficient for models under 1B parameters, so the hardware itself is likely not the bottleneck. Timeout and slow inference can result from inefficient batching, model loading, or configuration issues. Recent updates added batching support and an initial VLLM backend, which can improve performance—make sure you're using the latest configuration options for batching and model sharing (source, source).

For deployment, it's recommended to pre-download models and mount them into your container or use persistent storage to avoid runtime delays (source). Also, check your batch size and timeout settings in the config, and consider enabling model sharing among worker threads with DOCLING_SERVE_ENG_LOC_SHARE_MODELS=true to reduce memory usage and speed up inference.

If you’re processing large documents, splitting them into smaller chunks and handling them asynchronously can help avoid timeouts (source).

30s/document for the standard pipeline is within a reasonable range, but you may be able to optimize further by tuning batch size, worker count, and ensuring efficient GPU utilization.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[lfoppiano]

I tried with DOCLING_SERVE_ENG_LOC_SHARE_MODELS=true but no difference. I was wondering whether there is a way to see more logs of what's going on inside when a document arrives

[dosubot]

You can increase log verbosity in docling-serve by using the --verbose or -v flag when starting the service: -v sets INFO level (default), and -vv sets DEBUG level for even more detailed logs. This will show more about what happens when a document arrives, including events like file receipt and task processing. Logs are output to the console by default, with colored formatting for easier reading. There isn’t a dedicated environment variable for log level—use the CLI flags instead. If you’re running under uvicorn directly, you can also adjust uvicorn’s log level with its own flags, but the main app logging is controlled by the -v/-vv option.

Key document processing events (like receiving files) are logged at INFO level, so -v should already show those; -vv will add debug details if available in the code.

Reference: logging setup and CLI options in docling-serve

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[lfoppiano]

-vv and --verbose don't work with docling-serve cli. I also don't see these options here: https://github.com/docling-project/docling-serve/blob/main/docs/configuration.md

[dosubot]

The docling-serve CLI does support verbosity flags: you can use -v for INFO level and -vv for DEBUG level logging. These options are implemented using Typer and should be available globally when you run the CLI, even though they're not documented in the configuration docs. If they're not working for you, it could be a usage or environment issue—make sure you're running the CLI entry point directly (not via another wrapper or script) and that your installed version matches the codebase where these flags are present. You can see the implementation details in the main entry point of the project here.

If you continue to have trouble, let me know how you're invoking the CLI and which version you're running, and I can help troubleshoot further.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[lfoppiano]

I'm invoking with

docling-serve run --host 0.0.0.0 --port {PORT}

here the error:

Usage: docling-serve run [OPTIONS]
Try 'docling-serve run --help' for help.
╭─ Error ──────────────────────────────────────────────────────────────────────╮
│ No such option: --verbose                                                    │
╰──────────────────────────────────────────────────────────────────────────────╯

[dosubot]

The error happens because the verbosity flags (--verbose, -v, -vv) are global options in the docling-serve CLI and must be placed before the subcommand. To enable more detailed logging, use:

docling-serve --verbose run --host 0.0.0.0 --port {PORT}

or

docling-serve -vv run --host 0.0.0.0 --port {PORT}

Passing --verbose after run won't work because run does not accept it as an option directly. There are no environment variables or config file options for logging verbosity—only the CLI flags work for this purpose. Reference

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[lfoppiano]

OK that was helpful as I could fix the logs...

Now, from the log it seems that the granitedocling is not loaded:

INFO:docling.models.factories.base_factory:Loading plugin 'docling_defaults'
INFO:docling.models.factories:Registered ocr engines: ['easyocr', 'ocrmac', 'rapidocr', 'tesserocr', 'tesseract']
INFO:     Started server process [5]
INFO:     Waiting for application startup.
INFO:docling_jobkit.convert.manager:artifacts_path is set to a valid directory. No model weights will be downloaded at runtime.
INFO:docling.document_converter:Initializing pipeline for StandardPdfPipeline with options hash 2983a1a8b72c4ba9405dda65b9f3ee67
INFO:docling.models.factories.base_factory:Loading plugin 'docling_defaults'
INFO:docling.models.factories:Registered picture descriptions: ['vlm', 'api']
INFO:docling.utils.accelerator_utils:Accelerator device: 'cuda:0'
INFO:docling.utils.accelerator_utils:Accelerator device: 'cuda:0'
DEBUG:docling_ibm_models.layoutmodel.layout_predictor:LayoutPredictor settings: {'model_name': 'RTDetrV2ForObjectDetection', 'safe_tensors_file': '/opt/app-root/src/.cache/docling/models/ds4sd--docling-layout-heron/model.safetensors', 'device': 'cuda', 'num_threads': 1, 'image_size': {'height': 640, 'width': 640}, 'threshold': 0.3}
INFO:docling.utils.accelerator_utils:Accelerator device: 'cuda:0'
DEBUG:docling_jobkit.orchestrators.local.orchestrator:Starting worker 0
DEBUG:docling_jobkit.orchestrators.local.orchestrator:Starting worker 1
INFO:     Application startup complete.
DEBUG:docling_jobkit.orchestrators.local.worker:Starting loop for worker 0
DEBUG:docling_jobkit.orchestrators.local.worker:Starting loop for worker 1
INFO:     Uvicorn running on http://0.0.0.0:5001 (Press CTRL+C to quit)
DEBUG:python_multipart.multipart:Calling on_part_begin with no data
DEBUG:python_multipart.multipart:Calling on_header_field with data[50:69]
DEBUG:python_multipart.multipart:Calling on_header_value with data[71:123]
DEBUG:python_multipart.multipart:Calling on_header_end with no data
DEBUG:python_multipart.multipart:Calling on_header_field with data[125:137]
DEBUG:python_multipart.multipart:Calling on_header_value with data[139:154]
DEBUG:python_multipart.multipart:Calling on_header_end with no data
DEBUG:python_multipart.multipart:Calling on_headers_finished with no data
DEBUG:python_multipart.multipart:Calling on_part_data with data[158:211341]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:116104]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:16384]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:49152]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:16384]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:16384]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:32768]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:16384]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:16384]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:32768]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:16384]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:49152]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:16384]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:16384]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:32768]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:16384]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:49152]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:16384]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:49152]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:16384]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:49152]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:32768]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:32768]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:16384]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:32768]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:16384]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:16384]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:114688]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:32768]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:32768]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:81920]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:16384]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:32768]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:32768]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:98304]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:32768]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:32768]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:32768]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:131072]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:65536]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:49152]
DEBUG:python_multipart.multipart:Calling on_part_data with data[0:23152]
DEBUG:python_multipart.multipart:Calling on_part_end with no data
DEBUG:python_multipart.multipart:Calling on_part_begin with no data
DEBUG:python_multipart.multipart:Calling on_header_field with data[23204:23223]
DEBUG:python_multipart.multipart:Calling on_header_value with data[23225:23253]
DEBUG:python_multipart.multipart:Calling on_header_end with no data
DEBUG:python_multipart.multipart:Calling on_headers_finished with no data
DEBUG:python_multipart.multipart:Calling on_part_data with data[23257:23261]
DEBUG:python_multipart.multipart:Calling on_part_end with no data
DEBUG:python_multipart.multipart:Calling on_part_begin with no data
DEBUG:python_multipart.multipart:Calling on_header_field with data[23313:23332]
DEBUG:python_multipart.multipart:Calling on_header_value with data[23334:23360]
DEBUG:python_multipart.multipart:Calling on_header_end with no data
DEBUG:python_multipart.multipart:Calling on_headers_finished with no data
DEBUG:python_multipart.multipart:Calling on_part_data with data[23364:23367]
DEBUG:python_multipart.multipart:Calling on_part_end with no data
DEBUG:python_multipart.multipart:Calling on_end with no data
INFO:docling_serve.app:Received 1 files for processing.
INFO:docling_jobkit.orchestrators.local.worker:Worker 0 processing task a0393097-3823-43ae-9592-023fc492c5da
INFO:docling_jobkit.convert.manager:artifacts_path is set to a valid directory. No model weights will be downloaded at runtime.
INFO:docling.document_converter:Initializing pipeline for VlmPipeline with options hash 385bbcb6f959d7a2c19376da75f8910c
INFO:docling.utils.accelerator_utils:Accelerator device: 'cuda:0'
DEBUG:docling.models.vlm_models_inline.hf_transformers_model:Available device for VLM: cuda:0
INFO:docling.datamodel.document:detected formats: [<InputFormat.PDF: 'pdf'>]
INFO:docling.document_converter:Going to convert document batch...
DEBUG:docling.document_converter:Reusing cached pipeline for VlmPipeline with options hash 385bbcb6f959d7a2c19376da75f8910c
INFO:docling.pipeline.base_pipeline:Processing document 1606.06159v3.pdf
[modal-client] 2025-10-12T13:35:41+0000 Received a cancellation signal while processing input ('in-01K7C9YCW7107C8APAFS4GG8NR:1760275936141-0',)
2025-10-12 13:35:41,815 - Received a cancellation signal while processing input ('in-01K7C9YCW7107C8APAFS4GG8NR:1760275936141-0',)
DEBUG:docling.models.vlm_models_inline.hf_transformers_model:Generated 2125 tokens in 40.46s for batch size 4.
DEBUG:docling.pipeline.base_pipeline:Finished converting pages 4/18 time=224.437

I initially run docling-tools models download granitedocling to downlad the model in the default directory, anything wrong with that?

Any other clue what's going on here?

[dosubot]

There's nothing wrong with downloading the granitedocling model using docling-tools models download granitedocling, but for docling-serve to actually use it, you need to make sure two things are set up:

1. The environment variable DOCLING_SERVE_ARTIFACTS_PATH must point to the directory where you downloaded the model. If you downloaded to the default location, confirm that this path matches what docling-serve expects at runtime. For containerized deployments, mounting the model directory and setting this variable is the recommended approach source.

2. The pipeline configuration must explicitly reference the granitedocling model via its repo_id in the VLM pipeline options. If the pipeline config doesn't specify granitedocling, docling-serve won't load it, even if the weights are present source.

The logs you shared show the VLM pipeline initializing, but they don't confirm which model is being loaded. If you don't see granitedocling-specific log entries, it's likely the pipeline config isn't set to use it. Double-check your pipeline/task/API config to ensure granitedocling is selected.

If you set both the artifacts path and the pipeline config correctly, docling-serve should load the model and use it for inference. If you still see timeouts or cancellation signals, those may be unrelated to model loading and could be due to resource limits or runtime constraints on modal.com.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[lfoppiano]

@dosu

The pipeline configuration must explicitly reference the granitedocling model via its repo_id in the VLM pipeline options. If the pipeline config doesn't specify granitedocling, docling-serve won't load it, even if the weights are present source.

need a pointer to the cli or env variables

[dosubot]

There is no CLI flag or environment variable to set the VLM model repo_id (like granitedocling) in docling-serve. Model selection is not exposed at the serve layer—it's handled internally and set globally. To use granitedocling, just make sure you have downloaded the model files and that DOCLING_SERVE_ARTIFACTS_PATH points to the directory containing them. Each model directory must have a config.json with a "model_type" key. If the files are present, granitedocling will be used by default for the VLM pipeline as of recent versions. You can verify which models are loaded by checking the artifacts directory, reviewing logs, or looking at the autogenerated API docs at /docs on your running instance.
Reference

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[lfoppiano]

I think I"m doing all well..

/artifacts/ds4sd--docling-layout-heron:
config.json
model.safetensors
preprocessor_config.json
README.md

/artifacts/ds4sd--docling-models:
config.json
model_artifacts
README.md

/artifacts/ds4sd--DocumentFigureClassifier:
config.json
model.safetensors
README.md

/artifacts/EasyOcr:
craft_mlt_25k.pth
english_g2.pth
latin_g2.pth

/artifacts/ibm-granite--granite-docling-258M:
added_tokens.json
assets
chat_template.jinja
config.json
generation_config.json
granite_docling.png
merges.txt
model.safetensors
preprocessor_config.json
processor_config.json
README.md
special_tokens_map.json
tokenizer_config.json
tokenizer.json
vocab.json

And I'm runnning the server with --artifacts-path {ARTIFACTS_PATH}":

Starting production server 🚀

Server started at http://0.0.0.0:5001
Documentation at http://0.0.0.0:5001/docs
Scalar docs at http://0.0.0.0:5001/scalar

Logs:
INFO:docling.models.factories.base_factory:Loading plugin 'docling_defaults'
INFO:docling.models.factories:Registered ocr engines: ['easyocr', 'ocrmac', 'rapidocr', 'tesserocr', 'tesseract']
INFO:     Started server process [6]
INFO:     Waiting for application startup.
INFO:docling_jobkit.convert.manager:artifacts_path is set to a valid directory. No model weights will be downloaded at runtime.
INFO:docling.document_converter:Initializing pipeline for StandardPdfPipeline with options hash ee4f78680442883bba43982e70792372
INFO:docling.models.factories.base_factory:Loading plugin 'docling_defaults'
INFO:docling.models.factories:Registered picture descriptions: ['vlm', 'api']
INFO:docling.utils.accelerator_utils:Accelerator device: 'cuda:0'
INFO:docling.utils.accelerator_utils:Accelerator device: 'cuda:0'
DEBUG:docling_ibm_models.layoutmodel.layout_predictor:LayoutPredictor settings: {'model_name': 'RTDetrV2ForObjectDetection', 'safe_tensors_file': '/artifacts/ds4sd--docling-layout-heron/model.safetensors', 'device': 'cuda', 'num_threads': 1, 'image_size': {'height': 640, 'width': 640}, 'threshold': 0.3}
INFO:docling.utils.accelerator_utils:Accelerator device: 'cuda:0'
DEBUG:docling_jobkit.orchestrators.local.orchestrator:Starting worker 0
DEBUG:docling_jobkit.orchestrators.local.orchestrator:Starting worker 1
INFO:     Application startup complete.
DEBUG:docling_jobkit.orchestrators.local.worker:Starting loop for worker 0
DEBUG:docling_jobkit.orchestrators.local.worker:Starting loop for worker 1
INFO:docling.document_converter:Initializing pipeline for VlmPipeline with options hash 45fcff1f2076b0b276b4930e6e482218
INFO:docling.utils.accelerator_utils:Accelerator device: 'cuda:0'
DEBUG:docling.models.vlm_models_inline.hf_transformers_model:Available device for VLM: cuda:0
INFO:docling.datamodel.document:detected formats: [<InputFormat.PDF: 'pdf'>]
INFO:docling.document_converter:Going to convert document batch...
DEBUG:docling.document_converter:Reusing cached pipeline for VlmPipeline with options hash 45fcff1f2076b0b276b4930e6e482218
INFO:docling.pipeline.base_pipeline:Processing document 1606.06159v3.pdf
DEBUG:docling.models.vlm_models_inline.hf_transformers_model:Generated 2125 tokens in 29.98s for batch size 4.
DEBUG:docling.pipeline.base_pipeline:Finished converting pages 4/18 time=123.898

Now it seems that there are batch = 4 and 4 pages are converted in 123s. Anny clue how to make this faster?

[lfoppiano]

I increased the threads to 8 but still I get Generated XY tokens in 29.98s for batch size 4...

[dolfim-ibm]

if you have the possibility, I would try once to run the Docling CLI on your test document.

[lfoppiano]

@dolfim-ibm Is there any chance that this model can run on CPUs? (my curiosity)

Regarding running it "locally", I tried on a Mac -- on CPUs -- , and I can download the models:

(grobid-align) lfoppiano@MacBook-Air Downloads % docling-tools models download granitedocling layout tableformer picture_classifier easyocr  --output-dir /Volumes/ExtremePro/sciencialab/llms/docling
/Users/lfoppiano/anaconda3/envs/grobid-align/lib/python3.12/site-packages/pydantic/_internal/_fields.py:132: UserWarning: Field "model_path" in LayoutModelConfig has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
/Users/lfoppiano/anaconda3/envs/grobid-align/lib/python3.12/site-packages/pydantic/_internal/_fields.py:132: UserWarning: Field "model_spec" in LayoutOptions has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
Downloading layout model...                                                                                                                                      model_downloader.py:53
Fetching 5 files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:01<00:00,  4.50it/s]
Downloading tableformer model...                                                                                                                                 model_downloader.py:61
Fetching 8 files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 8/8 [00:00<00:00, 3552.24it/s]
Downloading picture classifier model...                                                                                                                          model_downloader.py:69
Fetching 4 files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 965.21it/s]
Downloading GraniteDocling model...                                                                                                                              model_downloader.py:94
Fetching 17 files: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 17/17 [00:00<00:00, 1433.26it/s]
Downloading easyocr models...                                                                                                                                   model_downloader.py:139
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 73.7M/73.7M [00:02<00:00, 26.3MB/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 13.4M/13.4M [00:00<00:00, 18.5MB/s]
100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 13.6M/13.6M [00:00<00:00, 16.5MB/s]

Models downloaded into: /Volumes/ExtremePro/sciencialab/llms/docling.

 Docling can now be configured for running offline using the local artifacts.

 Using the CLI: `docling --artifacts-path=/Volumes/ExtremePro/sciencialab/llms/docling FILE`
 Using Python: see the documentation at <https://docling-project.github.io/docling/usage>.

But then when I pass the path it seems not working:

(grobid-align) lfoppiano@MacBook-Air Downloads % docling --to=json --pipeline=vlm --vlm-model=granite_docling --artifacts-path=/Volumes/ExtremePro/sciencialab/llms/docling/  ~/Downloads/1606.06159v3.pdf
/Users/lfoppiano/anaconda3/envs/grobid-align/lib/python3.12/site-packages/pydantic/_internal/_fields.py:132: UserWarning: Field "model_path" in LayoutModelConfig has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
/Users/lfoppiano/anaconda3/envs/grobid-align/lib/python3.12/site-packages/pydantic/_internal/_fields.py:132: UserWarning: Field "model_spec" in LayoutOptions has conflict with protected namespace "model_".

You may be able to resolve this warning by setting `model_config['protected_namespaces'] = ()`.
  warnings.warn(
2025-10-13 12:29:24,888 - INFO - Loading plugin 'docling_defaults'
2025-10-13 12:29:24,890 - INFO - Registered ocr engines: ['easyocr', 'ocrmac', 'rapidocr', 'tesserocr', 'tesseract']
2025-10-13 12:29:25,330 - INFO - paths: [PosixPath('/var/folders/mk/scd8428n18jfgh3jdthbvpz00000gn/T/tmpitv5z4yz/1606.06159v3.pdf')]
2025-10-13 12:29:25,330 - INFO - detected formats: [<InputFormat.PDF: 'pdf'>]
2025-10-13 12:29:25,381 - INFO - Going to convert document batch...
2025-10-13 12:29:25,381 - INFO - Initializing pipeline for VlmPipeline with options hash 8718ac387f327d4914c803cc412ecebd
2025-10-13 12:29:25,387 - INFO - Loading plugin 'docling_defaults'
2025-10-13 12:29:25,388 - INFO - Registered picture descriptions: ['vlm', 'api']
Traceback (most recent call last):
  File "/Users/lfoppiano/anaconda3/envs/grobid-align/lib/python3.12/site-packages/mlx_vlm/utils.py", line 332, in load_config
    return AutoConfig.from_pretrained(model_path, **kwargs).to_dict()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/lfoppiano/anaconda3/envs/grobid-align/lib/python3.12/site-packages/transformers/models/auto/configuration_auto.py", line 1381, in from_pretrained
    raise ValueError(
ValueError: Unrecognized model in /Volumes/ExtremePro/sciencialab/llms/docling. Should have a `model_type` key in its config.json, or contain one of the following strings in its name: aimv2, aimv2_vision_model, albert, align, altclip, apertus, arcee, aria, aria_text, audio-spectrogram-transformer, autoformer, aya_vision, bamba, bark, bart, beit, bert, bert-generation, big_bird, bigbird_pegasus, biogpt, bit, bitnet, blenderbot, blenderbot-small, blip, blip-2, blip_2_qformer, bloom, blt, bridgetower, bros, camembert, canine, chameleon, chinese_clip, chinese_clip_vision_model, clap, clip, clip_text_model, clip_vision_model, clipseg, clvp, code_llama, codegen, cohere, cohere2, cohere2_vision, colpali, colqwen2, conditional_detr, convbert, convnext, convnextv2, cpmant, csm, ctrl, cvt, d_fine, dab-detr, dac, data2vec-audio, data2vec-text, data2vec-vision, dbrx, deberta, deberta-v2, decision_transformer, deepseek_v2, deepseek_v3, deepseek_vl, deepseek_vl_hybrid, deformable_detr, deit, depth_anything, depth_pro, deta, detr, dia, diffllama, dinat, dinov2, dinov2_with_registers, dinov3_convnext, dinov3_vit, distilbert, doge, donut-swin, dots1, dpr, dpt, edgetam, edgetam_video, edgetam_vision_model, efficientformer, efficientloftr, efficientnet, electra, emu3, encodec, encoder-decoder, eomt, ernie, ernie4_5, ernie4_5_moe, ernie_m, esm, evolla, exaone4, falcon, falcon_h1, falcon_mamba, fastspeech2_conformer, fastspeech2_conformer_with_hifigan, flaubert, flava, flex_olmo, florence2, fnet, focalnet, fsmt, funnel, fuyu, gemma, gemma2, gemma3, gemma3_text, gemma3n, gemma3n_audio, gemma3n_text, gemma3n_vision, git, glm, glm4, glm4_moe, glm4v, glm4v_moe, glm4v_moe_text, glm4v_text, glpn, got_ocr2, gpt-sw3, gpt2, gpt_bigcode, gpt_neo, gpt_neox, gpt_neox_japanese, gpt_oss, gptj, gptsan-japanese, granite, granite_speech, granitemoe, granitemoehybrid, granitemoeshared, granitevision, graphormer, grounding-dino, groupvit, helium, hgnet_v2, hiera, hubert, hunyuan_v1_dense, hunyuan_v1_moe, ibert, idefics, idefics2, idefics3, idefics3_vision, ijepa, imagegpt, informer, instructblip, instructblipvideo, internvl, internvl_vision, jamba, janus, jetmoe, jukebox, kosmos-2, kosmos-2.5, kyutai_speech_to_text, layoutlm, layoutlmv2, layoutlmv3, led, levit, lfm2, lfm2_vl, lightglue, lilt, llama, llama4, llama4_text, llava, llava_next, llava_next_video, llava_onevision, longcat_flash, longformer, longt5, luke, lxmert, m2m_100, mamba, mamba2, marian, markuplm, mask2former, maskformer, maskformer-swin, mbart, mctct, mega, megatron-bert, metaclip_2, mgp-str, mimi, minimax, ministral, mistral, mistral3, mixtral, mlcd, mllama, mm-grounding-dino, mobilebert, mobilenet_v1, mobilenet_v2, mobilevit, mobilevitv2, modernbert, modernbert-decoder, moonshine, moshi, mpnet, mpt, mra, mt5, musicgen, musicgen_melody, mvp, nat, nemotron, nezha, nllb-moe, nougat, nystromformer, olmo, olmo2, olmo3, olmoe, omdet-turbo, oneformer, open-llama, openai-gpt, opt, ovis2, owlv2, owlvit, paligemma, parakeet, parakeet_ctc, parakeet_encoder, patchtsmixer, patchtst, pegasus, pegasus_x, perceiver, perception_encoder, perception_lm, persimmon, phi, phi3, phi4_multimodal, phimoe, pix2struct, pixtral, plbart, poolformer, pop2piano, prompt_depth_anything, prophetnet, pvt, pvt_v2, qdqbert, qwen2, qwen2_5_omni, qwen2_5_vl, qwen2_5_vl_text, qwen2_audio, qwen2_audio_encoder, qwen2_moe, qwen2_vl, qwen2_vl_text, qwen3, qwen3_moe, qwen3_next, qwen3_omni_moe, qwen3_vl, qwen3_vl_moe, qwen3_vl_moe_text, qwen3_vl_text, rag, realm, recurrent_gemma, reformer, regnet, rembert, resnet, retribert, roberta, roberta-prelayernorm, roc_bert, roformer, rt_detr, rt_detr_resnet, rt_detr_v2, rwkv, sam, sam2, sam2_hiera_det_model, sam2_video, sam2_vision_model, sam_hq, sam_hq_vision_model, sam_vision_model, seamless_m4t, seamless_m4t_v2, seed_oss, segformer, seggpt, sew, sew-d, shieldgemma2, siglip, siglip2, siglip2_vision_model, siglip_vision_model, smollm3, smolvlm, smolvlm_vision, speech-encoder-decoder, speech_to_text, speech_to_text_2, speecht5, splinter, squeezebert, stablelm, starcoder2, superglue, superpoint, swiftformer, swin, swin2sr, swinv2, switch_transformers, t5, t5gemma, table-transformer, tapas, textnet, time_series_transformer, timesfm, timesformer, timm_backbone, timm_wrapper, trajectory_transformer, transfo-xl, trocr, tvlt, tvp, udop, umt5, unispeech, unispeech-sat, univnet, upernet, van, vaultgemma, video_llava, videomae, vilt, vipllava, vision-encoder-decoder, vision-text-dual-encoder, visual_bert, vit, vit_hybrid, vit_mae, vit_msn, vitdet, vitmatte, vitpose, vitpose_backbone, vits, vivit, vjepa2, voxtral, voxtral_encoder, wav2vec2, wav2vec2-bert, wav2vec2-conformer, wavlm, whisper, xclip, xcodec, xglm, xlm, xlm-prophetnet, xlm-roberta, xlm-roberta-xl, xlnet, xlstm, xmod, yolos, yoso, zamba, zamba2, zoedepth

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/lfoppiano/anaconda3/envs/grobid-align/lib/python3.12/site-packages/mlx_vlm/utils.py", line 335, in load_config
    with open(model_path / "config.json", encoding="utf-8") as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/Volumes/ExtremePro/sciencialab/llms/docling/config.json'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/lfoppiano/anaconda3/envs/grobid-align/bin/docling", line 8, in <module>
    sys.exit(app())
             ^^^^^
  File "/Users/lfoppiano/anaconda3/envs/grobid-align/lib/python3.12/site-packages/typer/main.py", line 341, in __call__
    raise e
  File "/Users/lfoppiano/anaconda3/envs/grobid-align/lib/python3.12/site-packages/typer/main.py", line 324, in __call__
    return get_command(self)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/lfoppiano/anaconda3/envs/grobid-align/lib/python3.12/site-packages/click/core.py", line 1442, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/lfoppiano/anaconda3/envs/grobid-align/lib/python3.12/site-packages/typer/core.py", line 694, in main
    return _main(
           ^^^^^^
  File "/Users/lfoppiano/anaconda3/envs/grobid-align/lib/python3.12/site-packages/typer/core.py", line 195, in _main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/lfoppiano/anaconda3/envs/grobid-align/lib/python3.12/site-packages/click/core.py", line 1226, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/lfoppiano/anaconda3/envs/grobid-align/lib/python3.12/site-packages/click/core.py", line 794, in invoke
    return callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/lfoppiano/anaconda3/envs/grobid-align/lib/python3.12/site-packages/typer/main.py", line 699, in wrapper
    return callback(**use_params)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/lfoppiano/anaconda3/envs/grobid-align/lib/python3.12/site-packages/docling/cli/main.py", line 775, in convert
    export_documents(
  File "/Users/lfoppiano/anaconda3/envs/grobid-align/lib/python3.12/site-packages/docling/cli/main.py", line 204, in export_documents
    for conv_res in conv_results:
                    ^^^^^^^^^^^^
  File "/Users/lfoppiano/anaconda3/envs/grobid-align/lib/python3.12/site-packages/docling/document_converter.py", line 268, in convert_all
    for conv_res in conv_res_iter:
                    ^^^^^^^^^^^^^
  File "/Users/lfoppiano/anaconda3/envs/grobid-align/lib/python3.12/site-packages/docling/document_converter.py", line 340, in _convert
    for item in map(
                ^^^^
  File "/Users/lfoppiano/anaconda3/envs/grobid-align/lib/python3.12/site-packages/docling/document_converter.py", line 387, in _process_document
    conv_res = self._execute_pipeline(in_doc, raises_on_error=raises_on_error)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/lfoppiano/anaconda3/envs/grobid-align/lib/python3.12/site-packages/docling/document_converter.py", line 408, in _execute_pipeline
    pipeline = self._get_pipeline(in_doc.format)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/lfoppiano/anaconda3/envs/grobid-align/lib/python3.12/site-packages/docling/document_converter.py", line 370, in _get_pipeline
    self.initialized_pipelines[cache_key] = pipeline_class(
                                            ^^^^^^^^^^^^^^^
  File "/Users/lfoppiano/anaconda3/envs/grobid-align/lib/python3.12/site-packages/docling/pipeline/vlm_pipeline.py", line 78, in __init__
    HuggingFaceMlxModel(
  File "/Users/lfoppiano/anaconda3/envs/grobid-align/lib/python3.12/site-packages/docling/models/vlm_models_inline/mlx_model.py", line 72, in __init__
    self.vlm_model, self.processor = load(artifacts_path)
                                     ^^^^^^^^^^^^^^^^^^^^
  File "/Users/lfoppiano/anaconda3/envs/grobid-align/lib/python3.12/site-packages/mlx_vlm/utils.py", line 297, in load
    model = load_model(model_path, lazy, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/lfoppiano/anaconda3/envs/grobid-align/lib/python3.12/site-packages/mlx_vlm/utils.py", line 142, in load_model
    config = load_config(model_path, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/lfoppiano/anaconda3/envs/grobid-align/lib/python3.12/site-packages/mlx_vlm/utils.py", line 338, in load_config
    raise FileNotFoundError(f"Config not found at {model_path}") from exc
FileNotFoundError: Config not found at /Volumes/ExtremePro/sciencialab/llms/docling

The models seems correctly downloaded:

(base) lfoppiano@MacBook-Air docling % ls
ds4sd--docling-layout-heron		ds4sd--DocumentFigureClassifier		ibm-granite--granite-docling-258M
ds4sd--docling-models			EasyOcr