How to run using vLLM with a docker image?

[pearsonkyle]

Question

Is there an optimal way to run this using vLLM serve? I don't think my outputs are correct

Code

LLM Hosting:

#!/bin/bash
CACHE_DIR="/data/model_cache/"

model="models--ibm--granite-docling-258m"
served_name="granite-docling"

HOST=0.0.0.0
PORT=10070

docker run --rm --gpus '"device=0"' \
    -v /data/model_cache:/data/model_cache \
    -p ${PORT}:${PORT} \
    vllm/vllm-openai:v0.11.0 \
    --model ${CACHE_DIR}/${model} \
    --served-model-name ${served_name} \
    --max-model-len 8192 \
    --gpu_memory_utilization 0.5 \
    --port ${PORT} \
    --host ${HOST}

Test:

import base64
import mimetypes
import os

def image_to_base64_data_url(image_path):
    if not os.path.isfile(image_path):
        raise FileNotFoundError(f"Image file not found: {image_path}")
    
    mime_type, _ = mimetypes.guess_type(image_path)
    if not mime_type or not mime_type.startswith('image/'):
        mime_type = 'image/jpeg'
    
    with open(image_path, 'rb') as image_file:
        encoded_data = base64.b64encode(image_file.read()).decode('utf-8')
    
    return f"data:{mime_type};base64,{encoded_data}"


from openai import OpenAI

client = OpenAI(
  api_key='none',
  base_url="http://localhost:10070/v1"
)

model_id = "granite-docling"

messages =  [{
    "role": "user",
    "content": [
        {
            "type": "text", 
            "text": "Convert this page to docling."
        },
        {
            "type": "image_url",
            "image_url": {
                "url": image_to_base64_data_url("test.png")
            }
        }
    ]
  }]

response = client.chat.completions.create(
    model=model_id, 
    messages=messages
)

print(response.choices[0].message.content)

Outputs:

<loc_12><loc_177><loc_242><loc_287><loc_22><loc_31><loc_129><loc_37>Table 2. Summary of Classifiers
<loc_258><loc_175><loc_297><loc_189><loc_22><loc_305><loc_232><loc_351>Figure 9. We test the sensitivity of our deep nets to data outside the range we trained it on. We generate 77760 light curves for each noise value. We find that the size of the transit depth does not influence the accuracy. Instead, the ratio of transit depth to noise dictates the accuracy of each detection algorithm. Based on this plot we can estimate the number of light curves required to significantly detect a planet below the noise by binning data together.
<loc_23><loc_13><loc_141><loc_21>102 K. A. Pearson et al.
<loc_22><loc_144><loc_144><loc_168>(Specificities like photometric ranges ∀>0: ∼4.4 and regular spectra and luminance domain) of details mentioned in the article and figures:
<loc_118><loc_44><loc_380><loc_141>BLSSVMMLPCNN 1DWavelet MLPInput features180180180180Trainable Parameters318113,93717,293Layers1454Total Neurons1105109105Neural Connections249225442494Training Accuracy (%)73.591.0899.7299.60Training False Pos.(%)22.343.050.080.21Training False Neg.(%)4.105.850.200.19Sensitivity Test (%)63.1483.1088.7388.45Test False Pos. (%)31.582.920.290.25Test False Neg. (%)5.3713.9810.9711.29
<loc_22><loc_357><loc_237><loc_467>presents the same model (examples in Table 3 with base proportions among laboratories). Pedestrian motion, like cartilage, and the ability to capture articulations of the brain were represented as a function of input pain responses. Both single eyes and relevant network behaviours relying on the left occipital lobe the same number of lights are simple features [2]. Parameter
<loc_254><loc_160><loc_465><loc_194>Length and Rotatory Unravelation
<loc_254><loc_191><loc_481><loc_318>Duration in which the incident lights have repeated stores. The observation that light spikes across Photoshop images has been found in Dr.

It ends up missing a lot of text and doesn't format tables correctly. Is there a better way to run this using vLLM without the python package or is that the only way?

[dosubot]

Hi @pearsonkyle! I'm Dosu and I’m helping the docling team.

There are known issues with incomplete outputs, duplicated content, and incorrect formatting (especially for tables) when using vLLM serve with the granite-docling model for multimodal (image+text) tasks. This is not unique to your setup—other users have reported similar problems, including the presence of internal structure tokens like <loc_*> and missing or poorly formatted tables. These issues often stem from how the backend (vLLM or similar) handles the model's output schema and special tokens, and not all endpoints fully support the required features for these models [discussion][issue].

For best results, try these approaches:

• Use the skip_special_tokens parameter if your client and backend support it, but be aware that not all vLLM endpoints respect this flag.
• Post-process the output to strip unwanted tokens (like <loc_*>) using regex or string replacement.
• Adjust your prompt to explicitly request "plain markdown, no tags or location markers."
• When possible, request output in DOCTAGS format rather than markdown, as the docling pipeline can convert DOCTAGS to markdown more reliably [issue].
• For table extraction, VLMs like granite-docling can help, but complex tables may still require manual post-processing or prompt tuning. The extraction quality depends on the model and prompt, and some edge cases remain challenging [discussion].

You do not need to use the Python package specifically—any client that can send OpenAI-compatible requests to your vLLM endpoint should work. However, the Python SDK makes it easier to set parameters like skip_special_tokens and handle multimodal payloads correctly.

Recent updates in docling-serve and vLLM integration have improved batching and backend support, but the core limitations with output formatting and special token handling remain [pull request].

If you need more reliable output, consider post-processing or using a backend/server that is confirmed to fully support the granite-docling model's output schema and special token handling.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}