Docling Performance Issue

[Kunal-Shah107]

I am getting performance issue even if running on big GPU machine, I am sure something is missing.

Here is my setup

VM - Standard NV12ads A10 v5 (12 vcpus, 110 GiB memory) (Azure)
Docker Container - docker run -d --gpus all --name docling -p 5001:5001 -e DOCLING_SERVE_ENABLE_UI=true -e DOCLING_SERVE_MAX_SYNC_WAIT=600 --restart unless-stopped quay.io/docling-project/docling-serve-cu124

API Call - curl --location 'http://:5001/v1alpha/convert/file'
--form 'files=@"c:\workspace\EN.pdf"'
--form 'ocr_engine="easyocr"'
--form 'ocr_language="en,ar"'
--form 'to_formats="md"'
--form 'pdf_backend="pypdfium2"'

Even after having GPU why it is taking a longer, Can some one assist here ! Thanks in advance.

[dosubot[bot]]

Hi @Kunal-Shah107! I'm Dosu and I’m helping the docling team.

A few things could be causing slow performance even with a strong GPU:

1. EasyOCR will only use the GPU if the device is set correctly. Make sure you set the environment variable DOCLING_DEVICE=cuda in your Docker run command. Without this, EasyOCR may default to CPU, which is much slower—even if the container can see the GPU. The correct Docker command would look like:

   docker run -d --gpus all --name docling -p 5001:5001 \
     -e DOCLING_SERVE_ENABLE_UI=true \
     -e DOCLING_SERVE_MAX_SYNC_WAIT=600 \
     -e DOCLING_DEVICE=cuda \
     --restart unless-stopped quay.io/docling-project/docling-serve-cu124
   

2. Check if the GPU is actually being used by running nvidia-smi both on the host and inside the container while a conversion is running. If you don’t see GPU utilization, something in the stack is falling back to CPU.

3. Look at the container logs for warnings about device selection or messages like “CUDA is not available in the system. Fall back to 'CPU'”. This will help confirm if the system is defaulting to CPU due to a misconfiguration or missing driver.

4. There is a known issue with GPU memory not being released after each job in this Docker image. If you’re processing many files, this can cause performance to degrade over time. The recommended fix is to explicitly delete large tensors or models after each job and call torch.cuda.empty_cache() in the cleanup logic. This is already present in the latest code, but if you’re running into memory issues, you may want to monitor GPU memory usage and restart the container if it grows too high source.

5. If you’re still seeing slow performance, try a small PDF with a single language to see if the issue is related to file size or language complexity.

6. Make sure your NVIDIA drivers and CUDA libraries are up to date and properly exposed to the container.

Let me know if setting DOCLING_DEVICE=cuda and checking the logs helps!

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[Kunal-Shah107]

Even after trying it and also changing EasyOCR to RapidOCR, I am not seeing any performance improvement.