The PicturesDescription functionality is not compatible with llama-stack chat API

[rafaeltuelho]

Is there plans to make it possible to integrate the PictureDescription shown here https://github.com/docling-project/docling/blob/main/docs/examples/pictures_description_api.py compatible with llama-stack API?

[dolfim-ibm]

Do you mean instead of using the OpenAI-Compatible API directly?

It can definitely be added. Do you have any example for it or you are willing to contribute it?

[dolfim-ibm]

We have already two implementation of the PictureDescription:

1. Using a local vision model with the transformers AutoModelForVision2Seq, see https://github.com/docling-project/docling/blob/main/docling/models/picture_description_vlm_model.py
2. Using a remote API compatible with the openai-specs (e.g. via vllm), see https://github.com/docling-project/docling/blob/main/docling/models/picture_description_api_model.py

llama-stack could be a simple addition as a third option.

[rafaeltuelho]

Nice!
Thanks for chiming in, @dolfim-ibm
I tried to use the remote API version of PictureDescription pointing it to a llama-stack chat-completion API endpoint, but it failed passing the wrong payload format. Llama stack API uses a different payload schema.

I will take a look at picture_description_vlm_model.py and see what does it takes to make one for llama-stack.

[rafaeltuelho]

Looking at the current implementation for the picture_description_vlm_model.py, the only change to make it work with llama-stack chat-completion API seems to be this part of the request payload.

https://github.com/docling-project/docling/blob/main/docling/models/picture_description_api_model.py#L98-L102

Which is used to send the image.

@dolfim-ibm I believe creating a new picture_description_vlm_model.py child class and overriding this method should be sufficient. What do you think? I can test it and submit a PR.

[dolfim-ibm]

I guess you are referring to picture_description_api_model.py not the picture_description_vlm_model.py (which runs the model locally), right?

From the lines you highlight I think the only difference is the usage of this payload, correct? https://github.com/meta-llama/llama-stack/blob/main/llama_stack/apis/common/content_types.py#L42

We could factory out the generation of the image block of the chat into a private method and overload that one instead of the whole _annotate_images() (assuming the response messages are compatible).

[rafaeltuelho]

I guess you are referring to picture_description_api_model.py not the picture_description_vlm_model.py (which runs the model locally), right?

Yes! Sorry about the confusion :-)

From the lines you highlight I think the only difference is the usage of this payload, correct? https://github.com/meta-llama/llama-stack/blob/main/llama_stack/apis/common/content_types.py#L42

We could factory out the generation of the image block of the chat into a private method and overload that one instead of the whole _annotate_images() (assuming the response messages are compatible).

Actually, I realized the llama-stack API response also differs a bit. So, it is better to implement a new PictureDescription Api model specific to llama-stack chat-completion API. I did a test here and it worked fine. I created this PR adding this capability. Please let me know what do you think.