I want to use Ollama llava to describe some pictures

[LJ-Hao]

So I write code as below:

def describe_images(image_path):
    image_documents = SimpleDirectoryReader(image_path).load_data()
    mm_model = OllamaMultiModal(model="llava")
    mm_model.complete(prompt='summerize the image',image_documents=image_documents)
    mm_model.chat()

but I got error as below:

root@jiahao:/data/Multimodal-RAG-on-Jetson# python3 mulitRAG.py 
Traceback (most recent call last):
  File "mulitRAG.py", line 56, in <module>
    describe_images(image_path)
  File "mulitRAG.py", line 44, in describe_images
    mm_model.complete(prompt='summerize the image',image_documents=image_documents)
  File "/usr/local/lib/python3.8/dist-packages/llama_index/multi_modal_llms/ollama/base.py", line 160, in complete
    response = self._client.generate(
  File "/usr/local/lib/python3.8/dist-packages/ollama/_client.py", line 126, in generate
    return self._request_stream(
  File "/usr/local/lib/python3.8/dist-packages/ollama/_client.py", line 97, in _request_stream
    return self._stream(*args, **kwargs) if stream else self._request(*args, **kwargs).json()
  File "/usr/local/lib/python3.8/dist-packages/ollama/_client.py", line 73, in _request
    raise ResponseError(e.response.text, e.response.status_code) from None
ollama._types.ResponseError: an unknown error was encountered while running the model 

So how can I solve this problem

[dosubot[bot]]

To solve the ollama._types.ResponseError: an unknown error was encountered while running the model error, follow these steps:

1. Ensure Correct Initialization: Make sure the OllamaMultiModal class is correctly initialized with the appropriate parameters.

2. Check the Base URL and Model Name: Verify that the base_url and model parameters are correctly set. The default base URL is http://localhost:11434.

3. Verify the API Server: Ensure that the Ollama API server is running and accessible at the specified base_url.

4. Validate Input Data: Ensure that the image_documents parameter is correctly formatted as a sequence of ImageDocument objects.

5. Handle Additional Parameters: Check if any additional parameters passed through additional_kwargs are correct and supported by the Ollama API.

Here is an example of how to initialize and use the OllamaMultiModal class correctly:

from llama_index.multi_modal_llms.ollama import OllamaMultiModal
from llama_index.core.readers.file.base import SimpleDirectoryReader

def describe_images(image_path):
    # Load image documents
    image_documents = SimpleDirectoryReader(image_path).load_data()
    
    # Initialize the OllamaMultiModal class
    mm_model = OllamaMultiModal(
        model="llava",
        base_url="http://localhost:11434",  # Ensure this matches your setup
        temperature=0.75,
        context_window=DEFAULT_CONTEXT_WINDOW,
        request_timeout=30.0  # Optional: specify a timeout
    )
    
    # Complete the prompt with image documents
    try:
        response = mm_model.complete(prompt='summarize the image', image_documents=image_documents)
        print(response.text)
    except ollama._types.ResponseError as e:
        print(f"An error occurred: {e}")

# Example usage
image_path = "/path/to/your/images"
describe_images(image_path)

Ensure that the base_url and model parameters are correctly set to match your setup. Additionally, verify that the Ollama API server is running and accessible. If the error persists, you may need to check the server logs for more detailed error information ^[1][2].

Details

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[LJ-Hao]

Thank you for your help, but it does not solve my problem.

[LJ-Hao]

Yes It works, maybe one floder can only put one picture.

[LJ-Hao]

here is the code I solve this problem:

class translate_image_to_text:
    def __init__(self, image_path, text_path):
        self.image_path = image_path
        self.text_path = text_path
        self.response = ''
    def get_image_path(self):
        image_folder = self.image_path
        image_files = [os.path.join(image_folder, f) for f in os.listdir(image_folder) if os.path.isfile(os.path.join(image_folder, f))]
        return image_files
    def image_to_text(self):
        mm_model = OllamaMultiModal(model='llava')
        image_file_names = self.get_image_path()
        
        for image in image_file_names:
            print(image)
            self.response += str(mm_model.complete(prompt='summarize the image', image_documents=[ImageDocument(image_path=image)]))
        print(self.response)

hope It will help someone who meet the same problem