Pdf read and analysis for gemini-1.5-flash

[lalineanegra]

I'm trying to get a pdf summarized using gemini-1.5-flash, but the OpenAI client only accept a url to the document, example:

client = openai.OpenAI(
    api_key = token,   # pass litellm proxy key, or google token if custom app enabled
    base_url = api_base # litellm-proxy-base url
)

response = client.chat.completions.create(
    model = "gemini-pro",
    messages = [
           {
            "role": "user",
            "content": "gs://cloud-samples-data/generative-ai/pdf/2403.05530.pdf"
            },
            {
            "role": "user",
            "content": "summarize pdf content"
            }
        ]
)

But for the gemini-1.5-flash model, the output is:

...
content='I do not have access to the internet or specific files, including the PDF you mentioned. Therefore, I cannot provide any information or key takeaways from the document "
...

Using the vertexai client, a valid example would be:

import vertexai

from vertexai.generative_models import GenerativeModel, Part

# TODO(developer): Update and un-comment below lines
# project_id = "PROJECT_ID"

vertexai.init(project=project_id, location="us-central1")

model = GenerativeModel(model_name="gemini-1.5-pro-preview-0409")

prompt = """
You are a very professional document summarization specialist.
Please summarize the given document.
"""

pdf_file_uri = "gs://cloud-samples-data/generative-ai/pdf/2403.05530.pdf"
pdf_file = Part.from_uri(pdf_file_uri, mime_type="application/pdf")
contents = [pdf_file, prompt]

response = model.generate_content(contents)
print(response.text)

Is there a way to make this compatible with the OpenAI example, and therefore, with litellm?