Requests with images fail often

[andresgit]

What happened?

I am using my phone camera to take an image and make a request to chatgpt-4o-latest, but often it fails while saying in the logs

2024-12-25T12:32:28.601Z debug: Making request to https://api.openai.com/v1/chat/completions
2024-12-25T12:32:28.719Z debug: [saveConvo] api/app/clients/BaseClient.js - saveMessageToDatabase #saveConvo
2024-12-25T12:32:31.775Z warn: [OpenAIClient.chatCompletion][stream] API error
2024-12-25T12:32:31.775Z error: [handleAbortError] AI response error; aborting request: 400 Timeout while downloading https://firebasestorage.googleapis.com/v0/b/libre-951c9.appspot.... [truncated]
2024-12-25T12:32:31.780Z warn: AI Response error for conversation dba9e402-a87f-4c1f-ba30-bbb7f7a747c7 likely caused by Google censor/filter
2024-12-25T12:32:31.900Z debug: [AskController] Request closed

See also the screenshot. I am using the firebase file storage option.
It seems that openai very frequently returns errors to image requests. For example this code:

from openai import OpenAI

api_key = "sk-proj-AAAAAAAA"
client = OpenAI(api_key = api_key)

for i in range(10):
  try:
    response = client.chat.completions.create(
      model="gpt-4o",
      messages=[
        {
          "role": "user",
          "content": [
            {"type": "text", "text": "What's in this image?"},
            {
              "type": "image_url",
              "image_url": {
                "url": "https://firebasestorage.googleapis.com/v0/b/libre-951c9.appspot.com/o/images%2F676ad9eeb1ab2f9f12e44885%2Fd4775eeb-a9de-45b8-822b-43081bc0f611__17351243526206805673596199865694.png?alt=media&token=c363bf92-4764-4beb-afc9-4d85fda2f90f",
              }
            },
          ],
        }
      ]
    )
    print(f"{i} SUCCESS: {response.choices[0].message.content[:60]}")
  except Exception as e:
    print(f"{i} ERROR: {e.status_code}, {e.type}, {e.code}, {e.body['message'][:60]}")

gives me this output:

0 ERROR: 400, invalid_request_error, invalid_image_url, Timeout while downloading https://firebasestorage.googleapis
1 ERROR: 400, invalid_request_error, invalid_image_url, Timeout while downloading https://firebasestorage.googleapis
2 ERROR: 400, invalid_request_error, invalid_image_url, Timeout while downloading https://firebasestorage.googleapis
3 ERROR: 400, invalid_request_error, invalid_image_url, Timeout while downloading https://firebasestorage.googleapis
4 ERROR: 400, invalid_request_error, invalid_image_url, Timeout while downloading https://firebasestorage.googleapis
5 SUCCESS: The image shows a close-up of a wooden surface with a light 
6 SUCCESS: The image shows a wooden surface with a natural grain patter
7 SUCCESS: This image shows a close-up of a wooden surface with a light
8 ERROR: 400, invalid_request_error, invalid_image_url, Timeout while downloading https://firebasestorage.googleapis
9 SUCCESS: The image shows a wooden surface, such as a tabletop or desk

Maybe if there is a timeout error, there could be a button right below the error to try again?

Steps to Reproduce

1

What browsers are you seeing the problem on?

No response

Relevant log output

No response

Screenshots

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

I’ve never had these issues with images from firebase provided to OpenAI. And this wouldn’t be a bug on our end if it can be reproduced in a small script like yours.

But in any case, you can use the local option to send base64. Also maybe we can provide an option to use base64 from firebase images, as is done for Google and Anthropic since they don’t support links. PRs welcome.