o1 image input doesn't work with custom instructions

[andresgit]

What happened?

Using o1 with custom instructions and image input the model output ignores the image as if it doesn't see it, see the image below. But if I don't use custom instructions, it works. With 4o or 4.5 for example it works fine with custom instructions.

I think it is because with o1 for some reason it doesn't use the system message for custom instructions, but starts editing the actual user message and that somehow removes the image. With 4o or 4.5 it uses the system message for instructions, maybe o1 can also use the system message for instructions?

I am using fileStrategy: "firebase"

Version Information

ghcr.io/danny-avila/librechat-dev:latest@sha256:f39eca6a09e326235c3acc1283c114764bf0f3c8235e9fb4033d9e8f583a69ed

Steps to Reproduce

See above

What browsers are you seeing the problem on?

No response

Relevant log output

Screenshots

Code of Conduct

• I agree to follow this project's Code of Conduct

[danny-avila]

LLM issue? Provider/proxy issue? Can't reproduce it.

Logs:

{
  "baseURL": "https://api.openai.com/v1",
  "level": "debug",
  "message": "[OpenAIClient] chatCompletion",
  "modelOptions": {
    "messages": [
      {
        "content": [
          {
            "text": "what do you see",
            "type": "text"
          },
          {
            "image_url": {
              "detail": "auto",
              "url": "data:image/png;base64,..."
            },
            "type": "image_url"
          }
        ],
        "role": "user"
      }
    ],
    "model": "o1",
    "stream": true,
    "user": "USER_ID"
  },
  "timestamp": "2025-03-17T20:44:50.300Z"
}
{"level":"debug","message":"Making request to https://api.openai.com/v1/chat/completions","timestamp":"2025-03-17T20:44:50.302Z"},
{
  "choices": [
    {
      "finish_reason": "stop",
      "index": 0,
      "logprobs": null,
      "message": {
        "annotations": [],
        "content": "It appears to be a stylized black-and-white logo made up of interlocking geometric shapes arranged in a circular pattern. The design resembles the well-known symbol often associated with OpenAI and ChatGPT.",
        "refusal": null,
        "role": "assistant"
      }
    }
  ],
  "created": 1742244290,
  "id": "chatcmpl-BCBcIPCD733e7UGS0CGhT4nN3FKoA",
  "level": "debug",
  "message": "[OpenAIClient] chatCompletion response",
  "model": "o1-2024-12-17",
  "object": "chat.completion",
  "service_tier": "default",
  "system_fingerprint": "fp_60442b0df0",
  "timestamp": "2025-03-17T20:45:03.080Z",
  "usage": {
    "completion_tokens": 819,
    "completion_tokens_details": {
      "accepted_prediction_tokens": 0,
      "audio_tokens": 0,
      "reasoning_tokens": 768,
      "rejected_prediction_tokens": 0
    },
    "prompt_tokens": 265,
    "prompt_tokens_details": {
      "audio_tokens": 0,
      "cached_tokens": 0
    },
    "total_tokens": 1084
  }
}

[andresgit]

bumping, bug is still there

You didn't reproduce it because you didn't use custom instructions.

In the logs it seems the image gets replaced by [object Object]

2025-03-19T10:45:40.631Z debug: [OpenAIClient] chatCompletion
{
  baseURL: "https://api.openai.com/v1",
    modelOptions.model: "o1",
    modelOptions.user: "66b9fed0b315cf7dba3edca5",
    modelOptions.stream: true,
    // 1 message(s)
    modelOptions.messages: [{"role":"user","content":"write capital letters\n[object Object],[object Object]"}],
}

[andresgit]

It doesn't work if you use custom instructions, like in the first post screenshot.

Without custom instructions it does work.