[vertex-ai] fix: allow object as tool content #1279
Conversation
![matter-code-review[bot]](https://avatars.githubusercontent.com/in/474549?s=60&v=4){kind=small}
| (typeof message.content === 'string' || | ||
| typeof message.content === 'object') |
There was a problem hiding this comment.
🛠️ Code Refactor
Issue: The type check could be more robust for object validation
Fix: Consider validating that object content is not null and has expected structure
Impact: Prevents potential runtime errors with malformed object content
| (typeof message.content === 'string' || | |
| typeof message.content === 'object') | |
| (typeof message.content === 'string' || | |
| (typeof message.content === 'object' && message.content !== null)) |
There was a problem hiding this comment.
Hey @arturfromtabnine
Openai chat completion content is always a string
| message.role === 'tool' && | ||
| typeof message.content === 'string' | ||
| (typeof message.content === 'string' || | ||
| typeof message.content === 'object') |
There was a problem hiding this comment.
OpenAI spec only supports sending text content or array of text content parts as part of tool message content https://platform.openai.com/docs/api-reference/chat/create#chat_create-messages
I believe what you are trying to do is to send an object like this
{
"role": "user",
"parts": [
{
"functionResponse": {
"name": "get_current_weather",
"response": {
"temperature": 20,
"unit": "C"
}
}
}
]
}instead of
{
"role": "user",
"parts": [
{
"functionResponse": {
"name": "get_current_weather",
"response": { "content": "{'temperature': 20, 'unit': 'C'}"}
}
}
]
}the current implementation is definitely incomplete, what we should be doing is
else if (
message.role === 'tool' &&
typeof message.content === 'string'
) {
let toolResponse;
try {
toolResponse = JSON.parse(message.content);
} catch (e) {
toolResponse = {
content: message.content,
};
}
parts.push({
functionResponse: {
name: message.name ?? 'gateway-tool-filler-name',
response,
},
});
} There was a problem hiding this comment.
we need to handle array of text contents as well, which we can change in a separate PR
There was a problem hiding this comment.
You're right, we need to do a strict check for array, but your example is not what we are trying to send. Our client is open ai compatible, so its trying to send an array in message.content
{
role: 'tool',
tool_call_id: 'call_ty2u3gJiEIsf9TLOwUB67opO',
content: [
{
type: 'text',
text: '{"filePath"}'
}
]
}
with suggested change there no issue, vertex api for gemini models allows to have response.content as an array or string
There was a problem hiding this comment.
Not sure if we need to parse content like was suggested
typeof message.content === 'string'
) {
let toolResponse;
try {
toolResponse = JSON.parse(message.content);
} catch (e) {
toolResponse = {
content: message.content,
};
}
Hey @narengogi it can be an array as well |
But it's an array of text content parts, you need an object for vertex right? |
Sorry, was off content: [
{
type: "text",
text: "content",
},
],will update suggested condition to be more precise on array instead of being too generic |
|
DescriptionSummary By MatterAI
🔄 What ChangedThe condition 🔍 Impact of the ChangeThis modification resolves an 📁 Total Files Changed
🧪 Test AddedManual testing was performed to verify the fix. 🔒Security VulnerabilitiesN/A - No new security vulnerabilities were introduced or detected. MotivationAllow tool content to be a typeof object for google models on vertex (which is compatible with open ai api https://github.com/openai/openai-node/blob/40b79cb8a372dce925b14a26e621110f9de6786ba/src/resources/chat/completions/completions.ts#L1407). Otherwise if content is a typeof object Type of Change
How Has This Been Tested?
Screenshots (if applicable)N/A Checklist
Related Issueshttps://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling Tip Quality Recommendations
Tanka Poem ♫
Sequence DiagramsequenceDiagram
participant User
participant Gateway
participant GoogleChatComplete as Google Chat Complete Provider
participant VertexAI as Vertex AI API
User->>Gateway: Send Chat Message (with tool output)
Gateway->>+GoogleChatComplete: processMessage(message: { role: "tool", content: any, name?: string })
Note over GoogleChatComplete: Handles message with role 'tool'
GoogleChatComplete->>GoogleChatComplete: Construct functionResponse
Note over GoogleChatComplete: Sets functionResponse.response.output = message.content (object or string)
GoogleChatComplete->>+VertexAI: POST /v1/projects/.../locations/.../publishers/google/models/...:generateContent
VertexAI-->>-GoogleChatComplete: API Response (Success or Error)
GoogleChatComplete-->>-Gateway: Formatted Response
Gateway-->>User: Chat Response
|
|
Important PR Review SkippedPR review skipped as per the configuration setting. Run a manually review by commenting /matter review 💡Tips to use Matter AICommand List
|
|
Important PR Review SkippedPR review skipped as per the configuration setting. Run a manually review by commenting /matter review 💡Tips to use Matter AICommand List
|
|
Important PR Review SkippedPR review skipped as per the configuration setting. Run a manually review by commenting /matter review 💡Tips to use Matter AICommand List
|
There was a problem hiding this comment.
I've removed the check entirely and updated the key from content to output as mentioned in the vertex spex
https://cloud.google.com/vertex-ai/docs/reference/rest/v1/Content#FunctionResponse
Description
Allow tool content to be a typeof object for google models on vertex (which is compatible with open ai api https://github.com/openai/openai-node/blob/40b79cb8a372dce925b14a26e62110f9de6786ba/src/resources/chat/completions/completions.ts#L1407). Otherwise if content is a typeof object,
rolewould not be handled properly and request will fail with following error:https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling
Motivation
Type of Change
How Has This Been Tested?
Screenshots (if applicable)
Checklist
Related Issues