Add AI Agent PDF Input Info #336
Conversation
|
|
|
Looks mostly good! One small point of clarification that @MaggieThomann will investigate (added her as a reviewer) — I believe some AI models require text parts in all messages (so can't just upload a photo for example, needs to include text). This has been a point of confusion for users so I think it's worth calling out. I'll let Maggie take a look. Thanks! 🙏 |
|
|
||
| Selecting multiple input types makes it easier for users to clearly communicate what they need. Instead of relying only on text descriptions, users can combine inputs—for example, uploading an image along with text to better illustrate their queries and help the agent provide more accurate responses. | ||
| Selecting multiple input types makes it easier for users to clearly communicate what they need. Instead of relying only on text descriptions, users can combine inputs. For instance, in an AI Stylist agent, enabling both Text and Image allows users to either describe their outfits in words or upload clothing photos for personalized analysis. |
There was a problem hiding this comment.
Can you add a caveat here that users typically need to include something in the text prompt to the agent?
or upload clothing photos for personalized analysis.
^ Basically, it would technically not be sufficient to do this in the OpenAI or Google cases. They would also need to put something in the "Text input" field in the "Send Message" action. Otherwise the agent will complain. Anthropic functions differently though. Anthropic is OK with just accepting the image.
There was a problem hiding this comment.
Actually correction -- all vendors require that Text is passed. Seeing this error from Anthropic now:
{
"error": {
"details": {
"message": "400 {\"type\":\"error\",\"error\":{\"type\":\"invalid_request_error\",\"message\":\"messages: text content blocks must be non-empty\"}}",
"details": "Error: 400 {\"type\":\"error\",\"error\":{\"type\":\"invalid_request_error\",\"message\":\"messages: text content blocks must be non-empty\"}}"
},
"message": "Error running assistant",
"status": "INTERNAL"
}
}
There was a problem hiding this comment.
Ok sorry for the back and forth 😅 @pinkeshmars
I'm going to make a change to the agent that will allow the user the ability to just send an image in their request by defaulting the text parameter that we send in the cloud function code to an empty string. This way, the API won't complain and the user can configure the agent with just image. So feel free to keep this language as is.
There was a problem hiding this comment.
cool. Thanks for the clarification @MaggieThomann
|
Hi @MaggieThomann could you please approve this so that we can merge? |
|
Hi @MaggieThomann if it looks good to you, could you please approve? |
There was a problem hiding this comment.
I'm approving this since this is already released, but @MaggieThomann let us know if there were any additional comments.
Description
Add AI Agent PDF Input Info
Linear ticket and magic word Fixes DEVR-898
Type of change