Extending the standard by multi-modal capabilities

[lLostincode]

Hello community,

I really like the idea of having a standardized protocol so thanks for the initiative!
The AG UI protcol acted as aa rough outline for my multi-agent WebSocket communications channel

Having adopted mostly to that - have some remarks / feedback on things i did / had to do differently than provided from the standard

1. Building Multi-Agent Systems using LangGraph, i find the "Frontend first" approach too far off i.e. the AbstractAgent definition doesnt fit into that picture. I have my own Agent definition, but still the messages, events & tool calling definitions are helpful
2. Speaking of events, the current standard has (Agent) Runs as the "highest execution layer" and Steps as the second highest. When it comes to running agents in parallel or after another, one needs a definition one level higher. (In my case: System Execution -> Agent Runs -> Agent Steps)
3. Lack of pre-defined HITL-scenarios (thats alright, since we can build on top of the Custom event), but this was actually one of the few things i expected AG UI to have, since that is so "undefined" over most frameworks
4. Multi-Modal Messages: The User / Assistants messages are only bound to string-based communication. In times of providing AI Systems with images or pdf files, this needs to be enhanced properly in my opinion.

Not exactly an issue, but a ticket for discussion, let me know what you guys think about adopting those points in the standard.
Im happy to exchange