Suggestion: real-time transport mode (AsyncApi?)

[rinode]

Just to provide some extra context and avoid misunderstanding: this suggestion concerns high frequency updates of graphics properties while it's on air and if OGraf should play a role here.

Last meeting got me thinking... @axelboberg raised a very interesting point about transport, whether we have ever considered using WebSockets. Generally, HTTP/3 (using QUIC, based on UDP) should provide a boost in performance over 'traditional' HTTP over TCP, with only one round-trip for the handshake and lost packets being non-blocking. However... Have we ever benchmarked this, especially for high framerate scenarios?

Should the need arise for low-latency communication, would it be an idea to adopt AsyncApi, but using the OpenApi schema? See: https://www.asyncapi.com/docs/tutorials/getting-started/coming-from-openapi, they seem quite clear about this:

AsyncAPI is compatible with OpenAPI schemas.

Perhaps this is a nice performance profiling/benchmark task, a cross-comparison of framerates and transport modes:

• Framerates: 23.976, 25fps, 29.97fps, 30fps, 50fpx, 59.94fps, 60fps, 120fps
• Transport: REST over HTTP/2 (TCP), REST over HTTP/3 (QUIC/UDP), WebSockets (TCP), WebTransport (HTTP/3)

Would love to hear your input! Let's get technical 🤓

[didikunz]

I don't know, if there is a misunderstanding, what the whole thing is about. The Server-API is used to control the play-out of graphic templates.

I don't know, if you are familiar with CasparCG There is the AMCP-protocol that does something similar. It's based on TCP-IP socket communication. You use it to tell the server, what and when you want to play something. You do not call it frame by frame. As the HTML-template itself usually needs a short time to load itself, that would not make sense. The idea is, that an operator loads a template (lower third etc.) a few seconds before it needs to go on air, and then plays it on cue. The communication latency is the least problem here. But usually this works quite well and timing is seldom an issue.

[axelboberg]

What you're saying @didikunz is true for simple graphics such as nameplates and titles. However whenever something needs to be updated on a frame-by-frame basis this starts becoming more interesting.

Some examples:

• A template rendering a clock from an external input such as a sports event
• Multiple redundant servers that need to depend on an external state
• Templates relying on positional data from external sources

Sure you can still do this with AMCP but then you'd have to abandon the standard. And even though you may have a faster transport the load command is still relevant.

[rinode]

I guess it comes down to whether those properties are controllable from an OGraph perspective, or should be considered "internally controlled" and not exposed through the protocol. It does raise questions about what the real-time mode in the OGraph spec actually entails – and more importantly: what not.

Regarding the misunderstanding @didikunz: this is exactly the answer I gave @axelboberg regarding the "purpose" of the Server API last meeting. But it doesn't completely rule it out.

In the end, if the API is not built for this, it might be good if we advise against this in its documentation.

[didikunz]

I am not completely against it, but most of the mentioned stuff is not done via the server API. For instance a clock is usually done with some internal logic, that either count on it's own, use the system clock or connect to a timing source directly.

For stuff like football or basketball the latency of these protocols are usually fast enough to set the game clock (or shot clock) from the server. For instance the ScoreboardOCR app does exactly this (not yet for OGraf).

[rinode]

For example, let's take this graphic that @nytamin shared:
#47

If the positions of those labels were to be provided by an external tracking system, sending updates every frame while the graphic is on-air... Would we say that's out of scope for OGraf and should be handled internally by the playout system itself? Or do we perhaps want to explore the possibilities and limitations here?

[axelboberg]

That is a good example. The question whether or not this is in scope of OGraf requires more knowledge of the intentions of the protocol than I currently have.

My initial thoughts however, and seeing this as a way to standardize how graphics templates are controlled both in live settings and offline (non real time), is that I fear lacking a good real time transport would force developers out of the standard for way too little or no reason.

[nytamin]

This is a quite interesting idea!
There was an intention when designing the OGraf API to make it versatile enough so most vendors wouldn't have to resort to secondary data channels when implementing their graphics in OGraf (that's why we included the "v_" prefix, for vendor-specific extensions of the API).

I can see a few possible options:

• Discard the REST API entirely, in favor of a websocket one (a very breaking change. Still allowed at this point, but would require much work and likely delay the release of the Server API).
• Keep the REST API, but also add a functionally equivalent one based on websockets (AsyncApi based on the same OpenApi schema). (A lot of work, but could be added in a later version)
• Add a SINGLE websocket API endpoint for the "update" method only (since that will likely be the one most commonly used for realtime updates). (Least amount of extra work)

---

Just out of curiosity; to see if a REST API implementation can be capable of handling realtime data updates I ran a quick test locally using OGraf Server, pushing data updates at 60 fps to a local renderer and filmed it in slow-motion:

VID_20260305_210047616.mp4

It looks like most_ of the data comes through in time for each frame to get updated, so that's pretty good.
The calls took about 3-5ms, and that includes a roundtrip of the message being passed from Controller to Server to Renderer and back again.
But this was a local test, perhaps network overhead would be higher over a typical network..

[rinode]

Certainly not an easy choice.

Discarding the REST API feels radical, but I would prefer that over maintaining two API's if we treat this as a priority. I do fear that it might affect the adoption of OGraf though.

We were developing a combined OpenAPI / AsyncAPI protocol when I worked at EVS (for Cerebrum), I'll check in with them to see how that's working out. It might help us to decide whether it's worth going down that road. I do recall some of your colleagues at SuperFly implementing that protocol @nytamin, would be good to hear about their experiences.

[dvdeurse]

Interesting discussion, but let's not forget our audience and Tuomo's infamous 80-20 rule. Why do we specify a schema for each parameter? Because implementers of controllers can then render them without prior knowledge. Why do you want to render them? So users can control it. In most of the cases, updating graphics will be triggered by manual interaction. Does that mean we prevent automated updates? Of course not. Most of the automated updates will not be every frame either, on the contrary. So I want to highlight that this case is relevant, but is only about a small portion of everyday's graphics updates.

Let's focus on these high-frequency update use cases. I see two categories:

• fully automated, no user interaction at all: the clock example falls in this category, as well as an algorithm that tracks players on a pitch and provides the bounding boxes. So for these use cases, what is the benefit to create an OGraf parameter for this? You will never have to visualize it in a controller. The only thing I see is that you expose a consistent way for external systems to 'push' this data into the template. But should we use (or misuse?) OGraf for that? You could easily build and define your own API (in whatever technology) and use it inside your template, you would obtain the same consistency.
• user-driven: for example a user manipulating a (physical or virtual) joystick, a T-bar, a jog to steer certain parameter values. You could still tackle this in the same way as the previous point (i.e. build an API where these controls are talking to and push them into your graphics via your preferred transport). However, I see more reasons that this category could benefit from being a parameter because it is possible to show them in the controller and to interact with it as a user. A noticeable difference here is that the interaction is not full time, but only when the user actually does something. However, when the user does something, it generate high-frequency updates.

So in my opinion, only user-driven high-frequency updates are valid use cases for this topic. That said, what are the options if we want to render based on high-frequency updates:

1. not make it a parameter at all, and implement the communication channel inside the template implementation. This solution will work in every compliant renderer and what's more, the implementer can choose the best protocol that suite his/her needs. You have access to all the options available and are not restricted to what the OGraf spec supports today (whatever that may be).
2. introduce a WebSocket endpoint for updates only (and I assume we keep the one in the REST API as well). A valid proposal although annoying for vendors that don't need this feature. You could make it optional but many optional things inside a spec is not great for the overall interoperability. We should think about whether it is worth to annoy the 80% in order to please the 20% :-).

For me the other options like fully replacing the API with an Async API or maintaining two APIs would be enormous overkill and a bad signal towards the vendors that focus on the 'simple' use cases.

[rinode]

Why do we specify a schema for each parameter? Because implementers of controllers can then render them without prior knowledge. Why do you want to render them? So users can control it.

I think it all boils down to whether this is indeed the focus, as you could also treat it as an abstraction layer for graphics control where you don't care whether it's controlled by a human, or automated and at what frequency. We did a show once where the position of a graphic was based upon multi-touch dragging on a touchscreen: high-frequency, user-driven, but causing a massive burst of updates that you probably don't want to wire through OGraf.

It would be good to formalize the intent of the protocol, to prevent anyone treating it like such an abstraction layer and building full-blown automated data connection workflows on top of it, if it's simply not built for it.

Just because you can, doesn't mean you should 😊

[dvdeurse]

Well, the use case does make sense, and can even be done with HTTP as Johan demonstrated. Since this was a local test, I assume this was HTTP 1.1? From HTTP2 and further, things can only get better in this context :). So we shouldn't prevent or discourage users from doing this, if it fits their requirements, why not. But I agree that for users that need to go beyond the boundaries of static HTTP requests, we can (and should) provide guidelines if we decide not to introduce an OGraf WebSocket endpoint.

I don't think this is about 'intents', but about how to achieve interoperability across multiple vendors without ruling out certain use cases, but at the same time make adoption as easy as possible.