Real-time Playback Slowdown with Multiple Stacked Plots, Throttle Rate 100

[altsv]

Hello,

When running multiple stacked plots in real-time playback on the openmct-quickstart docker container with throttleRate set to 100 in the config, we run into the following issue:

https://streamable.com/6l41as

At around 5 minutes, the interface begins to lag and the real-time clock ticks slow down.
Is there a way to prevent this slowdown? Or does Open MCT not support real-time playback of this many telemetry sources at this speed? We recorded this with the memory tab open in DevTools for your reference. Please let me know if I should provide any more information!

Thank you so much,
Simon

[ozyx]

Hi Simon! Thanks for reporting this. The video was very helpful, so thanks for that as well.

We have some changes related to telemetry batching and throttling coming in openmct-yamcs 4.0.0 that should directly address this issue. @akhenry can speak to this in a bit more detail.

When running multiple stacked plots in real-time playback on the openmct-quickstart docker container with throttleRate set to 100 in the config

By this do you mean you're altering the simulator script to send telemetry at 100Hz?

[altsv]

Hi @ozyx ,

Thank you so much for taking the time to respond!
We set the throttleRate to 100ms in the openmct-quickstart's config like so in openmct-quickstart.js to match the 10hz rate being generated by the simulator:

const installYamcsPlugin = window.openmctYamcs;

const config = {
    "yamcsDictionaryEndpoint": "http://localhost:8040/yamcs/",
    "yamcsHistoricalEndpoint": "http://localhost:8040/yamcs/",
    "yamcsWebsocketEndpoint": "ws://localhost:8040/yamcs/api/websocket",
    "yamcsUserEndpoint": "http://localhost:8040/yamcs/api/user/",
    "yamcsInstance": "myproject",
    "yamcsFolder": "myproject",
    "throttleRate": 100
};

[altsv]

@ozyx @akhenry
Hello, we tried out the most recent openmct-yamcs 4.0.0-next version, and there has been a visible improvement in performance due to the new batching changes which is great!

A video of the improved performance with 3 stacked plots can be found here: https://streamable.com/rnf2gh

When we add one more stacked plot, however, it starts lagging and dropping multiple telemetry points: https://streamable.com/gdhxu3

This is with a 10hz throttle rate (throttleRate set to 100) with a batch size of 20. The console logs are shown for your reference.

Is there something on our end we can do to improve performance? Or does Open MCT not support playback for this many sources at a 10hz speed? Please let me know if you need any more info. Thank you!

[akhenry]

@akhenry

[akhenry]

Hi @altsv,

We have made some changes to the telemetry API, and the concept of "batching" has changed to a more traditional buffering approach. The buffer size is specified in characters, with a default of one million characters (1000000). It is specified like so:

        const realtimeTelemetryProvider = new RealtimeProvider(
            openmct,
            configuration.yamcsWebsocketEndpoint,
            configuration.yamcsInstance,
            configuration.yamcsProcessor,
            configuration.throttleRate, // Default is 1000
            configuration.maxBufferSize // Default is 1000000
        );

I have done some testing with stacked plots on the latest Open MCT and openmct-yamcs masters on modest hardware (a 2019 Macbook Pro) and the performance was satisfactory. If you're still working on this project, please do try the latest Open MCT and Open MCT Yamcs repositories, and share videos again if you're still having issues.

Lots of factors can affect performance. A couple of thoughts -

• Plot performance will be better with a GPU, otherwise we fall back on software rendering. We have had some users utilizing desktop streaming from a cloud service where the cloud machine did not have a GPU, which impacted performance.
• If you see a lot of "Telemetry dropped" messages, try increasing your maxBufferSize
• The Chrome Dev tools window causes signficant performance degradation all of the instrumentation creates CPU overhead and memory leaks. It's great for developing, but does not give you performance that is representative of a production environment