Profile current resource utilization and improve performance

[peddybeats]

Currently the performance of the application is borderline -- if we do a bit more work per inference, we'll definitely start to slow down some more limited devices.

First we have to profile to understand where the cycles are going. How much compute resources is each of handpose, facemesh, inference, drawing utilities, and other utilities consuming?

To make things slightly more complicated, we have a few backends that all the models could be executing on: CPU, WASM, WebGL - Read more about it here

Ideas on how to improve performance:

• instead of running per frame redraw (here) run at fixed interval (once a second?)

[amirhmk]

Another idea on improving performance could be introducing a threshold on displacements from the face before running the inference model?

• WebGL looks pretty cool and it seems like it can buy us time, would be good to look at

[amirhmk]

I think the idea of frame-by-frame could be good, it may be better suited if we were to predict a user is about to touch their face by tracking the route?

[Berkmann18]

I think the idea of frame-by-frame could be good, it may be better suited if we were to predict a user is about to touch their face by tracking the route?

Yeah, using a bit of physics to predict that could be a good idea, but I can see some (potential) caveats to that:

• will it be more efficient and faster than doing the frame-by-frame way?
• will it be accurate and account for edge cases where the hand would change direction right before touching the face?
• how would the two above play out if the direction/speed/... of the face's movements were taken into account?