Bangle.js AI capabilities?

[RKBoss6]

I remember reading somewhere in espruino docs that the bangle.js was 'AI enabled'. I was wondering what that meant, and what exactly that entails. I would like to know if it can run a complex AI model like an LLM (I doubt it, but just asking in case it can), or a smaller gesture model, if it can run models at all. It would be cool to make a gesture app, or another AI app, and I was wondering what are some examples of models that Bangle.js can run, or how one makes an AI model for the Bangle.js.

[banglejsdev]

Gesture recognition, definitely! Check out the following tutorial:

https://www.espruino.com/Bangle.js+EdgeImpulse

One of the reasons I got my Bangles was because it supported Tensorflow Lite for Microcontrollers. But I guess Tensorflow Lite is now LiteRT, and I don't know whether that's supported.

[gfwilliams]

There's an app that you can just install and try it out: https://banglejs.com/apps/?id=gesture

We made it really easy to do the gesture recognition once you have a model on the device: https://github.com/espruino/BangleApps/blob/master/apps/gesture/gesture.js

But you can use it more explicitly too: https://www.espruino.com/Reference#tensorflow

Honestly though there's been basically zero interest over the last 5 years, as you can tell from the lack of apps using it. I'd thought at least someone might have made a launcher using it

I guess Tensorflow Lite is now LiteRT, and I don't know whether that's supported.

I think it's just a rebrand - the same models should work fine

[RKBoss6]

Also, just out of plain curiosity, what kind of launcher were you envisioning? Where would the AI come into play there? Gesture recognition to open apps was what I was envisioning, but that would get tedious very quickly for training...

[gfwilliams]

I was imagining there might be a model with a bunch of standard gestures built in, and then you could have 'twirl your wrist to open stopwatch', 'clap hands 3 times to start recording' or similar.

... but honestly pattern launcher does a good job using the touchscreen, and I think it's just an easier, more reliable input method

[RKBoss6]

I installed the 'gestures' app, I am now trying to make my own gesture model, maybe to add a feature to messages. In the new apple watches, you can quickly rotate the wrist to dismiss notifications, that might be a good feature for Bangle. Thanks guys!

[gfwilliams]

I just merged an app from @thyttan that does something very similar :) espruino/BangleApps#3866

[RKBoss6]

Thanks! Will definitely try that out! :)

[thyttan]

There's also this one: https://espruino.github.io/BangleApps/?q=msgwakefup

[RKBoss6]

One thing I am not really clear about: For creating your own gesture mode, when collecting data from bangle, can the data only be what the 'gestures' app defines as a gesture? Ex: Only swipe right, clap, swipe left. Is there a way to make a completely new gesture?

[RKBoss6]

Ah never mind, It should work for any gesture right?

[gfwilliams]

Yes - for gestures you have a .tfnames file (https://www.espruino.com/Bangle.js+EdgeImpulse#deployment) which maps each probability output of the tensorflow model to a gesture name. So when a gesture goes through, suppose the 4th element in the output vector is highest, the 4th entry in .tfnames is chosen as the gesture name

[gfwilliams]

... or you can ignore that and look directly at the weights variable - see https://www.espruino.com/Reference#l_Bangle_aiGesture

[RKBoss6]

Alright. Do you have any idea on how to implement this into apps? I had an idea where you could twist the watch to lock it, or when a notification is open, dismiss the notification first. How exactly would I do this, would I need a boot.js code, or can it work through apps?

[gfwilliams]

It could work either way. An app could integrate a handler for the aiGesture event (that'd be good for your apps - I'm not sure about adding this into the built-in ones).

... or if for example you want a gesture to emulate 'back' functionality then boot code is probably a good call. It could for instance search for an instance of the back widget and simulate a click on it when it got the aiGesture event, and then it'd work whenever an app displayed the red back icon.

[thyttan]

I guess https://espruino.github.io/BangleApps/?id=backswipe could be a good reference for something like that.

[RKBoss6]

Thanks! I think that's a good idea for me to see how it works under the hood.

[RKBoss6]

Do you guys have any ideas for possible gestures you think would be handy for bangle? (eg. twist the wrist outward to dismiss notifications, lock.) I can't think of many, and it's good to get others' gestures they want as well.

[gfwilliams]

When I was originally doing this, the trick was really in trying to get movements that everyone did the same way. It's amazing how you say "wave your hand" and different people do it different ways - it's worth thinking about.

... also for it to be useful you really need to record gestures from several different people. If it's just you, it'll learn to do your gestures just great but it may well not work for others :)

[RKBoss6]

Also, for putting the app on the app store, how do you add the model? In the gestures app, it's added as a js file that looks to be compressed. Do you need to do that, or can you just pop the model into github, add it in metadata, and let the bangle do the rest?

[gfwilliams]

It's literally just the file as-is, like you load in using the IDE at https://www.espruino.com/Bangle.js+EdgeImpulse#deployment

[RKBoss6]

Oh ok, thanks so much!

[RKBoss6]

@gfwilliams Just out of questioning, how would I get the atob(....) from that file that I can then upload as a javascript file?

[gfwilliams]

The file that you uploaded? print(btoa(require("Storage").read(".tfmodel")))

[RKBoss6]

@gfwilliams thanks! We were having some trouble with the raw file, so I might use the js instead...