Espruino JavaScript
A JS smartwatch + ChatGPT, this is the future guys! #6561
Replies: 1 comment
-
|
Posted at 2024-01-11 by @thyttan Could you point ChatGPT to all available Bangle.js/Espruino documentation now when doing initial prompts setting up? I think I tried that a while back but without too much luck. It was kind of cool using it as an aid, but not super helpful yet. I guess mainly since it probably didn't/couldn't parse websites and I'm not sure the docs would fit within the token limit. Posted at 2024-01-11 by grandrew "Point to" - no, still not enough compute/context But doing smart prompting will definitely do it. I would approach it like this:
Even better results will be with changing current applications - like changing / inventing new watch faces by modifying code Posted at 2024-01-12 by @gfwilliams If it gets to that point it would be very cool! I feel like you've almost got two types of programming - you've got one where you have a task and there's some real problem-solving involved in trying to figure out how to make a computer do the thing you want. Then you've got the other one where the task is quite straightforward, but you just have to figure out what names of functions to call and in what order - and normally you'd be spending all your time consulting API references, Google and StackOverflow I reckon the first one, I'm not sure an AI will help much with at the moment - but the second one it's already semi-useful at and will end up being awesome. I'm sure I saw something a few months ago where someone had the AI actually rigged up to Python and it was able to try things out for itself and iterate? I guess if it were possible to actually connect the AI to your Bangle so it could run the code it created, it could iterate itself and might actually be able to come up with something working in its first answer Posted at 2024-01-12 by grandrew These two actually intersect a lot as there is no fine line between "what calls to call and in what order" and "make computer do the thing you want". And technically, they are the same and called A.I. planning and program synthesis and the difference is probably in the grounding level of the generated code- either AI can write calls with unrolled loops call-by-call, or create higher-order logic to describe the program with flow control primitives. Lifting the logic is computationally expensive though. Where this is becoming really promising is that Bangle.js provides a confined and simple enough "world model" for the LLM to be able to handle it much more efficiently than the "open-world" approach of generic Python code writing.
I believe that if automatic programming will ever happen "in a mass consumer product" it will happen here, in the watch applications, first. Or at least in the first batch of breakthroughs. I don't believe multi-shot (feedback from interpreter) approach is feasible in general, but might improve the accuracy by some significant percentage points. The problem is in the cost of it - while trying a couple attempts might be OK, running thousand run/feedback shots of LLM is extremely expensive to do just to throw away all the missed results. The problem of writing larger applications can be solved by fine-tuning the LLM by providing more examples of the code, and/or training a world model using some kind of metaheuristic-guided exploration on a simulator and then mass-training on that simulator in a GAN-like approach to get a corpus of feasible working programs. Still orders of magnitude cheaper to do than open-world problem. As a product development "feature" (if you allow me to wildly hypothesize), one could collect the feature requests from users that the LLM was not able to code in one-shot approach, and then use those to re-train the LLM model so that these "featureless features" will become gradually available for one-shot implementation by user request. Posted at 2024-01-12 by @thyttan What do you think about [Open Interpreter] (https://www.youtube.com/watch?v=xPd8FFzIeOw) , could it be used together with the Espruino Command-line tool to automate development? (I don't have experience with the Command-Line tool myself) Posted at 2024-01-13 by grandrew I'm not familiar with Open Interpreter, looks cool, but from what I can see in the video it is more about controlling the machine with commands that are "interpreted" by LLM, while we do not have this capacity on the watch. We need to write very small code, then store and make launchable & usable. It would be interesting to look inside, they seem to have some neat ideas and a lot of experience with feeding just enough context for it to work even on open source models. Btw, ChatGPT seems to know Bangle.js already and can write code without pre-feeding context. Posted at 2024-01-14 by @thyttan Last night I let a gpt4 assistant I set up as 'Espruini Guide' start writing a work logger app I first thought up on the App Ideas conversation (I probably could have done a better job in this setup phase, adding more documentation or something). The first bump in the road was it tried using a 'append' storage method that espruino doesn't have. I just gave it the error from the console and it then switched to a write call which worked. It created the file with the first line, but then it could not be appended to. I told it this but it struggled to figure a correct solution. Then I asked something like 'but isn't there another way to write to storage as described in the hardware reference?', and with that it figured out it should be using the 'open' method. With that I had a working proof of concept! I've since let it implement a menu with options to add annotations to a work session, as well as creating and switching between different tasks (each writing to separate files). It struggled with clearing and setting up input handlers when moving between parts of the ui, so I fixed that myself, mostly without prompting 'Espruino Guide'. It needed some help with adhering to system theme as well. Pretty neat, but not entirely trivial. Edit: Added the code, screenshots and a log with entries and annotations. Edit2: Now saving my progress here: https://github.com/thyttan/BangleApps/blob/worklog/apps/worklog/app.js Attachments:
Posted at 2024-01-14 by grandrew This is super cool! I will see if I can enhance this process and get back Posted at 2024-01-14 by @thyttan One thing I think I noticed, is in order to keep the cost of api calls down I can clear the chat and reinitialize the 'Espruino Guide', pasting the current iteration of the full code in the first user prompt (I do it inside Posted at 2024-01-14 by @thyttan The setup (see attachment): Instructions:
Attachments: Posted at 2024-01-16 by grandrew So I created the RSS reader app with ChatGPT, not without issues though. Where it failed was:
I believe it would have been doable in one shot if I fed it ahead of time the API nuances with tiny use case examples, specifically the "rough edges" like unsupported RegExp. I can definitely create a neat "GPTs" demo case where it writes the app in one shot. Full transcript to the point where I understood it won't fix itself: https://chat.openai.com/share/18d6df03-7af0-41da-a660-f419c1abd686 Final RSS reader app attached. Attachments: Posted at 2024-01-16 by grandrew ... and also Ember BLE mug temperature reader that I wrote with ChatGPT Attachments: Posted at 2024-01-16 by @thyttan Maybe we could compile a list of stuff it gets wrong together with instructions/nudges to what it should do instead? And that list could be included in the setup phase? Posted at 2024-01-16 by grandrew Yes that was exactly my thinking. We should also think of compatible method to deploy & publish those apps, as regular DevOps doesn't make sense here as presumably the app is defined by English not code. Posted at 2024-01-16 by @thyttan
I'm not sure I follow. Do you suggest we store the prompt in english on the BangleApps repository instead of the code we get from working with the gpt-model? Isn't a good intermediate solution to make the gpt-assistant as competent as possible by having good initial setup prompts, and then when working app code has been generated we store that as an app on the BangleApps repo like for all other apps? Another question I have not thought about is copyright and licences when it comes to apps created this way. Can they safely be added to the repo, @gfwilliams? (I guess this could be an argument for storing the prompts rather than the generated code @grandrew. I just feel like it's much harder to get it to be reliable...) Posted at 2024-01-16 by grandrew I suggest to experiment with completely different UX Like,
it's just and idea and still needs a lot of polishing and figuring out the details but I guess high-level properties of this are smth like:
Posted at 2024-01-17 by @thyttan That sounds like a cool end point! To me it feels like a big task getting all those steps to work. Doesn't mean it can't be done of course, but I would think it makes sense to start with getting it to reliably create app code in as few shots as possible. When that's nailed down it would be really cool to look at those other things though. My two cents! Posted at 2024-01-17 by grandrew I agree. So there are two options:
I'm for option 1 - although this "end user" is probably still a developer enthusiast but personally for me it is exactly the fun I'm looking for, and it shoots directly into the vast space of unknowns of what the future interface would look like. Posted at 2024-01-17 by @thyttan Just coming back to Open Interpreter, the point I meant to make wasn't to get Open Interpreter to run on the Bangle. But rather to let Open Interpreter run on the computer - creating, running and debugging apps in a automated fashion. Posted at 2024-01-17 by @thyttan I experimented some today with having a setup where I advice agains stuff, like trying to use a I get the sense it will be much harder to give it a bigger instruction and expect good results. It seemed to perform better the way I did it the first time - starting with a small and simple description and iterating in smaller steps to arrive at my vision for the worklog app. Posted at 2024-01-18 by grandrew As Open Interpreter is just an alternative (and weaker) version of ChatGPT, I would probably see using open interpreter as using a Motorola 6800 when Intel 8086 was already available... Posted at 2024-01-18 by @thyttan It depends on the LLM you chose to use under the hood. As I understand it you can configure it to do api calls to OpenAI GPT-4. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Posted at 2024-01-11 by grandrew
I'm still processing this, but it is insane
Imagine a watch that has no functions at start.
You describe functions in English, ChatGPT writes code.
I was able to combine ChatGPT with code examples and it wrote me almost-working code to read and display current tea temperature from my Ember smart mug over BLE. For smaller programs it would just write working code directly, potentially with no user interaction needed, today.
Here is how it works:
I believe it can be automated to the point of just asking for specific functionality and it writes code. Maybe it will need emulator feedback to fix errors, and some additional documentation trickery - like asking it to summarize documentation first and paste that summary as initial prompt so it is more aware about custom function call patterns.
This is absolutely stunning. Wow. I have the smartwatch with all imaginable functions at once.
Beta Was this translation helpful? Give feedback.
All reactions