Using OpenAI API to create an AI assistant for our IoT projects

[jgpeiro]

Hello All,

I have found a method to use OpenAI Codex as AI assistant for our IoT projects.
Perhaps that we can access to OpenAI API via urequests with a simple code and convert any IoT device to an cool AI assistant.
The code to make a request is this:

import urequests as requests
import ujson as json

def codex( key, prompt, max_tokens, stop ):
    url = "https://api.openai.com/v1/completions"
    headers = {
        "Content-Type": "application/json",
        "Authorization" : "Bearer " + key
    }
    data = {
        "model":       "code-davinci-002",
        "prompt":      prompt,
        "temperature": 0.5,
        "max_tokens":  max_tokens,
        "top_p":       1,
        "frequency_penalty": 0.0,
        "presence_penalty":  0.25,
        "stop":        [stop]
    }
    resp = requests.post( url, headers=headers, data=json.dumps(data) )
    if( resp.status_code != 200 ):
        print( "resp.status_code != 200", resp.status_code )
    text = resp.json()["choices"][0]["text"]
    finish_reason = resp.json()["choices"][0]["finish_reason"]
    return text

You need your own API key, but the price is too low for any personal use (0.02$/1K tokens)

A typical exampe can be like this one, where you provide a promp with a "casual" HAL definition and then ask things to the assistant and execute the response as micropython code via eval/exec/import

It can handle common micrpython machine classes, timers, network, sockets, asyncio, framebuf or even asyncio. And if you have a custom API, you can simply add in the conversation as example.

For ESP32 devices with enought RAM, a Google voice-to-text can be used:

import requests
import base64
import json

def recognize( key, fl_wav_name ):
    url_recognize = "https://speech.googleapis.com/v1/speech:recognize?key=" + key
    buf = open( fl_wav_name, "rb" ).read()
    headers = { "Content-Type": "application/json" }
    data = {
        "config": {
            "encoding": "LINEAR16",
            "sampleRateHertz": 8000,
            "languageCode": "es-ES"
            #"languageCode": "en-US"
        },
        "audio": {
            "content": base64.b64encode( buf ).decode( "utf-8" )
        }
    }
    resp = requests.post( url_recognize, headers=headers, json=data )
    print( "resp.status_code", resp.status_code )
    transcript = resp.json()["results"][0]["alternatives"][0]["transcript"]
    #transcript = "hello world"
    return transcript

Another usage example is this one, where a complete new pong game is created every minute (about 200 lines of python code) and then executed on the second core of the PICO_W. A video and prompt used is shown here:
https://www.youtube.com/watch?v=TgRffMAOubQ

# The game class includes "show_instructions" method, witch is shown at the start of the program.
# The program includes a call to the main function.
# A print("done") marks the fishish of the program source code.
# The game staus is shown in the top-left part of the screen, in red color.
# If the A and B buttons are pressed at the same time, a debug mode is entered. Here all objects prints its coordinates on top of them when they are rendered.
# The game includes the following classes:
#     Paddle: This class represents a paddle. It has a position and a size.
#     Ball: This class represents the ball. It has a position and a velocity.
#     Pong: This class represents the game itself. It has two paddles and a ball. It knows how to check if the game is over.

import time
import math
import random
import machine
import st7789_fb

time.sleep(2)

sw_a  = machine.Pin( 15, machine.Pin.IN, machine.Pin.PULL_UP )
sw_b  = machine.Pin( 17, machine.Pin.IN, machine.Pin.PULL_UP )
joy_u = machine.Pin(  2, machine.Pin.IN, machine.Pin.PULL_UP )
joy_d = machine.Pin( 18, machine.Pin.IN, machine.Pin.PULL_UP )
joy_l = machine.Pin( 16, machine.Pin.IN, machine.Pin.PULL_UP )
joy_r = machine.Pin( 20, machine.Pin.IN, machine.Pin.PULL_UP )
joy_c = machine.Pin(  3, machine.Pin.IN, machine.Pin.PULL_UP )

bl = machine.Pin( 13, machine.Pin.OUT )
bl.value( 1 )

lcd = st7789_fb.LCD()
"""
class LCD:
    red, green, blue, white, black, yellow, cyan, magenta, gray, darkgray and lightgray
    def width( self ):
        return 240
    def height( self ):
        return 135
    def pixel( self, x, y, c ):
        ...
    def rect( self, x, y, w, h, c ):
        ...
    def fill_rect( self, x, y, w, h, c ):
        ...
    def line( self, x0, y0, x1, y1, c ):
        ...
    def circle( self, x, y, r, c ):
        ...
    def fill_circle( self, x, y, r, c ):
        ...
    def text( self, txt, x, y, c ):
        ...
"""
def show_welcome_message():
    lcd.fill( lcd.black )
    lcd.text( "Press A to start the game!", 10, 20, lcd.white )
    lcd.show()
    while( joy_c.value() == 1 ):
        time.sleep_ms( 10 )

running = True
def main():
    global running
    show_welcome_message()
    show_instructions_message()
    
    pong = Pong()
    while running:
        if( sw_a.value() == 0 and sw_b.value() == 0 ):
            pong.debug = not pong.debug
        
        if( sw_a.value() == 0 ):
            event = "A"
        elif( sw_b.value() == 0 ):
            event = "B"
        elif( joy_u.value() == 0 ):
            event = "U"
        elif( joy_d.value() == 0 ):
            event = "D"
        elif( joy_l.value() == 0 ):
            event = "L"
        elif( joy_r.value() == 0 ):
            event = "R"
        else:
            event = ""
        
        pong.update(event)
        lcd.fill(0)
        pong.draw()
        lcd.show()
        
        time.sleep_ms( 1 )

class Pong:

[jgpeiro]

I forgot to comment here, but few weeks ago I uploaded the finished toy project to hackaday.io

As said, it can record voice via i2s microphone, transcribe and translate via Whisper model (or Google API) and then convert your requests to micropython code via Codex API.

The project documentation
https://hackaday.io/project/188303-esp32-ai-assistant

The project source code
https://github.com/jgpeiro/esp32_ai_assistant