audio stream between 2 esp32

[Schneckensuppe]

Hello everyone,

I'm looking to pass audio data between two ESP32s. The hadware installation is as follows, on each esp32 is connected: an INMP441 microphone and a PCM5102a DAC audio output. Both are connected in I2S. I followed @miketeachman 's tutorials to achieve this. (thanks!!)
On the software side, it's written in micropython, which is why I'm on this forum.
I tried using espnow asynchronously. The result is not conclusive: the sound seems metallic and a bit choppy. I played with the parameters, in particular ibuf, and implemented a buffer. But that didn't solve the problem. I tried UDP with the same results.
I wonder if the error is due to the implementation of my buffer or to the WIFI transmission speed on esp32 (the rate of esp32 cannot be modified from micropython, so it is blocked at 1 Mb/s.
What do you think?
have any of you managed to pass audio from a microphone to a loudspeaker via wifi/espnow between 2esp32 with acceptable sound quality? Do you have any documentation to help me?
Does this seem more feasible to you in C? (I don't know C, but if I have to I can try) Do I need to build my own micropython like here?

Thanks in advance to all of you

Here is the code with the espnow which works in reception and transmission::

import time
import asyncio
from machine import Pin, I2S
import network
import aioespnow

sta = network.WLAN(network.STA_IF)
sta.active(True)

esp = aioespnow.AIOESPNow()
esp.active(True)
peer = b'MAC_ADR'
esp.add_peer(peer)

#### I2S config micro ####
SCK_PIN_MIC = 13
WS_PIN_MIC = 14
SD_PIN_MIC = 34
I2S_ID_MIC = 0
BUFFER_LEN_MIC = 4_096

#### I2S config Haut Parleur ####
SCK_PIN_HP = 33
WS_PIN_HP = 25
SD_PIN_HP = 32
I2S_ID_HP = 1
BUFFER_LEN_HP = 4_096


audio_out = I2S(
    I2S_ID_HP, sck=Pin(SCK_PIN_HP), ws=Pin(WS_PIN_HP), sd=Pin(SD_PIN_HP),
    mode=I2S.TX, bits=16, format=I2S.MONO, rate=8000, ibuf=BUFFER_LEN_HP)  # 16 ko/s

audio_in = I2S(
    I2S_ID_MIC, sck=Pin(SCK_PIN_MIC), ws=Pin(WS_PIN_MIC), sd=Pin(SD_PIN_MIC),
    mode=I2S.RX, bits=16, format=I2S.MONO, rate=8000, ibuf=BUFFER_LEN_MIC) # 16 ko/s


#### led de clignotement ####
TEMOIN_CLIGNOTEMENT = 2
print("debut")

async def clignote(led):
    led = Pin(led, Pin.OUT)
    while True:
        led.on()
        await asyncio.sleep(0.5)
        led.off()
        await asyncio.sleep(0.5)
        
async def mic_to_esp(audio_in, esp):
    chunk = 512
    nb_slot_on_chunk = 10
    esp_chunk = 128
    usr_buffer = bytearray(chunk * nb_slot_on_chunk)
    usr_buffer_mv = memoryview(usr_buffer)
    
    sreader = asyncio.StreamReader(audio_in)
    
    num_bytes_read_from_mic = 0
    num_bytes_send = 0
    
    while 1:
        num_bytes_read_from_mic += await sreader.readinto(usr_buffer_mv[num_bytes_read_from_mic:])
        for i in range(num_bytes_read_from_mic // esp_chunk):
            await esp.asend(peer, usr_buffer[i * esp_chunk: (i+1) * esp_chunk])
            num_bytes_send += esp_chunk
        num_bytes_read_from_mic = num_bytes_read_from_mic - num_bytes_send
        usr_buffer_mv[:num_bytes_read_from_mic] = usr_buffer_mv[num_bytes_send:num_bytes_send + num_bytes_read_from_mic]
        num_bytes_send = 0
             
        
async def esp_now_to_hp(audio_out, esp):
    chunk = 128
    nb_slot_on_chunk = 4 # attention 1 ou 2 suvant si t'es une fangeo ou pas
    usr_buffer = bytearray(chunk * nb_slot_on_chunk)
    usr_buffer_mv = memoryview(usr_buffer)
    swriter = asyncio.StreamWriter(audio_out)
    while 1:
        for i in range(nb_slot_on_chunk):
            _, usr_buffer_mv[i*chunk: (i+1)*chunk ] = await esp.arecv() # choix
        swriter.out_buf = usr_buffer_mv
        await swriter.drain()


async def main(audio_in, audio_out, esp, led):
    print("debut 2")
    asyncio.create_task(clignote(led))
    asyncio.create_task(mic_to_esp(audio_in, esp))
    asyncio.create_task(esp_now_to_hp(audio_out, esp))
    while 1:
        await asyncio.sleep(0)
        
asyncio.run(main(audio_in, audio_out, esp, TEMOIN_CLIGNOTEMENT))

and Here is the code with udp on transmit (udp on receive below):

import network
import socket
import asyncio
from machine import I2S, Pin

SSID = 'my_ssid'
pwd = 'pswrd'

PEER = 'peer_IP'
ME = 'my_ip'
PORT = 5005
PEER_ADDR = (PEER, PORT)

sta = network.WLAN(network.STA_IF)
sta.active(True)

while not sta.isconnected():
    sta.connect(SSID, pwd)
    time.sleep(0.1)
    
print("connected to ", SSID)

#### I2S config micro ####
SCK_PIN_MIC = 13
WS_PIN_MIC = 14
SD_PIN_MIC = 34
I2S_ID_MIC = 0
BUFFER_LEN_MIC = 4_096
RATE = 8000
CHUNK = 1024
#### I2S config micro ####

#### led de clignotement ####
LED = 2
#### led de clignotement ####

audio_in = I2S(
    I2S_ID_MIC, sck=Pin(SCK_PIN_MIC), ws=Pin(WS_PIN_MIC), sd=Pin(SD_PIN_MIC),
    mode=I2S.RX, bits=16, format=I2S.MONO, rate=RATE, ibuf=BUFFER_LEN_MIC) # 16 ko/s


async def clignote(led):
    # print("debut de clignote")
    led = Pin(led, Pin.OUT)
    while True:
        led.on()
        await asyncio.sleep(0.5)
        led.off()
        await asyncio.sleep(0.5)


async def mic_to_udp(s, audio_in, peer):
    sreader = asyncio.StreamReader(audio_in)
    
    chunk = 512
    nb_slot_on_chunk = 2
    
    num_bytes_read_from_mic = 0
    num_bytes_send = 0
    
    usr_buffer = bytearray(chunk * nb_slot_on_chunk)
    usr_buffer_mv = memoryview(usr_buffer)
    
    while 1:
        for i in range(nb_slot_on_chunk):
            num_bytes_read_from_mic += await sreader.readinto(usr_buffer_mv[num_bytes_read_from_mic:num_bytes_read_from_mic + chunk ])
    
        num_bytes_send = s.sendto(usr_buffer_mv[:num_bytes_read_from_mic], peer)
        num_bytes_read_from_mic = 0
        num_bytes_send = 0
        await asyncio.sleep_ms(0)
        

async def udp_to_hp():
    pass

async def main(led, audio_in, addr_peer):
    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
    
    asyncio.create_task(clignote(led)) 
    mic = asyncio.create_task(mic_to_udp(sock, audio_in, addr_peer))
    #hp = asyncio.create_task(web_to_hp())
    
    while True:
        await asyncio.sleep(0)
        
try:
    asyncio.run(main(LED, audio_in, PEER_ADDR))
except (KeyboardInterrupt, Exception) as e:
    print(f"Exception {type(e).__name__} {e}\n")
finally:
    ret = asyncio.new_event_loop()

and udp on receive:

import network, socket, asyncio
from machine import Pin, I2S

#### I2S config Haut Parleur #### config HP
SCK_PIN_HP = 33
WS_PIN_HP = 25
SD_PIN_HP = 32
I2S_ID_HP = 1
BUFFER_LEN_HP = 4096

audio_out = I2S(
    I2S_ID_HP, sck=Pin(SCK_PIN_HP), ws=Pin(WS_PIN_HP), sd=Pin(SD_PIN_HP),
    mode=I2S.TX, bits=16, format=I2S.MONO, rate=8000, ibuf=BUFFER_LEN_HP) # 16 ko/s

ssid = "my_ssid"
pswrd = "my_pswrd"
peer = "peer_ip"  
me = "my_ip"
sta = network.WLAN(network.STA_IF)
sta.active(True)
sta.connect(ssid, pswrd)
print(sta.ifconfig())

BLINK_LED = 2

sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
bd = sock.bind((me, 5005))

async def clignote(led):
    print("debut de clignote")
    led = Pin(led, Pin.OUT)
    while True:
        led.on()
        await asyncio.sleep(0.5)
        led.off()
        await asyncio.sleep(0.5)

async def udp_to_hp(audio_out, udp_buffer, sock):
    nb_slot_buffer = 2
    usr_buffer = bytearray(nb_slot_buffer * udp_buffer)
    usr_buffer_mv = memoryview(usr_buffer)
    swriter = asyncio.StreamWriter(audio_out)
    while 1:
        data, _ = sock.recvfrom(udp_buffer)
        swriter.out_buf = data
        await swriter.drain()
    
    
async def main(audio_out, udp_buffer, sock, led):
    asyncio.create_task(clignote(led))
    asyncio.create_task(udp_to_hp(audio_out, udp_buffer, sock))
    while 1:
        await asyncio.sleep(0)
        
asyncio.run(main(audio_out, 1024, sock, BLINK_LED))

[peterhinch]

Your code looks good. I wrote this demo of a music player which plays stereo WAV files stored on an SD card. It uses similar techniques to cope with the fact that reading from disk is discontinuous (albeit by using much larger buffers). I think the problem is that radio communication is also discontinuous. My gut feeling is that ESPNow is probably a better choice than WiFi (less contention), but I don't know if anyone has quantified the throughput attainable.

There are two possibilities:

1. ESPNow cannot maintain a mean 16Kbps throughput.
2. ESPNow can do this easily, but with gaps in communication.

The second possibility could be handled with a larger buffer on the receiver. It's also possible that a larger transmit buffer is required: it's possible that esp.asend pauses while communication is established, during which time the I2S buffer overflows.

Applications of this type need some experimentation and testing to achieve continuous throughput.

The other aspect is minimising allocation. You've clearly given this some thought, but you might get some other ideas from my example. You might also want to consider an asynchronous iterator for receiving ESPNow messages.

[peterhinch]

I did some tests to determine the bandwidth of an asynchronous ESPnow link. The best I could achieve, in minimal test scripts, was about 17,000 bytes/sec. This is very close to your required 16,000 bytes/sec. I don't think this rate would be achievable in a practical application. If the receiver fails to keep up with the transmitter, messages will be lost. The overhead in actually doing something with the received data would almost certainly cause this to happen.

[Schneckensuppe]

Firstly, thank you for your prompt replies.
I have several questions, perhaps you can answer them.
I thought it was possible to pass audio between several esp32 with espNow (I've also done some speed tests and I'm getting about 16kbps, like you). I have read in the esspressif documentation that the speed could reach 1 Mbps, I don't understand why we have such a difference, do you have any clues or insights to enlighten me?
Is it possible that the micropython implementation can slow the speed so much?
On the other hand, I've tried with larger buffers and got a higher latency, which makes sense, but the sound is still metallic, similar to the smaller buffer setting. So more latency but no improvement in sound. I concluded (rightly or wrongly) that the audio_out i2s buffer was under-nourished. What do you think?

[peterhinch]

To answer your questions would require delving into the source, unless someone emerges who already knows it. I think you're breaking new ground with a high bandwidth application. My experience with ESPNow is at the opposite end of the scale, where a device wakes up from deepsleep occasionally, sends an MQTT message, and dozes off again.

It's highly likely that the audio_out i2s buffer is under-nourished as your bit rate is so close to the MicroPython limit that we've both measured.

I'm not sure what to suggest. Options are:

1. Reduce the bit rate (probably unacceptable).
2. Investigate how MP deals with ESPnow and propose improvements.
3. Consider using C.

[glenn20]

1Mbps is the raw bit rate of the wifi transmission. ESPNow throughput rates are also impacted by the limited packet size for espnow traffic, congestion and importantly, the built in wait for ack on send. You can get much fasdter throughput with reduced reliability by setting sync=False in ESPNow.send() (not dissimilar from UDP). In fact, when sending with sync=False, the send is sufficiently fast that you can just use send() without the overhead of await asend().

[glenn20]

Oh, and another thing to try is to set a larger buffer for holding incoming espnow messages (ESPNow.config(rxbuf=XXX)), especially if you are sending with sync mode off. You can monitor the packet loss rate on the sender and receiver with ESPNow.stats().

[peterhinch]

@Schneckensuppe An alternative approach to ESPnow or UDP would be to use a socket, kept permanently open. This would avoid any UDP overhead. The StreamReader and StreamWriter methods support sockets using select.poll to achieve concurrency. An alternative approach would be nonblocking sockets, as used here but I would start with the stream approach: it seems well suited to the application.

[Schneckensuppe]

thank you @peterhinch , it's going to take me some time to understand the logic and the libraries that I don't yet understand, I'll make some test programs and I'll keep you informed of the results.