Binary over USB/serial

[stephanelsmith]

On ESP32S3 (not sure if the port matters), I'm transfering binary over USB to my application. I'm having two issues which I think should have simple configuration issues which I can't seem to find. My goal is to update firmware, ota partition update, without going into bootloader. I already have this working over wifi, but want USB update option as well. Should be staight forward, right?

I'm sending binary data over USB and reading in the bytes via streamreader. Something like:

sreader = asyncio.StreamReader(sys.stdin)
sreader.readinto(mv)

I have kbd_intr(-1) set to prevent KeyboardInterrupt.

As a test, transferring 256 bytes (eg. bytes([x for x in range(256)])) in a go, in I receive the bytes sent with two artifacts:

1. Value 13 is being automatically converted to 10 (auto CR->LF conversion). I tried sending 13 on both Windows and Linux machine, results the same, so I'm fairly confident it's on embedded side. Where would one find the configuration that's manipulating the CR->LF? This is very annoying, I'd like to avoid base conversion, it shouldn't be necessary.
   I've found the following ESP-IDF settings added to sdkconfig, which has no effect.

CONFIG_NEWLIB_STDIN_LINE_ENDING_CR=n
CONFIG_NEWLIB_STDIN_LINE_ENDING_CRLF=n
CONFIG_NEWLIB_STDIN_LINE_ENDING_LF=y
CONFIG_NEWLIB_STDOUT_LINE_ENDING_CR=n
CONFIG_NEWLIB_STDOUT_LINE_ENDING_CRLF=n
CONFIG_NEWLIB_STDOUT_LINE_ENDING_LF=y
CONFIG_NEWLIB_STDIN_LINE_ENDING_CR=n

2. I also think there's some RX buffer sensitivity. In a single trasfer, I get 251 bytes sent correctly. If I add pauses, or chunk, etc... I get all my bytes, presumably giving the RX side enough time to drain the buffer. I found the following setting which has no effect.
   CONFIG_USB_CDC_RX_BUFSIZE=512
   Is there an RX buffer I can expand?

I'm hoping someone has run into this as it's driving me a bit nuts!

[stephanelsmith]

For anyone else that expects to read the same bytes that were sent over stdio:

Here's the culprit for item 1. In shared/runtime/sys_stdio_mphal.c, read method. Commenting out the \r to \n conversion will fix it.
#11898

STATIC mp_uint_t stdio_read(mp_obj_t self_in, void *buf, mp_uint_t size, int *errcode) {
    sys_stdio_obj_t *self = MP_OBJ_TO_PTR(self_in);
    if (self->fd == STDIO_FD_IN) {
        for (uint i = 0; i < size; i++) {
            int c = mp_hal_stdin_rx_chr();
            if (c == '\r') {
                c = '\n';
            }
            ((byte *)buf)[i] = c;
        }
        return size;
    } else {
        *errcode = MP_EPERM;
        return MP_STREAM_ERROR;
    }
}

[robert-hh]

Did you try sys.stdin.buffer.read() and sys.stdout.buffer.write()?

[stephanelsmith]

First, thank you for the response. First yes, it works. Now that I've found shared/runtime/sys_stdio_mphal.c I can see it.

I'm using asyncio, this seems to do the trick:
asyncio.StreamReader(sys.stdin.buffer)

From there on out, my readinto's are working as expected.

Still feels quite quirky as I would expect readintos to straight up copy byte for byte, but this will do.

[jimmo]

@stephanelsmith readinto/read is just about whether it reads into a buffer or returns the bytes.

The key thing here is that sys.stdin.buffer is the underlying stream, whereas sys.stdin does line ending conversion.

[stephanelsmith]

Thank you for the kind response. Mentally I was thinking stdin was the underlying stream which is why I got snared. Learn something new everyday. Again, thank you for the response!

[stephanelsmith]

Part two answer, the stdin buffer size on esp32 is set here:
ports/esp32/mphalport.c
STATIC uint8_t stdin_ringbuf_array[260];

Increasing the size works as expected. Should this be a #define option?

[Baker68]

@stephanelsmith ; Are you willing to share your solution (Python/MPY scripts)?

Thanks in advance