Abstract UART class for creating virtual UARTs?

[ThinkTransit]

I want to create some virtual UARTs in micropython. Is there an abstract class I can extend or do I need to build something from the ground up?

The background to this is I want to try and implement UART multiplexing, where multiple UARTS can be used over a single physical UART. The data is sent in frames and addressed to the relevant virtual UART as outlined in a 3GPP standard. This is for communicating with modems where by one virtual UART can be used for AT commands and another for PPP traffic etc..

Ideally the virtual UARTs would be seamless so they would work with existing micropython modules like ppp and stream reader etc.

Any thoughts, crazy idea?

[harbaum]

This would also allow to use i2c UARTs and the like transparently.

[nestorld]

I have exactly the same requirement today. I have created a library to handle the cmux protocol with MicroPython. It is working fine multiplexing the channels but trying to figure out how to emulate the UART for ppp. Did you find any solution for it?

[ThinkTransit]

No unfortunately I haven't done any work on this but I still think it would be a great feature to get working.

[nestorld]

I tried to emulate the UART by creating a subclass of BytesIO (https://docs.micropython.org/en/latest/library/io.html#module-io), and inside it, I defined two buffers (one for reading and one for writing) of BytesIO class too. I also did overwrite the any, read and write methods to use those buffers, and created the input and output methods to emulate data coming in through the rx pin and going out through the tx pin respectively. Basically, those two methods are writing to the buffer_in and reading from the buffer_out. All this works fine for virtual cmux channels in command mode, but when I pass one of these virtual UARTs to network.ppp (after setting it to data mode by using ATD*99# on it), I get the following error:

 Connection lost -- read failed: [Errno 6] Device not configured
 Use Stop/Restart to reconnect.
 Process ended with exit code 1.

It seems like for some reason it is stepping on the UART 0 used for the REPL USB port.
This is my virtual_uart.py code, I have tried many things but the network.ppp is still not able to successfully use it.

from io import BytesIO
from time import sleep, ticks_diff, ticks_ms


CHANNEL_CLOSED = 0


class VirtualUART(BytesIO):
    def __init__(self, timeout_ms=3000):
        self.buffer_in = BytesIO(2048)
        self.buffer_out = BytesIO(2048)
        self.timeout_ms = timeout_ms
        self.status = CHANNEL_CLOSED
        self.v24Signals = None
        
        super().__init__()


    def input(self, buf):
        # This simulates data coming in via the rx pin
        self.buffer_in.write(buf)


    def output(self):
        # This simulates data going out via the tx pin
        size = self.buffer_out.tell()
        self.buffer_out.seek(0)
        data = self.buffer_out.read(size)
        self.buffer_out.seek(0)
        
        return data


    def clear_buffer_in(self):
        self.buffer_in.seek(0)


    def clear_buffer_out(self):
        self.buffer_out.seek(0)


    def any(self):
        print("Using any")
        startTime = ticks_ms()
        while ticks_diff(ticks_ms(), startTime) < self.timeout_ms:
            if self.buffer_in.tell() > 0:
                return self.buffer_in.tell()
            else:
                sleep(0.1)


    def read(self, nbytes=-1):
        print("Using read")
        if nbytes > -1:
            currentSize = self.buffer_in.tell()
            if nbytes > currentSize:
                nbytes = currentSize
            self.buffer_in.seek(0)
            data = self.buffer_in.read(nbytes)
            remaining = self.buffer_in.read(currentSize - self.buffer_in.tell())
            self.buffer_in.seek(0)
            self.buffer_in.write(remaining)
        else:
            size = self.buffer_in.tell()
            self.buffer_in.seek(0)
            data = self.buffer_in.read(size)
            self.buffer_in.seek(0)
        
        return data


    def readinto(self, buf, nbytes=-1):
        print("Using readinto")
        buf = self.read(nbytes)


    def write(self, buf):
        print("Using write")
        return self.buffer_out.write(buf)

I have already confirmed that my parsed cmux data works fine with nerwork.ppp when I pass it to a real UART.

Sorry for directly pinging you @peterhinch and @jimmo, maybe you guys could have useful thoughts about this?

[vshymanskyy]

I can only confirm that I also had failed to attach a virtual UART to PPP in the past (a few month ago). Probably there's some discrepancy...
Then I decided just to "decorate" a native uart with a virtual wrapper, and I was only either getting crashes or it just didn't work.

[andrewleech]

I too have wanted to multiplex multiple streams over USB / UART and was thinking of using cmux to help. I'd be quite interested in seeing your library for this @ThinkTransit !

As a building block for this I created a python interface over the C ringbuffer to act as a controlled size fast/efficient version of bytesio with all the correct interfaces to work as a "real" python stream for things like asyncio.Streamwriter.
#9458

I'm not sure about the PPP module but I know asyncio.Streamwriter relies on select behind the scenes which itself uses the C ioctl function when it polls.

I know io.BytesIO also provides this functionality, however I don't think it's possible to subclass it, see here for related issue: #10402

But when I was trying to use my ringbuffer I would use two instances of it, one of them wrapped in a asyncio.Streamwriter and the other in asyncio.Streamreader and then pass them both to microwebsrv or similar as separate read/write streams like a "socket pair".

I didn't think of usages like this where there's expected to be a single object like a UART which essentially has two streams behind the scenes.

I'm really not sure if it's currently possible to emulate this sort of "stream pair" object in micropython, maybe I need to extend my ringbuffer class to support dual streams...

[ThinkTransit]

Unfortunately I never got around to creating the library! I went through the 3GPP multiplex standard and decided that it was overkill for the particular project I was working on. The standard requires the individual packets to be packaged into addressed frames and then those frames are transmitted over the physical uart.

[andrewleech]

Oh whoops I mixed up the messages, the cmux implementation was mentioned by @nestorld

[nestorld]

@andrewleech I agree the custom virtual UART class, has to implement all the methods used by the real UART to work with the network.PPP service. I tried to do that with my VirtualUart class, but apparently, something is missing. I'm not sure how to check if the ioctl method that you are saying is needed here, and what it should do.
Here I did publish my library (https://github.com/MicropythonShare/cmux). Please take a look and maybe you can help to add what is missing to the VirtualUart class for the PPP connection.

[nestorld]

The ringbuffer concept would be way more efficient than the way I'm writing and reading the streams in the VirtualUart class. Something that can be improved later for sure. My main concern now is to have the PPP connection working with the cmux data flow.

[ThinkTransit]

@nestorld This looks really good, I will test it on my Sara R5 ublox modems which also support UART multiplexing.

[andrewleech]

Oh, actually yes you can implement it completely in python! Here's an example: https://github.com/micropython/micropython/blob/master/examples/bluetooth/ble_uart_repl.py

Essentially you need to subclass io.IOBase and ensure you implement all the stream functions plus (crucually) ioctl - if that's missed you'll segfault if used with select.

[andrewleech]

The network.PPP doesn't actually need a UART instance, what the code looks for is a stream capable object. This is anything that implements the steam protocol, like file, BytesIO, network sockets, etc.

I weirdly needed almost the exact same thing at work yesterday, a mock UART for testing application code with. Similarly though what I actually needed wasn't a UART but a stream that could be passed into asyncio.StreamReader.

Looking into all this I see there's really no docs that go into detail about how this is made in practice, nor how to implement it in python... I'd like to write some, maybe later.

I've updated my ringbuffer mr a bit and made this library to work with it: micropython/micropython-lib#907

That uses a bi-directional pair of ringbuffers to make an object that will work exactly like a virtual UART. But yeah, needs my ringbuffer pr.

It's harder to emulate with BytesIO because you need to manage the read/write pointers manually with that like you have @nestorld

The virtual UART class you've got might work correctly though if you change it to subclass io.BaseIO and copy the ioctl function from micropython/micropython-lib#907 - that's the key function needed by stream protocol that's missing from the docs!

[andrewleech]

I'm pretty sure for stream use any() shouldn't have a timeout, it should return immediately. eg.

    def any(self):
        return self.buffer_in.tell()

I also think at the moment readinto is replacing buf local variable, not filling the existing buffer. Maybe should be:

def readinto(self, buf, nbytes=-1):
    currentSize = self.buffer_in.tell()
    if nbytes > currentSize:
        nbytes = currentSize
    self.buffer_in.seek(0)
    read = self.buffer_in.readinto(buf, nbytes)
    remain = self.buffer_in.read(currentSize - read)
    self.buffer_in.seek(0)
    self.buffer_in.write(remain)
    return read

edit: So that took me a couple of goes to get right (I think) - it really is tricky to emulate a ringbuffer with BytesIO. Might be better with deque really.... or a bytearray and moving memoryview. Not sure, the real ringbuffer is certainly a lot easier!

[andrewleech]

Might be simpler for now to just use a basic bytes rather than BytesIO

    def __init__(self):
        self.buffer_in = b''

    def input(self, buf: bytes):
        # This simulates data coming in via the rx pin
        self.buffer_in += buf

    def read(self, nbytes=-1):
        if nbytes > -1:
            currentSize = len(self.buffer_in)
            if nbytes > currentSize:
                nbytes = currentSize
            data = self.buffer_in[0:nbytes]
            self.buffer_in = self.buffer_in[nbytes:]
        else:
            data = self.buffer_in
            self.buffer_in = b''
        return data

[nestorld]

@andrewleech You are right, readinto was just replacing buf local variable, my bad. But in any case, BytesIO.readinto seems not to be working, I tested it with a simple BytesIO object and it is never filling the buf I'm passing to it (passing a bytearray).
I decided to go with your ringbuffer and StreamPair, which look great. To use that, I got the files from your PR to create a custom firmware and modified my cmux code to use it. But the bad news is that even with that I'm getting exactly the same error when trying to use PPP. There is something that PPP is expecting and still missing in both my initial virtual uart design and this other virtual uart using your code. Something that I don't understand yet.
About the .any() method, looks like you are right too, it should return immediately, but because a timeout is useful for my code I added it as optional, being zero the default timeout if not specified.
I updated my shared cmux repo with the new code using ringbuffer and StreamPair.
BTW, the .flush() method seems to be doing nothing too.
I think this will require reviewing more in detail what the PPP code is expecting to be able to complete the cmux solution.

[nestorld]

@andrewleech unfortunately this looks like a dead road to me, so close but apparently the PPP connection is always using the UART driver and it can not work with a normal stream buffer. I found something that should be the path to resolve this: the esp-protocols repo already has an implementation to use cmux (https://github.com/espressif/esp-protocols/blob/master/components/esp_modem/examples/simple_cmux_client/main/simple_cmux_client_main.cpp). I was trying to create a module but the documentation to do that doesn't cover how to include these libraries to compile a module like that. I was reviewing https://micropython-usermod.readthedocs.io/en/latest/usermods_05.html and https://docs.micropython.org/en/latest/develop/natmod.html#natmod. I can create something like the basic example presented there, but it gets more complicated when I try to include the esp-protocols libraries. I would like to try first with a native .mpy file because it is easier to troubleshoot, and later go with a module to integrate it into a custom firmware.

[andrewleech]

The stream stuff really is tricky to get right, it's taking me a good few goes to get it right for basic compatibility with asyncio too.

From looking at the code, I'm pretty sure the esp32 ppp driver is only using stream APIs, there doesn't appear to be any direct UART interaction. It would possibly take a debugger or extra printf's in C to inspect the data streams to see what's going wrong.

The alternate esp component that directly uses cmux is interesting, I don't really know how Espressif's component managers work to enable these in a build though.

Regarding C modules, dynamic native modules can't access vendor C libraries from the micropython build, you would need to find a way to build and link directly with your own copy of esp idf - this might not even be possible.

It's usually much easier to get user C modules working that are directly built with micropython, they have access to everything in C at the same time micropython does.

[lbuque]

I want to create some virtual UARTs in micropython. Is there an abstract class I can extend or do I need to build something from the ground up?

The background to this is I want to try and implement UART multiplexing, where multiple UARTS can be used over a single physical UART. The data is sent in frames and addressed to the relevant virtual UART as outlined in a 3GPP standard. This is for communicating with modems where by one virtual UART can be used for AT commands and another for PPP traffic etc..

Ideally the virtual UARTs would be seamless so they would work with existing micropython modules like ppp and stream reader etc.

Any thoughts, crazy idea?

I've always wanted to do this, but unfortunately I haven't had the time.

[lbuque]

network.PPP finally calls the following function to get the ppp data. Using python to override the write/read method is invalid.

micropython/extmod/machine_uart.c

STATIC const mp_stream_p_t uart_stream_p = {
    .read = mp_machine_uart_read,
    .write = mp_machine_uart_write,
    .ioctl = mp_machine_uart_ioctl,
    .is_text = false,
};

[lbuque]

You need to write a VirtualUART class in C language to make networkPPP work properly.

[andrewleech]

The network_ppp driver in esp32 does not directly call the UART C functions, it calls the higher level stream API:
mp_stream_rw(self->stream, data, len, &err, MP_STREAM_RW_WRITE);

micropython/ports/esp32/network_ppp.c

Line 104 in 6c3dc0c

return mp_stream_rw(self->stream, data, len, &err, MP_STREAM_RW_WRITE);

Though now I look at that function more closely I see it's not much higher level... more just a C helper. I did think the standard stream API would handle python or C but this particular one looks like it's just for fast C handling.

A quick search through the entire codebase shows this C handler is used in only 2 places; esp32/network_ppp and vfs_reader (used by mpremote mount).

This would be an argument for extending my ringbuffer python API to directly support two stream / bidirectional use, then it would work here.

[nestorld]

I confirmed that PPP uses the "mp_machine_uart_read" and "mp_machine_uart_write" functions from micropython/ports/esp32/machine_uart.c
I was thinking that maybe the virtual UART C class could be created in that file. Something like:

typedef struct _machine_vuart_obj_t {
    mp_obj_base_t base;
    uart_port_t uart_num;
    ringbuf_t tx;
    ringbuf_t rx;
    uint16_t txbuf;
    uint16_t rxbuf;
} machine_vuart_obj_t;

Which would be a reduced version of the UART class in line 56 of the same file.
Later in "mp_machine_uart_make_new" function it can decide what UART to create. If the id of the UART is between >=0 and <UART_NUM_MAX, it would decide to create a normal physical UART, and if it is == UART_NUM_MAX, it would create a virtual one.
Also, the "mp_machine_uart_read" and "mp_machine_uart_write" can evaluate the id of the UART and based on that they would execute the read/write functions for a physical UART or for a virtual one (read/write of ringbuf). A similar logic could be used for the rest of the functions.
Does this make sense for you guys?
Unfortunately, I'm not sure how that new class can be added to the C internals of MicroPython.

[andrewleech]

I've heard that a python class really should work here instead of the uart as long as the python class is subclassed from io.IOBase - that base class provides the C functions of the stream protocol which should be called from things like network_ppp

The python streampair class I've proposed already does that though so it leaves little to assist in debugging why it's not already working for you.

Alternatively I trialled a C version of streampair already on top of my ringbuffer class (which has been renamed to RingIO) which you could test instead to prove it it's a C vs python issue.
master...andrewleech:micropython:ringio_link

In this form instead of the streampair function returning two objects, you manually create the two RingIO objects and then .link() one to the other, the test script in the commit has an example. It might not be very clear though.

[nestorld]

Well, still trying to figure out this... I was able to debug a little more (not easy with the REPL console crashing on each attempt). Both physical and virtual UARTs are going through the micropython/py/stream.c file and executing this line (line 60):

mp_uint_t out_sz = io_func(stream, buf, size, errcode);

With the physical UART it goes to micropython/ports/esp32/machine_uart.c and executes "mp_machine_uart_write" or "mp_machine_uart_read" depending on if it's writing or reading, which is defined in lines 52 and 54 of micropython/py/stream.c

With the virtual UART, when the Micropython code executes ppp.connect(), it crashes at that line 60 of micropython/py/stream.c when trying to do the first write.

[nestorld]

Finally, I got it working! The key part is the protocol class used in line 194 of micropython/extmod/machine_uart.c, in the same section pointed by @lbuque:

static const mp_stream_p_t uart_stream_p = {
    .read = mp_machine_uart_read,
    .write = mp_machine_uart_write,
    .ioctl = mp_machine_uart_ioctl,
    .is_text = false,
};

MP_DEFINE_CONST_OBJ_TYPE(
    machine_uart_type,
    MP_QSTR_UART,
    MP_TYPE_FLAG_ITER_IS_STREAM,
    make_new, mp_machine_uart_make_new,
    print, mp_machine_uart_print,
    protocol, &uart_stream_p,
    locals_dict, &machine_uart_locals_dict
    );

and defined in micropython/py/stream.h:

// Stream protocol
typedef struct _mp_stream_p_t {
    // On error, functions should return MP_STREAM_ERROR and fill in *errcode (values
    // are implementation-dependent, but will be exposed to user, e.g. via exception).
    mp_uint_t (*read)(mp_obj_t obj, void *buf, mp_uint_t size, int *errcode);
    mp_uint_t (*write)(mp_obj_t obj, const void *buf, mp_uint_t size, int *errcode);
    mp_uint_t (*ioctl)(mp_obj_t obj, mp_uint_t request, uintptr_t arg, int *errcode);
    mp_uint_t is_text : 1; // default is bytes, set this for text stream
} mp_stream_p_t;

and:

// Object is assumed to have a non-NULL stream protocol with valid r/w/ioctl methods
static inline const mp_stream_p_t *mp_get_stream(mp_const_obj_t self) {
    return (const mp_stream_p_t *)MP_OBJ_TYPE_GET_SLOT(((const mp_obj_base_t *)MP_OBJ_TO_PTR(self))->type, protocol);
}

That is only present when the class is directly defined in C, and not adding the read/write/ioctl methods to a class in Python.
I was adding a virtualUART class in micropython/ports/esp32/machine_uart.c and micropython/extmod/machine_uart.c, but noticed that the RingIO class added by you @andrewleech to micropython module was already containing what was needed, so I decided to try getting it from your PR, and it worked fine.
I updated my repo with the working code.
I hope your PR is accepted soon for the next MicroPythoin release @andrewleech, Thanks!