Saving memory while converting to utf-16be

[scruss]

I have a device that expects data in UTF-16BE format. MicroPython doesn't have a .encode('utf-16be') method, so I had to roll my own function:

def fauxutf16(s, badchar=" "):
    # encodes string s as UTF-16BE, two bytes per char, no BOM
    # eg: "hello" => b'\x00h\x00e\x00l\x00l\x00o'
    # char codes > 65535 encoded as badchar
    # MicroPython can only encode to UTF-8 bytes, so we need this
    buf = bytearray(len(s) * 2)  # preallocate empty buffer
    for i, c in enumerate(s):
        n = ord(c)
        if n > 65535:
            n = ord(badchar)
        buf[2 * i] = n // 256
        buf[2 * i + 1] = n % 256
    return bytes(buf)

This routine works well for the range of characters the device expects to see in its input stream, including those in the CJK Unified Ideographs block, U+4E00 - U+9FFF.

I'm concerned, however, that allocating a bytearray of a known size and then converting it to bytes is somehow wasteful of memory. Some of the strings a user might send to the device could be up to 4096 characters long, and I'd want to avoid allocating that amount twice.

[rkompass]

Hello Scruss,

so what prevents you from returning the bytearray?
I thought what you could use were uctypes.bytes_at() . But now checked the docs and they say that the contents is copied :-(
uctypes.bytearray_at() does not copy the bytes, but gives a reference instead, but that does not help you, as you have a bytearray again.
You need a bytearray recast as bytes object?

As a workaround you may recode your function in viper, or use a viper subfunction that operates on the buffer of the bytes object that you allocate (instead of the bytearray) in your function and have to return.
As an illustration that this is possible:

@micropython.viper
def count32(p:ptr32):
    p[0] = p[0] + 1

@micropython.viper
def count16a(p:ptr16):
    p[2] = p[2] + 1

gv = bytes(8)

print('Before: ', int.from_bytes(gv, 'little'))

count32(gv)
count32(gv)
count32(gv)
count16a(gv)
count16a(gv)

print('After: ', int.from_bytes(gv[0:4], 'little'))
print('After: ', int.from_bytes(gv[4:], 'little'))

yields:

Before:  0
After:  3
After:  2

[scruss]

wow, thanks!

I thought I'd need bytes because I'd had problems returning bytearrays before.

This is where I use the fauxutf16 code:

        data_bytes = self.fauxutf16(data)  # returns bytes object
        tx_len = len(data_bytes) + 2
        ## assumes UTF-16BE encoding
        buf = bytes([0xFD, tx_len // 256, tx_len % 256, 0x01, 0x04])
        self.uart.write(buf + data_bytes)

(referenced here: https://github.com/scruss/micropython-SYN6988/blob/03ce4268cc47c91d8a57dd63d1a3eb10e3cf5a99/syn6988.py#L47C11-L47C11)

More work needed from me. I guess I landed on the first, simplest solution I could understand. Much appreciated.

[rkompass]

From this code it looks like you could generate in and return with fauxutf16() a bytearray that is 5 bytes longer (reserve the space for the 5 header bytes). In fauxutf16() you would fill the bytearray starting at indices 5, 6, ... .
And in the above calling code you would then write into indices 0, 1, .. , 4 the 0xFD, tx_len // 256, tx_len % 256, 0x01, 0x04 header bytes.
This would prevent a new allocation when executing (buf + data_bytes).

[scruss]

return bytes(array.array('H', (ox if ((ox := ord(x)) < 65536) else badchar for x in s)))

Wouldn't that be ord(badchar) instead of badchar?

This code seems to crash CircuitPython 8.1 on a Raspberry Pi Pico, btw

[rkompass]

Certainly @jimmo has missed that detail and it should be ord(badchar). This would correspond to your expression

if n > 65535:
            n = ord(badchar)

that worked.

[jimmo]

Wouldn't that be ord(badchar) instead of badchar?

@scruss yep exactly that. That'll learn me for not testing it :)

buf = bytes([0xFD, tx_len // 256, tx_len % 256, 0x01, 0x04])

This will allocate a list (of size 4*len because integers), then allocate the bytes (of size len). Better to use struct.pack.

self.uart.write(buf + data_bytes)

Instead you could just do two separate calls to write.

data_bytes = self.fauxutf16(data)  # returns array.array object
self.uart.write(struct.pack('BHBB', 0xfd, len(data_bytes) * 2 + 2, 0x01, 0x04))
self.uart.write(data_bytes)

(note the len*2 because the len of the array will be in shorts not bytes).

Also struct.pack_into means you can avoid the allocation in struct.pack with a pre-allocated header buf:

# global
header_buf = bytearray(5)

struct.pack_into('BHBB', header_buf, 0, 0xfd, len(data_bytes) * 2 + 2, 0x01, 0x04)
self.uart.write(header_buf)
self.uart.write(data_bytes)