Unicode error on serial init

[davefes]

Sometimes when I re-boot a datalogger using an ESP32-SPIRAM variant
I get a unicode error.

The SBMS0 is a battery management system which is sending out data
once a second (for about 30ms). It seems that when I re-boot, at some critical time
during this 1 second period, I get a unicode error.

I read one "fix" could be to deinit the UART and init it again, but this
doesn't help. Then I tried some delay between the two attempts without
success. Maybe, a 1 second delay is the worst choice?

Maybe a try/except block, but what would I do in the except block to give
me success on the next attempt? Re-boot?

I have a watchdog timer that will kick-in after 90 seconds, which gives me
further chances for the system to start running properly.

Any suggestions on a better way to do this are appreciated.

# open serial connection, SBMS0 must be set to same BAUD rate
# and a 1sec logging interval on the SBMS0

#ser = UART(1) #  ok for a ESP32
#ser.init(baudrate=19200, tx=19, rx=18, bits=8, parity=None, stop=1, timeout=300)

# for an ESP32_SPIRAM or you get a Guru Meditation Error on boot-up
# see https://github.com/micropython/micropython/issues/6166
ser = UART(1, baudrate=19200, bits=8, parity=None, stop=1, rx=18, tx=19, timeout=300)
ser.deinit()  # to hopefully stop a unicode error on boot-up
time.sleep(1)  # nope ... try a sleep
ser = UART(1, baudrate=19200, bits=8, parity=None, stop=1, rx=18, tx=19, timeout=300)

This still seems problematic:

try:
    ser = UART(1, baudrate=19200, bits=8, parity=None, stop=1, rx=18, tx=19, timeout=300)
except:
   ser.deinit()  # to hopefully stop a unicode error on boot-up
   time.sleep_ms(500)
   ser = UART(1, baudrate=19200, bits=8, parity=None, stop=1, rx=18, tx=19, timeout=300)

[ned-pcs]

The unicode error is likely due to getting garbage characters at start-up. You don't show what you're doing with the bytes you read from the UART, but in my experience it helps to process the bytes as they come in and reject inappropriate ones. Don't assume that ser.readline() will give you reliable data. For instance, I often do something like this:

def clean_bytes(byte_string):
    retval = []
    for b in byte_string:
        if 32 <= b < 127:
            retval.append(chr(b))
        elif b == 13:
            retval.append('\r')
        elif b == 10:
            retval.append('\n')
    return ''.join(retval)

[Josverl]

not clear if you see the error in mpremote, but as a receiver it also has the same problem, though mostly with esp8266 due to it's changes in UART baudrate during the bootcycle.
For mpremote there is a PR to handle that

after receiving , in Cpython you can ignore/replace these errors during conversion :

>>> str( b'\xff\xfe\xfa', "utf8")
Traceback (most recent call last):
  File "<python-input-2>", line 1, in <module>
    str( b'\xff\xfe\xfa', "utf8")
    ~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
>>> str( b'\xff\xfe\xfa', "utf8", "replace")
'���'
>>> str( b'\xff\xfe\xfa', "utf8", "ignore")
''

However in MicroPython neither is implemented.
There is one option , and that is to build with MICROPY_PY_BUILTINS_STR_UNICODE_CHECK = 0 , then the check is not done at all

[davefes]

Thanks both for your responses. I am using rshell to monitor the boot-up process.

Fundamental question ... if the serial port is enabled during a data packet from the TX-end will that packet be ignored? The code at the bottom of the original post seems to allow the system to recover before a watchdog timeout.

The rest of the processing code:

    try:
      # walrus operator, needs the parentheses
        while True:
            if (data := ser.read()):
                var_sbms = data.decode("utf-8")

            if len(var_sbms) == 61:
                break

    except OSError as error:
        try:
            with open('errors.txt', 'a') as outfile:
                outfile.write('failed to get the SBMS0 data' + '\n')
                outfile.write(str(error) + '\n')
        except OSError:
            pass

      # deinit UART before a up-coming machine.reset()
        ser.deinit()

        print('do a machine.reset()')
        time.sleep(1)
        machine.reset()

and I have never seen a 'failed to get the SBMS0 data' message.

[ned-pcs]

Your unicode error is coming from the data.decode("utf-8") part. Unless you really have UTF-8 data coming in, you probably want to accumulate and filter the incoming bytes. Also, this assumes that you're going to get an entire 61 characters received at once, which may not be the case depending on when you start receiving. Do you have a delimiter in your protocol? What kind of characters are you expecting (i.e. all ASCII? what range?)?

You could also not try to decode unless your data is 61 bytes long. And unless it's UTF-8 you may be better off using data.decode('ascii').

[Josverl]

or change to var_sbms = data.decode("utf-8","ignore") ( Assuming CPython)
Which will handle bytes , utf-8 , and any malformed.
A bit more work for the computer , much less worry for the operator :-)

https://docs.python.org/3/library/codecs.html#error-handlers

[davefes]

@ned-pcs This is an example format from the manual:
var sbms="YMDHMS%%C1C2C3C4C5C6C7C8ITET+BATPV1PV2EXTAD2AD3AD4ht1ht2ERR"

This is what I use for testing:
var_sbms = '8(&8>)#{GPGRGOGRGPGPGOGL*\*\-#D[#######DR###############%N('
this is a compressed data format

delimiter? Don't think so.
range, I think it is all ASCII

Test length before decode sounds like a better approach. I will do some testing, which is not easy as the error only happens rather infrequently.

@Josverl I am only doing this in upython

Thanks for the suggestions.

[davefes]

How do I test the length before decoding?

[ned-pcs]

while True:
    if (data := ser.read()) and len(data) == 61:
	    try:
		var_sbms = data.decode()
		break
	    except UnicodeError:
		try:
		    with open('errors.txt', 'a') as outfile:
			outfile.write('failed to get the SBMS0 data' + '\n')
			outfile.write(str(error) + '\n')
			outfile.write(f"data={repr(var_sbms)}\n")
		except OSError:
		    pass

# do something with var_sbms

[davefes]

Right ... I made a mistake thinking I had to do length var_sbms !

Give me a few days and I will report back hopefully with a Mark as answer.

[davefes]

@ned-pcs
A random error that has bothered me for more than 5 years.

Thanks