extract numbers from uart.read

[keble6]

I'm using a microbit to read bytes from a serial stream. I'm having a problem with understanding how to extract the byte values.
I'm using this micropython test code:

from microbit import *

def uart_9600():
    uart.init(9600, tx=pin0, rx=pin1)
    
def uart_usb():
    uart.init(1115200)

uart_9600()
while not uart.any():
    pass
    
sleep(400)
content = uart.read()
uart_usb()
#print(str(content))
print(content)
if (content is not None):
    print(content[1])
    print(content[2])

This is what I see in the serial window:
b'26,43,60,77
54
44

So when I print(content) I see the 4 bytes as numbers - fine.
But when I print content[1] I see 54 - why is that?
And how should I extract each byte as a number?
Thanks!

[karfas]

Your content is a byte array, holding the ASCII codes of each character. The character 6 (from "26") in content[1] is decimal ASCII 54.
This has nothing to do with micropython or that the data comes from an UART.

Something along (in REPL) should help...

a = b'26,43,60,77'
numbers = str(a).split(',') # create string+split
numbers[0]
int(str(numbers[0]))

Edit: I have no idea why with my Python 3.6.8 str(a).split(',') returns bytes for the first array element, so requiring another str() conversion.

[keble6]

Hmm, the doc for uart.read states "Return value: a bytes object or None on timeout". Is that the same as a byte array?

[karfas]

I assume so. For most practical purposes, a bytes object behaves like an array.
A quick google search gave me https://en.wikiversity.org/wiki/Python_Concepts/Bytes_objects_and_Bytearrays.

[keble6]

That looks useful - thanks!

[sosi-deadeye]

A bytes object is immutable and a bytearray is mutable. You can expand and extend a bytearry, but bytes could not be changed. An array.array is a different type, with a C datatype. An array.array is mutable like a bytearray. uart.read() should return bytes and not a bytearray.

The main problem with bytes and bytearray is the mixed behavior.

greeting = b"Hello World"

print(greeting)
# b'Hello World'

# subscription of bytes > int
print(greeting[0])
# 72

# slicing bytes -> bytes
print(greeting[6:11])
# b'World'

# same for bytearray
# same for iteration

The square-bracket access (subscription) returns the value of the single byte as an int.
If the text was encoded with ASCII, then it's the code point in the ASCII Table.

ASCII characters are frequently used for serial protocols and everything else.

The same behaviour of bytes and bytearray is true for iteration.

greeting = b"Hello World"

print(list(greeting))
# [72, 101, 108, 108, 111, 32, 87, 111, 114, 108, 100]
# ASCII representation

print(tuple(greeting))
# (72, 101, 108, 108, 111, 32, 87, 111, 114, 108, 100)

for value in greeting:
    print(value, type(value))

# 72 <class 'int'>
# 101 <class 'int'>
# 108 <class 'int'>
# 108 <class 'int'>
# 111 <class 'int'>
# 32 <class 'int'>
# 87 <class 'int'>
# 111 <class 'int'>
# 114 <class 'int'>
# 108 <class 'int'>
# 100 <class 'int'>

To solve your task, you can do the following steps:

1. read the whole line. I assume that you send at the end of a line a \r\n. This is often used in serial ASCII protocols.
2. split the bytes at b','
3. apply for each element of the split result int(element)

The code could look like this:

from microbit import *

# init code
# code is not portable to Micropython
# it used microbit's version of Micropython


def line_reader():
    """
    Generator Function which reads uart line by line
    split the byte(string) at the ,
    and convert each number to an integer
    """
    while True:
        line = uart.readline()
        
        if line is None:
            # timeout, just go back to the top and
            # wait for next line
            continue
            
        try:
            yield list(map(int, line.split(b",")))
            # or as a list comprehension
            # yield [int(val) for val in line.split(b",")]
        except ValueError:
            print("Could not convert line to integers:", line)
            continue


for values in line_reader():
    print(values)

[keble6]

Thank you! I am getting closer to understanding the issues now. However the data being sent to me is not ASCII, so I can't use readline().

In fact the data comes from an air quality sensor (PM1006) and the format is (in hex) 16 11 0B DF1-DF4 DF5-DF8 DF9-DF12 DF13 DF14 DF15 DF16[CS]

So I know there are 20 bytes and the first 3 bytes are 16 11 0B.
The useful numbers (for display) are in bytes DF3-4.
I plan to check that I am receiving a correct message using the first 3 bytes, then use DF3-4. for display

If you can help me understand how to return the unsigned integer value of each byte (by slicing or subscription) then I think I can generate code!

[karfas]

Well, there is a reason the bytes object is called bytes. Each array element is a byte (an integer between 0 and 255) regardless of any encoding. ASCII, utf8, EBCDIC, whatever is a stream of bytes.

Maybe you should learn very basic things first (e.g. the difference between a byte and a character, subscription and slice of python arrays).

This thread (even when started using completely wrong assumptions, with no connection to your real problem) has enough code for you to study...

[keble6]

Well karfas, I'm sorry that I do not meet your high standards.
Be assured that I do understand what a byte is, what a character is, and so on.

[robert-hh]

To extract values from a byte array, you can use struct module and the method struct.unpack() or struct.unpack_from(). You can look at the generic Python manual for instructions, how to use it.
Since you are only interested in DF3 and DF4, which have index 5 and 6 in the bytes/bytearray object, you can access them directly and as two integers as content[5] and content[6]. If these two form a 16 bit number, you can calculate that as e.g. content[5] * 256 + content[6]. That is more efficient than using struct. One would use the latter to unpack all values from the message, which then is more efficient and shorter.

[keble6]

To extract values from a byte array, you can use struct module and the method struct.unpack() or struct.unpack_from(). You can look at the generic Python manual for instructions, how to use it. Since you are only interested in DF3 and DF4, which have index 5 and 6 in the bytes/bytearray object, you can access them directly and as two integers as content[5] and content[6]. If these two form a 16 bit number, you can calculate that as e.g. content[5] * 256 + content[6]. That is more efficient than using struct. One would use the latter to unpack all values from the message, which then is more efficient and shorter.

Thank you, that is very helpful!