The little endian blues

[kjm1102]

I've got a byte string that is supposed to be the number of seconds since Jan1 2018 in little endian format, so I'm expecting roughly 5 years and 10 months worth of seconds, about 180 million
Annoyingly my efforts so far are either too big or too small

>>> int.from_bytes(b'0033EB0A', 'little')
4697327276326596656
>>> import ustruct; ustruct.unpack("<h", b'0033EB0A')
(12336,)

Can some knowledgeable soul please show me how to get the goldilocks number?

[robert-hh]

That should work:
int.from_bytes(b'\x00\x33\xEB\x0A', 'little')
or that:
import ustruct; ustruct.unpack("<I", b'\x00\x33\xEB\x0A')

To get b'\x00\x33\xEB\x0A' from b'0033EB0A', you can use binascii.unhexlify(), like
int.from_bytes(binascii.unhexlify(b'0033EB0A'), "little")

[kjm1102]

How did you know to unhexlify the bytes before doing the little-to-big conversion? Is that just experience?

[glenn20]

As a convenience, on newer micropython versions: int.from_bytes(bytes.fromhex(b'0033EB0A'), "little") avoids the need to import binascii.

[kjm1102]

tnx Glenn. I've abandoned trying to comprehend the labyrinthine types, will add this incantation to my cheat sheet of spells to try when moving between data representations. Who needs encryption when we've got type obfuscation!

[jimmo]

@kjm1102

the labyrinthine types

There really is only one type here -- bytes. The confusing thing is that Python allows you to enter bytes literals in their ascii representation, and does its best to display in this form too. A lot of confusion would be avoided if Python always printed bytes type with every character escaped. On the other hand, I much prefer it the way it is because it's really useful to immediately see the ascii data embedded in a bytes string.

These are the exact same thing:

b'123456'

b'\x31\x32\x33\x34\x35\x36'

The former says: "make a byte string containing the byte values for ascii 1 (49 == 0x31), 1 (50 == 0x32)".
The latter says: "make a byte string containing the byte values for these escaped hex values 0x31, etc."

The important thing is that inside a literal string (i.e. between quotes) the \x says "interpret the next two characters as a hexadecimal value".

You can see that they are the same by:

>>> list(b'\x31\x32\x33\x34\x35\x36')
[49, 50, 51, 52, 53, 54]
>>> list(b'123456')
[49, 50, 51, 52, 53, 54]

And you can see where these numbers come from because:

>>> ord('1'), ord('2'), ord('3')
(49, 50, 51)
>>> hex(ord('1')), hex(ord('2')), hex(ord('3'))
('0x31', '0x32', '0x33')

So b'aa' is two bytes with the value of ascii 'a' (i.e. 97). But b'\xaa' is a single byte with value 0xaa (i.e. 170).

BUT. '313233343536' (or b'313233343536') is not the same thing as the examples amove. This is the hexadecimal encoding of the bytes above into a string (and is now 12 bytes long). To turn this back into the byte values, you need to effectively compute the value of each pair of digits and assemble them into a list. This is what bytes.fromhex(b) does.

>>> bytes.fromhex('313233343536')
b'123456'

And in reverse, b.hex() allows you to generate this hexadecimal encoding.

>>> b'123456'.hex()
'313233343536'

[ricksorensen]

This seems like a great entry for the Wiki/ Tips and Tricks section? Or FAQ?

[robert-hh]

Just the old story about data types and representation of hex numbers by ASCII letters, where each letter represents 4 bit binary data.

[Josverl]

Little endian blues

I love a good title 😍

[kjm1102]

Probably more accurately 'the type blues' Jos. b'\x00\x33\xEB\x0A' and b'0033EB0A' are both type bytes and, to paraphrase Orwell, all bytes are created equal, but some bytes are more equal than others.

[kjm1102]

I think I need to read this a couple of more times, but it was nice to eyeball the reverse of bytes.fromhex there at the end. Tnx for taking the time to craft the reply.

…

On Wed, 25 Oct 2023, 4:22 pm Jim Mussared, ***@***.***> wrote: @kjm1102 <https://github.com/kjm1102> the labyrinthine types There really is only one type here -- bytes. The confusing thing is that Python allows you to enter bytes literals in their ascii representation, and does its best to display in this form too. A lot of confusion would be avoided if Python always printed bytes type with every character escaped. On the other hand, I much prefer it the way it is because it's really useful to immediately see the ascii data embedded in a bytes string. These are the exact same thing: b'123456' b'\x31\x32\x33\x34\x35\x36' The former says: "make a byte string containing the byte values for ascii 1 (49 == 0x31), 1 (50 == 0x32)". The latter says: "make a byte string containing the byte values for these escaped hex values 0x31, etc." The important thing is that inside a literal string (i.e. between quotes) the \x says "interpret the next two characters as a hexadecimal value". You can see that they are the same by: >>> list(b'\x31\x32\x33\x34\x35\x36') [49, 50, 51, 52, 53, 54]>>> list(b'123456') [49, 50, 51, 52, 53, 54] And you can see where these numbers come from because: >>> ord('1'), ord('2'), ord('3') (49, 50, 51)>>> hex(ord('1')), hex(ord('2')), hex(ord('3')) ('0x31', '0x32', '0x33') So b'aa' is two bytes with the value of ascii 'a' (i.e. 97). But b'\xaa' is a single byte with value 0xaa (i.e. 170). BUT. '313233343536' (or b'313233343536') is *not* the same thing as the examples amove. This is the hexadecimal encoding of the bytes above into a string (and is now 12 bytes long). To turn this back into the byte values, you need to effectively compute the value of each pair of digits and assemble them into a list. This is what bytes.fromhex(b) does. >>> bytes.fromhex('313233343536')b'123456' And in reverse, b.hex() allows you to generate this hexadecimal encoding. >>> b'123456'.hex()'313233343536' — Reply to this email directly, view it on GitHub <#12766 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AV562M34NUSKBSXOIFHEWHTYBCOZNAVCNFSM6AAAAAA6KU2N5CVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TGNZXGA3TK> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[kjm1102]

Jim, I take your point that b'123456' & b'\x31\x32\x33\x34\x35\x36' are the same byts & indeed I get the same answer if I feed them into int.from_bytes(byts, 'little'). Which brings me back to my question to Rob. When you get something like b'0033EB0A' how does an experienced programmer know they have to unhexlify it before feeding it into int.from_bytes? I mean it's obvious here because I get the wrong answer if I don't but what if you didn't know what the answer should be? How would you know that unhexlification was required before converting to an integer?

[jimmo]

How would you know that unhexlification was required before converting to an integer?

In this case you were expecting an integer (i.e. four bytes) and you got 8 bytes that are all ascii 0-9 A-F (i.e. hex digits).

[kjm1102]

I guess that brings me to why data that goes through a UART has to hexlified in the first place?

…

On Thu, 26 Oct 2023, 11:22 am Jim Mussared, ***@***.***> wrote: How would you know that unhexlification was required before converting to an integer? In this case you were expecting an integer (i.e. four bytes) and you got 8 bytes that are all ascii 0-9 A-F (i.e. hex digits). — Reply to this email directly, view it on GitHub <#12766 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AV562M4C2TUYSPNJBOVCSPLYBGUKZAVCNFSM6AAAAAA6KU2N5CVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TGOBWG4ZTS> . You are receiving this because you were mentioned.Message ID: ***@***.*** com>

[jimmo]

There are a lot of older systems that need to be 7-bit ascii safe.