AXML/ARSC UTF-8 string decoding improvements

[obfusk]

I noticed androguard doesn't handle all the oddities of AXML/ARSC UTF-8 string decoding (specifically, the presence of surrogate pairs and strings with a length that doesn't fit into two bytes); you might want to take a look at how my parser handles those.

[erev0s]

Hi @obfusk thanks for pointing this out. Indeed I had noticed that the case when the length does not fit into two bytes for both utf8 and utf16 is not covered as this is one of the 'tricks' used by some malware to attempt to break static analysis tools.
Though I did not even know about the surrogate pairs and that they should be supported.
Will try to take a look in the upcoming days/weeks

[obfusk]

Though I did not even know about the surrogate pairs and that they should be supported.

I think it's because Java uses UTF-16 internally and somehow they end up used in UTF-8-encoded strings as well even though they should not be used there and will result in something lke UnicodeDecodeError: 'utf-8' codec can't decode byte 0xed in position 20: invalid continuation byte in Python by default.

For example, the NewPipe APK contains a UTF-8-encoded string containing the 🔍 emoji (encoded as \xed\xa0\xbd\xed\xb4\x8d) in its resources.arsc. But I've seen those pairs show up in many APKs having emoji in a string (as those are not part of the Basic Multilingual Plane and thus encoded as a surrogate pair in UTF-16).

An example of the "length overflow" occurring in a non-malware APK is e.g. this APK.

Looking at it again now I just realised my linked code had a bug which I just pushed a fix for 😅

[WeissRu]

The following code can decode utf16 surrogate pairs, and calculate the length in Java style for comparison with the record in dex:

data1 = data.decode('utf-8','surrogatepass').encode('utf-16','surrogatepass')
len1 = (len(data1) - 2)//2 # sub 2 (for utf16 BOM \xff\xfe), then div 2 (for 2byte -> 1char)
string = data1.decode('utf-16')