Thumb2 decoder generator fails to run out-of-the-box due to `grako` not being updated for python 3.10

[jonathanzetier]

Version and Platform (required):

• Binary Ninja Version: 5.0.7218-dev
• OS: Ubuntu
• OS Version: 22.04
• CPU Architecture: x64

Bug Description:
Dependency on a dead pip dependency, grako

I've got a bug fix I'd like to make for thumb2, but I can't run binaryninja-api/arch/armv7/thumb2_disasm/generator.py because it depends on grako, which hasn't been updated in 8 years. grako fails to import with the following error:

ImportError: cannot import name 'Mapping' from 'collections' (/usr/lib/python3.10/collections/__init__.py)

In python 3.10, Mapping was moved from collections to collections.abc.

Steps To Reproduce:

pip install grako
cd binaryninja-api/arch/armv7/thumb2_disasm
./generator.py

Expected Behavior:
One potential solution is to vendor grako into the thumb2 directory so that it can be updated as python3 changes. I'm sure workarounds exist from people who know python/python dependency management better than me, but the longer this goes on the more interesting those workarounds get.

Screenshots/Video Recording:
N/A

Binary:
N/A

Additional Information:
Please add any other context about the problem here.

[galenbwill]

I am aware of this issue. I am already in the process of addressing this, but I do not have a time line for when it will be done.

If you really need to regenerate the disassembler, tatsu is an almost drop-in replacement for grako, but I expect there will be some things you'll need to fix up to get it to work.

[galenbwill]

Also, what is the nature of your bug fix? Can it be patched in manually or does it actually require regenerating the disassembler?

[jonathanzetier]

The bug is that thumb mode doesn't disassemble MSR/MRS (banked register) properly, it's getting lumped in with other flavors of MSR/MRS instructions. It could probably be manually patched, but I worried it would cause maintenance headaches every time the disassembler needs to be regenerated.

[galenbwill]

Can you provide an illustration of specifically what is going wrong? As it happens I am working on the lifting of those instructions and it would be helpful to know what is wrong.

[jonathanzetier]

Oh neat! I think what's happening is that there are several different instructions that all share the same opcode, but have slightly different meanings; for example, there's MSR (immediate), there's MSR, and there's MSR (banked register). It seems that binaryninja doesn't distinguish MSR (banked register) from MSR, so it's printing nonsensical disassembly: it's attempting to interpret a bitfield of the instruction as a part of the CPSR for the MSR instruction, and getting a nonsensical value.

Feel free to use the thumb binary from the PR I made a few weeks ago to add support for these instructions in ghidra (which flat out didn't even try to disassemble them): NationalSecurityAgency/ghidra#8013. Just open it in a mapped view at 0, choose thumb2, and start disassembling at 0. Here's some excerpts of their differences:

Binaryninja:

00000000  e0f32080   mrs     r0, 
00000004  e0f32081   mrs     r1, 
00000008  e0f32082   mrs     r2, 
0000000c  e0f32083   mrs     r3, 
00000010  e0f32084   mrs     r4, 
00000014  e0f32085   mrs     r5, 
00000018  e0f32086   mrs     r6, 
0000001c  e0f32087   mrs     r7, 
00000020  e0f32088   mrs     r8, 
00000024  e0f32089   mrs     r9, 
00000028  e0f3208a   mrs     r10, 
0000002c  e0f3208b   mrs     r11, 
00000030  e0f3208c   mrs     r12, 
00000034  e0f3208d   mrs     sp, 
00000038  e0f3208e   mrs     lr, 
0000003c  e0f3208f   mrs     pc, 
00000040  e1f32080   mrs     r0, 
00000044  e1f32081   mrs     r1, 
00000048  e1f32082   mrs     r2, 
0000004c  e1f32083   mrs     r3, 
00000050  e1f32084   mrs     r4, 

...

00001220  88f32088   msr     , r8
00001224  89f32088   msr     , r9
00001228  8af32088   msr     , r10
0000122c  8bf32088   msr     , r11
00001230  8cf32088   msr     , r12
00001234  8df32088   msr     , sp
00001238  8ef32088   msr     , lr
0000123c  8ff32088   msr     , pc
00001240  80f32089   msr     cpsr_fc, r0
00001244  81f32089   msr     cpsr_fc, r1
00001248  82f32089   msr     cpsr_fc, r2
0000124c  83f32089   msr     cpsr_fc, r3
00001250  84f32089   msr     cpsr_fc, r4
00001254  85f32089   msr     cpsr_fc, r5
00001258  86f32089   msr     cpsr_fc, r6

Objdump:

       0:	f3e0 8020 	mrs	r0, R8_usr
       4:	f3e0 8120 	mrs	r1, R8_usr
       8:	f3e0 8220 	mrs	r2, R8_usr
       c:	f3e0 8320 	mrs	r3, R8_usr
      10:	f3e0 8420 	mrs	r4, R8_usr
      14:	f3e0 8520 	mrs	r5, R8_usr
      18:	f3e0 8620 	mrs	r6, R8_usr
      1c:	f3e0 8720 	mrs	r7, R8_usr
      20:	f3e0 8820 	mrs	r8, R8_usr
      24:	f3e0 8920 	mrs	r9, R8_usr
      28:	f3e0 8a20 	mrs	r10, R8_usr
      2c:	f3e0 8b20 	mrs	r11, R8_usr
      30:	f3e0 8c20 	mrs	r12, R8_usr
      34:	f3e0 8d20 	mrs	sp, R8_usr
      38:	f3e0 8e20 	mrs	lr, R8_usr
      3c:	f3e0 8f20 	mrs	pc, R8_usr
      40:	f3e1 8020 	mrs	r0, R9_usr
      44:	f3e1 8120 	mrs	r1, R9_usr
      48:	f3e1 8220 	mrs	r2, R9_usr
      4c:	f3e1 8320 	mrs	r3, R9_usr
      50:	f3e1 8420 	mrs	r4, R9_usr

...

    1220:	f388 8820 	msr	R8_fiq, r8
    1224:	f389 8820 	msr	R8_fiq, r9
    1228:	f38a 8820 	msr	R8_fiq, r10
    122c:	f38b 8820 	msr	R8_fiq, r11
    1230:	f38c 8820 	msr	R8_fiq, r12
    1234:	f38d 8820 	msr	R8_fiq, sp
    1238:	f38e 8820 	msr	R8_fiq, lr
    123c:	f38f 8820 	msr	R8_fiq, pc
    1240:	f380 8920 	msr	R9_fiq, r0
    1244:	f381 8920 	msr	R9_fiq, r1
    1248:	f382 8920 	msr	R9_fiq, r2
    124c:	f383 8920 	msr	R9_fiq, r3
    1250:	f384 8920 	msr	R9_fiq, r4
    1254:	f385 8920 	msr	R9_fiq, r5
    1258:	f386 8920 	msr	R9_fiq, r6