CRASH -- drrun all executables on ARM32 from LDREXB mangling bug

[sab24]

Head version of dynamorio generates a SEGFAULT when running a simple drrun command both on rpi2 with armv7 and also with qemu userspace emulation.

glibc 2.41, gcc 14.2.1, Archlinux arm head

uname -an
Linux alarmpi 6.12.21-1-rpi #1 SMP Mon Mar 31 13:47:52 MDT 2025 armv7l GNU/Linux

git clone https://github.com/DynamoRIO/dynamorio.git
cd dynamorio
git submodule update --init
mkdir build && cd build
cmake ..
bin32/drrun -dumpcore_mask 0x8bff -- ls
<Application /usr/bin/ls (7118).  DynamoRIO internal crash at PC 0xb6e4dd2c.  Please report this at http://dynamorio.org/issues/.  Program aborted.
Received SIGSEGV at pc 0xb6e4dd2c in thread 7118
Base: 0xb6db6000
Registers:  r0 =0x00000000 r1 =0x00000000 r2 =0x00000002 r3 =0xe58a0000
        r4 =0x00000000 r5 =0x41f12a3c r6 =0x41f0ed24 r7 =0x00000000
        r8 =0x41f0ed28 r9 =0x00000000 r10=0x41f0ed20 r11=0x00000001
        r12=0x00000005 r13=0x41f0ed18 r14=0xb6e4dd1d r15=0xb6e4dd2c
        eflags=0x80081830
version 11.90.20183, custom build
-no_dynamic_options -code_api -dumpcore_mask 0x8bff -stack_size 56K -signal_stack_size 32K -max_elide_jmp 0 -max_elide_call 0 -early_inject -emulate_brk -no_inline_ignored_syscalls -native_exec_default_list '' -no_native_exec_managed_code -no_indcall2direct >
Reading symbols from /usr/bin/ls...
(No debugging symbols found in /usr/bin/ls)
"/home/alarm/dynamorio/build/core" is not a core dump: file format not recognized
(gdb) No stack.
(gdb) Working directory /home/alarm/dynamorio/build.
(gdb) Segmentation fault (core dumped)

with debug build, coredump files (2 generated

coredumps.tar.gz

) of this run are attached

mkdir builddebug && cd builddebug
cmake -DCMAKE_BUILD_TYPE=Debug ..
[alarm@alarmpi builddebug]$ bin32/drrun -debug -dumpcore_mask 0x8bff -- ls
<Starting application /usr/bin/ls (12576)>
<Initial options = -no_dynamic_options -code_api -dumpcore_mask 0x8bff -stack_size 56K -signal_stack_size 32K -max_elide_jmp 0 -max_elide_call 0 -early_inject -emulate_brk -no_inline_ignored_syscalls -native_exec_default_list '' -no_native_exec_managed_code -no_indcall2direct >
<Paste into GDB to debug DynamoRIO clients:
set confirm off
add-symbol-file '/home/alarm/dynamorio/builddebug/lib32/debug/libdynamorio.so' 0xb6bc57a0
>
<Destination operand #0 has wrong type/size>
<Application /usr/bin/ls (12576).  Internal Error: DynamoRIO debug check failure: /home/alarm/dynamorio/core/emit.c:363 pc != NULL
(Error occurred @1416 frags in tid 12576)
version 11.90.20183, custom build
-no_dynamic_options -code_api -dumpcore_mask 0x8bff -stack_size 56K -signal_stack_size 32K -max_elide_jmp 0 -max_elide_call 0 -early_inject -emulate_brk -no_inline_ignored_syscalls -native_exec_default_list '' -no_native_exec_managed_code -no_indcall2direct
0xb6c348df 0xf8af04b0
/home/alarm/dynamorio/builddebug/lib32/debug/libdynamorio.so=0xb6ba0000>
<------------------------------------------->
<stackdump: --- now running the debugger --->
<gdb -q /usr/bin/ls core>
<------------------------------------------->
Reading symbols from /usr/bin/ls...
(No debugging symbols found in /usr/bin/ls)
"/home/alarm/dynamorio/builddebug/core" is not a core dump: file format not recognized
(gdb) No stack.
(gdb) Working directory /home/alarm/dynamorio/builddebug.
(gdb) <------------------------------------------->
<Crashing the process deliberately for a core dump for: |DynamoRIO debug check failure: /home/alarm/dynamorio/core/emit.c:363 pc != NULL
(Error occurred @1416 frags in tid 12576)
version 11.90.20183, custom build
-no_dynamic_options -code_api -dumpcore_mask 0x8bff -stack_size 56K -signal_stack_size 32K -max_elide_jmp 0 -max_elide_call 0 -early_inject -emulate_brk -no_inline_ignored_syscalls -native_exec_default_list '' -no_native_exec_managed_code -no_indcall2direct
0xb6c348df 0xf8af04b0
/home/alarm/dynamorio/builddebug/lib32/debug/libdynamorio.so=0xb6ba0000|>
Segmentation fault (core dumped)

How to reproduce on X86:

cd ~/Downloads
git clone https://github.com/crosstool-ng/crosstool-ng
cd crosstool-ng
./bootstrap
./configure --enable-local
./ct-ng menuconfig

enable arm as target, select gcc newest version and glibc newest version. Enable C++ support for gcc so that libstdc++.so.6 will be built.

./ct-nt build

compile dynamorio, but first edit
make/toolchain_arm32.cmake
addd

set(TARGET_ABI "unknown-linux-gnueabi")

before the line

if (NOT DEFINED TARGET_ABI)

now compile dynamorio

 -cd /home/<username>/Downloads/dynamorio && rm -rf ./buildarm
    cd /home/<username>/Downloads/dynamorio && mkdir buildarm && cd buildarm && \
    PATH=/home/<username>/x-tools/arm-unknown-linux-gnueabi2.30/bin/:$$PATH && \
    cmake -DDEBUG=ON -DCMAKE_TOOLCHAIN_FILE=../make/toolchain-arm32.cmake -DCMAKE_C_LIBRARY_ARCHITECTURE=gnueabi -DBUILD_DOCS=NO -DTARGET_ABI='unknown-linux-gnueabi' -DCMAKE_SYSROOT=/home/<username>/x-tools/arm-unknown-linux-gnueabi2.30/arm-unknown-linux-gnueabi/sysroot/  -DZLIB_LIBRARY=/home/<username>/Downloads/zlib-arm-install/lib/libz.a -DZLIB_INCLUDE_DIR=/home/<username>/Downloads/zlib-arm-install/include/ .. && \
    grep -r -l "arm-unknown-linux-gnueabi-as --sysroot" | xargs sed -i 's/arm-unknown-linux-gnueabi-as --sysroot=\/home\/<username>\/x-tools\/arm-unknown-linux-gnueabi2\.30\/arm-unknown-linux-gnueabi\/sysroot\//arm-unknown-linux-gnueabi-as/' && \
    make

Have qemu user mode emulation installed, and a basic arm executable ready.

export QEMU_LD_PREFIX=/home/<username>/x-tools/arm-unknown-linux-gnueabi/arm-unknown-linux-gnueabi/sysroot/
qemu-arm-static bin32/drrun -debug -dumpcore_mask 0x8bff -- /home/<username>/Downloads/busybox-1.37.0/_install/bin/ls

When using a client, there is a floating point exception

qemu-arm-static bin32/drrun -debug -dumpcore_mask 0x8bff -c api/bin/libmemval_simple.so -- /home/<username>/Downloads/busybox-1.37.0/_install/bin/ls
<Starting application /home/<username>/Downloads/busybox-1.37.0/_install/bin/busybox (2992663)>
<Initial options = -no_dynamic_options -client_lib '/home/<username>/Downloads/dynamorio/buildarm/api/bin/libmemval_simple.so;0;' -client_lib32 '/home/<username>/Downloads/dynamorio/buildarm/api/bin/libmemval_simple.so;0;' -code_api -dumpcore_mask 0x8bff -stack_size 56K -signal_stack_size 32K -max_elide_jmp 0 -max_elide_call 0 -early_inject -emulate_brk -no_inline_ignored_syscalls -native_exec_default_list '' -no_native_exec_managed_code -no_indcall2direct >
<WARNING! symbol lookup error: libdrx.so undefined symbol drx_scatter_gather_thread_exit>
<WARNING! symbol lookup error: libdrx.so undefined symbol drx_scatter_gather_thread_init>
<WARNING! symbol lookup error: libdrx.so undefined symbol drx_scatter_gather_restore_state>
qemu: uncaught target signal 8 (Floating point exception) - core dumped
Floating point exception (core dumped)

[sab24]

cronbuild-8.0.18585 starts the application properly, but release_8.0.18611-2 fails with a SEGFAULT.

[abhinav92003]

Thanks for the detailed bug report. A few ideas below on how we can investigate and root-cause this.

When using a client, there is a floating point exception
...
version 11.90.20183, custom build

Only last week, we fixed an FPE on AArch32 running under QEMU with glibc 2.35 3. I'm surprised to see that "version 11.90.20183, custom build" (which is presumably from Saturday, 5 April 2025) continues showing this error. Can you check if the same error happens on any unit test (run our regression test suite using ctest6)?

select gcc newest version and glibc newest version.

Assuming you're using glibc 2.41 for both the real-hardware and QEMU runs: to help us narrow down the issue, is it possibly to retry with an older glibc? As I mentioned above, we recently fixed some issues that affected DynamoRIO on glibc 2.35; it's likely we need to add similar handling for newer glibc.

<Application /usr/bin/ls (12576). Internal Error: DynamoRIO debug check failure: /home/alarm/dynamorio/core/emit.c:363 pc != NULL

To get more details, there are some error logs4 that you can look for. Try running the debug build with -loglevel 1 -log_to_stderr 5.

cronbuild-8.0.18585 starts the application properly, but release_8.0.18611-2 fails with a SEGFAULT.

It's interesting that a 4+ year old build works.

I assume the above is the behavior on real hardware (rpi2 with armv7). Is the same also true when run under QEMU?

There were 36 commits between the releases 1 and 2 (cronbuild-8.0.18585...release_8.0.18611-2). May I ask if you could bisect the range to figure out when the failures started precisely?

[derekbruening]

I would also add a request to always get and provide a symbolized callstack for any SIGSEGV or SIGFPE so we don't have to speculate and guess where the crash is.

[sab24]

I would also add a request to always get and provide a symbolized callstack for any SIGSEGV or SIGFPE so we don't have to speculate and guess where the crash is.

Hi Derek,

How would you like me to give a stacktrace of a SIGILL or a SIGSEGV, usually the stack is corrupt right?

When running gdb:

[alarm@alarmpi builddebug]$ gdb -args bin32/drrun -debug -- ls
GNU gdb (GDB) 16.2
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "armv7l-unknown-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from bin32/drrun...
Reading symbols from /home/alarm/dynamorio/builddebug/bin32/drrun.debug...
(gdb) r
Starting program: /home/alarm/dynamorio/builddebug/bin32/drrun -debug -- ls
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
process 31652 is executing new program: /home/alarm/dynamorio/builddebug/lib32/debug/libdynamorio.so
<Starting application /usr/bin/ls (31652)>

Program received signal SIGILL, Illegal instruction.
syscall_0_or_1arg () at /home/alarm/dynamorio/core/drlibc/drlibc_arm.asm:94
94      /* FIXME i#1551: just a shell to get things compiling.  We need to fill
(gdb) bt
#0  syscall_0_or_1arg () at /home/alarm/dynamorio/core/drlibc/drlibc_arm.asm:94
#1  0xb6e92450 in thread_signal (pid=<error reading variable: Cannot access memory at address 0x118>, tid=<error reading variable: Cannot access memory at address 0x114>, signum=<error reading variable: Cannot access memory at address 0x110>) at /home/alarm/dynamorio/core/unix/os.c:3731
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

It does not give me much info.

[sab24]

Thanks for the detailed bug report. A few ideas below on how we can investigate and root-cause this.

When using a client, there is a floating point exception
...
version 11.90.20183, custom build

Only last week, we fixed an FPE on AArch32 running under QEMU with glibc 2.35 3. I'm surprised to see that "version 11.90.20183, custom build" (which is presumably from Saturday, 5 April 2025) continues showing this error. Can you check if the same error happens on any unit test (run our regression test suite using ctest6)?

select gcc newest version and glibc newest version.

Assuming you're using glibc 2.41 for both the real-hardware and QEMU runs: to help us narrow down the issue, is it possibly to retry with an older glibc? As I mentioned above, we recently fixed some issues that affected DynamoRIO on glibc 2.35; it's likely we need to add similar handling for newer glibc.

<Application /usr/bin/ls (12576). Internal Error: DynamoRIO debug check failure: /home/alarm/dynamorio/core/emit.c:363 pc != NULL

To get more details, there are some error logs4 that you can look for. Try running the debug build with -loglevel 1 -log_to_stderr 5.

cronbuild-8.0.18585 starts the application properly, but release_8.0.18611-2 fails with a SEGFAULT.

It's interesting that a 4+ year old build works.

I assume the above is the behavior on real hardware (rpi2 with armv7). Is the same also true when run under QEMU?

There were 36 commits between the releases 1 and 2 (cronbuild-8.0.18585...release_8.0.18611-2). May I ask if you could bisect the range to figure out when the failures started precisely?

Thanks for the suggestions. I am disecting the older commits, but I am having difficulties compiling. Many errors pop up that are caused by a too new libc version, a too new gcc version and a too new cmake version. The last error is probably due to the toolchain not using fpu operations.

/home/<username>/x-tools/arm-unknown-linux-gnueabi/lib/gcc/arm-unknown-linux-gnueabi/14.2.0/../../../../arm-unknown-linux-gnueabi/bin/ld: ../bin/tracedump: hidden symbol `__aeabi_dcmple' in /home/<username>/x-tools/arm-unknown-linux-gnueabi/lib/gcc/arm-unknown-linux-gnueabi/14.2.0/libgcc.a(_arm_cmpdf2.o) is referenced by DSO
/home/<username>/x-tools/arm-unknown-linux-gnueabi/lib/gcc/arm-unknown-linux-gnueabi/14.2.0/../../../../arm-unknown-linux-gnueabi/bin/ld: final link failed: bad value

Probably have to recompile the toolchain again with different floating point settings or manually modify the static archive to make the symbol visible.

I tried running dyanmorio after compiling with toolchains that use glibc versions 2.41, 2.30, and 2.17. I recompiled only dynamorio, not busybox, so does the target application also needs to be recompiled with a newer glibc?

I did not yet get it running in qemu:

[<username>@thinkbooklinux DynamoRIO-ARM-Linux-EABIHF-8.0.18585]$ qemu-arm-static -L /home/<username>/x-tools/arm-unknown-linux-gnueabi/arm-unknown-linux-gnueabi/sysroot/  bin32/drrun -debug -- /home/<username>/Downloads/busybox-1.37.0/_install/bin/busybox
<Application /home/<username>/Downloads/busybox-1.37.0/_install/bin/busybox (3198782) DynamoRIO usage error : Failed to read ELF interpreter headers.>
<Usage error: Failed to read ELF interpreter headers. (/home/travis/build/DynamoRIO/dynamorio/core/unix/loader.c, line 2000)
version 8.0.18585, custom build
-early_inject -emulate_brk >

[derekbruening]

How would you like me to give a stacktrace of a SIGILL or a SIGSEGV, usually the stack is corrupt right?

You should be able to get a reasonable callstack for these.

Program received signal SIGILL, Illegal instruction.
...
#0 syscall_0_or_1arg () at /home/alarm/dynamorio/core/drlibc/drlibc_arm.asm:94
#1 0xb6e92450 in thread_signal
...
It does not give me much info.

Please see https://dynamorio.org/page_debugging.html#autotoc_md142. You can see from "thread_signal" and being after a system call (SYS_kill) that this is just DR sending a SIGILL which it uses to take over threads and for nudges. Just continue past it. You want to only stop at the fatal SIGSEGV you hit. (Continue past non-fatal safe-read SIGSEGV as well, as noted in the documentation.)

[sab24]

How would you like me to give a stacktrace of a SIGILL or a SIGSEGV, usually the stack is corrupt right?

You should be able to get a reasonable callstack for these.

Program received signal SIGILL, Illegal instruction.
...
#0 syscall_0_or_1arg () at /home/alarm/dynamorio/core/drlibc/drlibc_arm.asm:94
#1 0xb6e92450 in thread_signal
...
It does not give me much info.

Please see https://dynamorio.org/page_debugging.html#autotoc_md142. You can see from "thread_signal" and being after a system call (SYS_kill) that this is just DR sending a SIGILL which it uses to take over threads and for nudges. Just continue past it. You want to only stop at the fatal SIGSEGV you hit. (Continue past non-fatal safe-read SIGSEGV as well, as noted in the documentation.)

Ah, I see, DynamoRIO is first catching the signals for communication. When continuing there is only an assert failure in /home/alarm/dynamorio/core/emit.c:363 :

[alarm@alarmpi builddebug]$ gdb -args bin32/drrun -debug -- ls
GNU gdb (GDB) 16.2
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "armv7l-unknown-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from bin32/drrun...
Reading symbols from /home/alarm/dynamorio/builddebug/bin32/drrun.debug...
(gdb) set confirm off
(gdb) add-symbol-file '/home/alarm/dynamorio/builddebug/lib32/debug/libdynamorio.so' 0xb6cb87a0
add symbol table from file "/home/alarm/dynamorio/builddebug/lib32/debug/libdynamorio.so" at
        .text_addr = 0xb6cb87a0
Reading symbols from /home/alarm/dynamorio/builddebug/lib32/debug/libdynamorio.so...
Reading symbols from /home/alarm/dynamorio/builddebug/lib32/debug/libdynamorio.so.debug...
(gdb) r
Starting program: /home/alarm/dynamorio/builddebug/bin32/drrun -debug -- ls
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
process 31699 is executing new program: /home/alarm/dynamorio/builddebug/lib32/debug/libdynamorio.so
<Starting application /usr/bin/ls (31699)>

Program received signal SIGILL, Illegal instruction.
syscall_0_or_1arg () at /home/alarm/dynamorio/core/drlibc/drlibc_arm.asm:94
94      /* FIXME i#1551: just a shell to get things compiling.  We need to fill
(gdb) c
Continuing.
<Initial options = -no_dynamic_options -code_api -stack_size 56K -signal_stack_size 32K -max_elide_jmp 0 -max_elide_call 0 -early_inject -emulate_brk -no_inline_ignored_syscalls -native_exec_default_list '' -no_native_exec_managed_code -no_indcall2direct >
<Paste into GDB to debug DynamoRIO clients:
set confirm off
add-symbol-file '/home/alarm/dynamorio/builddebug/lib32/debug/libdynamorio.so' 0xb6cb87a0
>
<Destination operand #0 has wrong type/size>
<Application /usr/bin/ls (31699).  Internal Error: DynamoRIO debug check failure: /home/alarm/dynamorio/core/emit.c:363 pc != NULL
(Error occurred @1418 frags in tid 31699)
version 11.90.20183, custom build
-no_dynamic_options -code_api -stack_size 56K -signal_stack_size 32K -max_elide_jmp 0 -max_elide_call 0 -early_inject -emulate_brk -no_inline_ignored_syscalls -native_exec_default_list '' -no_native_exec_managed_code -no_indcall2direct
0xb6d278df 0xf8af04b0
/home/alarm/dynamorio/builddebug/lib32/debug/libdynamorio.so=0xb6c93000>
[Inferior 1 (process 31699) exited with code 0377]
(gdb) bt
No stack.
(gdb) c
The program is not being run.

When putting a breakpoint at that location:

[alarm@alarmpi builddebug]$ gdb -args bin32/drrun -debug -- ls
GNU gdb (GDB) 16.2
Copyright (C) 2024 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "armv7l-unknown-linux-gnueabihf".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from bin32/drrun...
Reading symbols from /home/alarm/dynamorio/builddebug/bin32/drrun.debug...
(gdb) b emit.c:361
No source file named emit.c.
Make breakpoint pending on future shared library load? (y or [n]) n
(gdb) add-symbol-file '/home/alarm/dynamorio/builddebug/lib32/debug/libdynamorio.so' 0xb6cb87a0
add symbol table from file "/home/alarm/dynamorio/builddebug/lib32/debug/libdynamorio.so" at
        .text_addr = 0xb6cb87a0
(y or n) y
Reading symbols from /home/alarm/dynamorio/builddebug/lib32/debug/libdynamorio.so...
Reading symbols from /home/alarm/dynamorio/builddebug/lib32/debug/libdynamorio.so.debug...
(gdb) b emit.c:361
Breakpoint 1 at 0xb6d368a2: file /home/alarm/dynamorio/core/emit.c, line 361.
(gdb) r
Starting program: /home/alarm/dynamorio/builddebug/bin32/drrun -debug -- ls
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/usr/lib/libthread_db.so.1".
process 31833 is executing new program: /home/alarm/dynamorio/builddebug/lib32/debug/libdynamorio.so
<Starting application /usr/bin/ls (31833)>

Program received signal SIGILL, Illegal instruction.
syscall_0_or_1arg () at /home/alarm/dynamorio/core/drlibc/drlibc_arm.asm:94
94      /* FIXME i#1551: just a shell to get things compiling.  We need to fill
(gdb) i b
Num     Type           Disp Enb Address    What
1       breakpoint     keep y   0xb6d368a2 in set_linkstub_fields at /home/alarm/dynamorio/core/emit.c:361
(gdb) c
Continuing.
<Initial options = -no_dynamic_options -code_api -stack_size 56K -signal_stack_size 32K -max_elide_jmp 0 -max_elide_call 0 -early_inject -emulate_brk -no_inline_ignored_syscalls -native_exec_default_list '' -no_native_exec_managed_code -no_indcall2direct >
<Paste into GDB to debug DynamoRIO clients:
set confirm off
add-symbol-file '/home/alarm/dynamorio/builddebug/lib32/debug/libdynamorio.so' 0xb6cb87a0
>

Breakpoint 1, set_linkstub_fields (dcontext=0x45a90160, f=0x45a6f418, ilist=0x45adb928, num_direct_stubs=1, num_indirect_stubs=0, emit=true) at /home/alarm/dynamorio/core/emit.c:361
361                     pc = instr_encode_to_copy(dcontext, inst, vmcode_get_writable_addr(pc),
(gdb) p *dcontext
$1 = {upcontext = {separate_upcontext = 0x0, upcontext = {mcontext = {r0 = 0, r1 = 0, r2 = 0, r3 = 0, r4 = 0, r5 = 0, r6 = 0, r7 = 0, r8 = 0, r9 = 0, r10 = 0, r11 = 0, r12 = 0, {r13 = 3204447072, sp = 3204447072, xsp = 3204447072}, {r14 = 0, lr = 0}, {r15 = 0, pc = 0x0}, {xflags = 0, apsr = 0, cpsr = 0}, padding = "\000\000\000", simd = {
          {s = {0, 0, 0, 0}, d = {0, 0, 0, 0}, u32 = {0, 0, 0, 0}} <repeats 16 times>}}, dr_errno = 0, at_syscall = false, exit_reason = 0, inline_spill_slots = {0, 0, 0, 0, 0}}}, upcontext_ptr = 0x45a90160, next_tag = 0xb6c7c3a0 "8\240\237\345\r", last_exit = 0xb6ef8fc0 <linkstub_starting>, dstack = 0x45ad9000 "", is_exiting = false,
  coarse_exit = {src_tag = 0x0, dir_exit = 0x0}, whereami = DR_WHERE_INTERP, signals_pending = 0 '\000', initialized = true, owning_thread = 31833, owning_process = 31833, thread_record = 0x45a8a1b0, allocated_start = 0x45a90160, last_fragment = 0xb6ef8f9c <linkstub_empty_fragment>, sys_num = 0, sys_param0 = 0, sys_param1 = 0,
  sys_param2 = 0, sys_param3 = 0, sys_param4 = 0, sys_was_int = false, sys_xbp = false, mprot_multi_areas = false, isa_mode = DR_ISA_ARM_A32, encode_state = {0, 0}, decode_state = {0, 0}, link_field = 0x45adb7b8, monitor_field = 0x45adb6a0, fcache_field = 0x45adb7a0, fragment_field = 0x45a90bf8, heap_field = 0x45a8ae10,
  vm_areas_field = 0x45adb5a0, os_field = 0x45adb028, synch_field = 0x45adb578, signal_field = 0x45adb130, pcprofile_field = 0x0, private_code = 0x0, asynch_target = 0x0, native_exec_postsyscall = 0x0, native_retstack = {{retaddr = 0x0, retloc = 0x0}, {retaddr = 0x0, retloc = 0x0}, {retaddr = 0x0, retloc = 0x0}, {retaddr = 0x0,
      retloc = 0x0}, {retaddr = 0x0, retloc = 0x0}, {retaddr = 0x0, retloc = 0x0}, {retaddr = 0x0, retloc = 0x0}, {retaddr = 0x0, retloc = 0x0}, {retaddr = 0x0, retloc = 0x0}, {retaddr = 0x0, retloc = 0x0}}, native_retstack_cur = 0, logfile = -1, thread_stats = 0x45a65048, expect_last_syscall_to_fail = false, in_opnd_disassemble = false,
  thread_owned_locks = 0x45a65038, thread_kstats = 0x0, client_data = 0x45adb7f8, is_client_thread_exiting = false, trace_sysenter_exit = false, forged_exception_addr = 0x0, single_step_addr = 0x0, try_except = {try_except_state = 0x0, unwinding_exception = false}, local_state = 0x45ac4000, bb_build_info = 0x45ad8e0c,
  nudge_pending = 0x0, interrupted_for_nudge = 0x0, libc_errno = 0, currently_stopped = false, go_native = false, rseq_entry_state = {gpr = {0 <repeats 16 times>}}}
(gdb) bt
#0  set_linkstub_fields (dcontext=0x45a90160, f=0x45a6f418, ilist=0x45adb928, num_direct_stubs=1, num_indirect_stubs=0, emit=true) at /home/alarm/dynamorio/core/emit.c:361
#1  0xb6d3a516 in emit_fragment_common (dcontext=0x45a90160, tag=0xb6c7c3a0 "8\240\237\345\r", ilist=0x45adb928, flags=16777216, vmlist=0x45a6f400, link_fragment=true, add_to_htable=true, replace_fragment=0x0) at /home/alarm/dynamorio/core/emit.c:601
#2  0xb6d3e4d4 in emit_fragment_ex (dcontext=0x45a90160, tag=0xb6c7c3a0 "8\240\237\345\r", ilist=0x45adb928, flags=16777216, vmlist=0x45a6f400, link=true, visible=true) at /home/alarm/dynamorio/core/emit.c:854
#3  0xb6e6ca42 in build_basic_block_fragment (dcontext=0x45a90160, start=0xb6c7c3a0 "8\240\237\345\r", initial_flags=0, link=true, visible=true, for_trace=false, unmangled_ilist=0x0) at /home/alarm/dynamorio/core/arch/interp.c:5166
#4  0xb6d28150 in d_r_dispatch (dcontext=0x45a90160) at /home/alarm/dynamorio/core/dispatch.c:211
#5  0xb6e8d6b0 in call_dispatch_alt_stack_no_free () at /home/alarm/dynamorio/core/arch/arm/arm.asm:96
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) p *inst
$2 = {flags = 1075904512, encoding_hints = 0, length = 0, category = 0, {bytes = 0x0, label_cb = 0x0}, translation = 0x0, opcode = 3, {operation_size = 0 '\000'}, num_dsts = 0 '\000', num_srcs = 0 '\000', isa_mode = 2 '\002', {{src0 = {kind = 0 '\000', size = 0 '\000', aux = {far_pc_seg_selector = 0, segment = 0, disp = 0, shift = 0,
          flags = 0}, value = {immed_int = 0, immed_int_multi_part = {low = 0, high = 0}, immed_float = 0, immed_double = 0, pc = 0x0, instr = 0x0, reg_and_element_size = {reg = 0, element_size = 0 '\000'}, base_disp = {disp = 0, base_reg = 0, index_reg = 0, shift_type = 0 '\000', shift_amount_minus_1 = 0 '\000'}, addr = 0x0}},
      srcs = 0x0, dsts = 0x0}, label_data = {data = {0, 0, 0, 0}}}, prefixes = 0, eflags = 0, note = 0x0, prev = 0x0, next = 0x45adbb10, offset = 0}
(gdb) p vmcode_get_writable_addr(pc)
$3 = (byte *) 0x45ab4008 '\253' <repeats 200 times>...
(gdb)

What more info can I send you?

[sab24]

Now with more logging and a conditional breakpoint on when the pc==NULL

https://gist.github.com/sab24/6da8cc4f0ed323ff8003ec88c10cdeb7

[abhinav92003]

Thanks for sharing the logs. Please use http://gist.github.com/ for large snippets. Adding them whole makes the issue page unnecessarily long and less readable.

From your logs, here's the culprit:

SYSLOG_ERROR: Destination operand #0 has wrong type/size
SYSLOG_ERROR: Destination operand #0 has wrong type/size
<Destination operand #0 has wrong type/size>
ERROR: Could not find encoding for: ldrb   (%r4)[1byte] -> %r3[1byte]
Reason: Destination operand #0 has wrong type/size

[sab24]

So the instructions cbz cbnz and ldrb are not yet implemented inside the encoding implementation of DR? So I should just look them up inside the ARM architecture reference manual and add them to DR?

[abhinav92003]

Could you please edit your above comments to replace the long snippets with gist.github.com links?

Re the bad instruction:

I see an ldrb is added by DR's mangling on ARM32 at 1 and 2. The issue you hit is likely 2 (because if it were 1, we'd have seen a DR_REG_R2 as dest)

dynamorio/core/arch/aarchxx/mangle.c

Line 3127 in 9232385

case OP_ldrexb: return INSTR_CREATE_ldrb(dcontext, regop, memop);

Could you try modifying the above line to something like the following?

case OP_ldrexb: return INSTR_CREATE_ldrb(dcontext, opnd_create_reg(reg_resize_to_opsz(regop, OPSZ_4)), memop);

Notes:

• I'm trying to match the following existing usage of INSTR_CREATE_ldrb where we do the same
  

  dynamorio/ext/drx/drx_buf.c

  Line 726 in 9232385

  drcontext, opnd_create_reg(reg_resize_to_opsz(src, opsz)),

• Also, there's this comment

  dynamorio/core/ir/opnd_shared.c

  Line 2555 in 9232385

  /* On ARM, we use the same reg for the size of 8, 16, and 32 bit */

[abhinav92003]

So the instructions cbz cbnz and ldrb are not yet implemented inside the encoding implementation of DR?

Those instructions are pretty common and are already supported by DR's decoder. As my previous comment notes, there's likely a bug in how we specify the destination register in core/arch/aarchxx/mangle.c that needs to be corrected.