macOS a64: cannot guarantee client reachability w/ default 1G vm_size

[ndrewh]

on macOS arm64 w/ default vm_size 1G

bin64/drrun -debug -c api/bin/libbbcount.dylib -- ./main

<Application /Users/andrew/dynamorio-macos/build/main (84912).  Internal Error: DynamoRIO debug check failure: /Users/andrew/dynamorio-macos/core/heap.c:537 heap_allowable_region_start <= must_reach_region_start && "x64 reachability contraints not satisfiable"
(Error occurred @0 frags in tid 84912)
version 11.90.20142, custom build
-client_lib '/Users/andrew/dynamorio-macos/build/api/bin/libbbcount.dylib;0;' -client_lib64 '/Users/andrew/dynamorio-macos/build/api/bin/libbbcount.dylib;0;' -code_api -stack_size 64K -signal_stack_size 32K -max_elide_jmp 0 -max_elide_call 0 -vmm_block_size 16K -initial_heap_unit_size 32K -initial_heap_nonpers_size 16K -

Sometimes, mmap returns a region that is not 32-bit reachable from the main application binary. It does not fail every time, but >30% of the time.

This issue does not occur if you limit the vm_size to 512M.

PR #7171 updates a64 macOS tests to use the 512M limit for now, but we may want to just change the default on a64 macOS for now.

[derekbruening]

Sometimes, mmap returns a region that is not 32-bit reachable from the main application binary.

Not being near the app executable should not lead to a fatal error, as you can see in the option description:

OPTION_DEFAULT(bool, vm_base_near_app, true,
               "allocate vm region near the app if possible (if not, if "
               "-vm_allow_not_at_base, will try elsewhere)")

So there must be something else going on.
But did you try -no_vm_base_near_app as a sanity check?

It sounds like there is something going wrong somewhere.

[ndrewh]

It still fails ~half the time w/ -no_vm_base_near_app.

It looks like OS wants to place vm region at 0x0000000300000000

os_heap_reserve: 1073758208 bytes @ 0x0000000300000000
vmm_heap_unit_init unable to allocate at preferred=0x000000012d2c0000 letting OS place sz=1024M addr=0x0000000300000000

client is at 0x0000000101290000

Adding must-be-reachable-from-heap region 0x0000000101290000-0x00000001012b0000
Existing must-be-reachable region 0x0000000300000000-0x000000033fffffff
Existing allowed range 0x00000002bfffffff-0x000000037fffffff
Added must-be-reachable-from-heap region 0x0000000101290000-0x00000001012b0000
New must-be-reachable region 0x0000000101290000-0x000000033fffffff
New allowed range 0x00000002bfffffff-0x000000018128ffff
SYSLOG_ERROR: Application /Users/andrew/pyda/tests/thread_10 (49698).  Internal Error: DynamoRIO debug check failure: /Users/andrew/dynamorio-macos/core/heap.c:537 heap_allowable_region_start <= must_reach_region_start && "x64 reachability contraints not satisfiable"

-no_reachable_client prevents the assert from being hit, though. I don't know what the consequences are of this?

[derekbruening]

Mac has no private loader, so it has much less control over where the client library ends up: so probably we should turn off -reachable_client on Mac and document we do not provide reachability concerns on that platform. The consequences are the client having to use absolute references from instrumentation to client library data sections or client library functions, including clean calls. The docs at https://dynamorio.org/using.html#sec_64bit_reach should be updated to mention Mac too.