AddressSanitizer: unknown-crash in `umfMemoryTrackerAdd()`

[ldorau]

ERROR: AddressSanitizer: unknown-crash on address :
https://github.com/ldorau/unified-memory-framework/actions/runs/14172971689/job/39700871936

[ RUN      ] disjointPoolTests/umfPoolTest.multiThreadedpow2AlignedAlloc/0
=================================================================
==1159214==ERROR: AddressSanitizer: unknown-crash on address 0x7f3e9feac248 at pc 0x5621d13ce718 bp 0x7f3e9a9fea00 sp 0x7f3e9a9fe9f0
READ of size 8 at 0x7f3e9feac248 thread T9
    #0 0x5621d13ce717 in utils_atomic_load_acquire_u64 /home/runner/work/unified-memory-framework/unified-memory-framework/src/utils/utils_concurrency.h:166
    #1 0x5621d13cfe1c in umfMemoryTrackerAdd /home/runner/work/unified-memory-framework/unified-memory-framework/src/provider/provider_tracking.c:202
    #2 0x5621d13d29b0 in trackingAlloc /home/runner/work/unified-memory-framework/unified-memory-framework/src/provider/provider_tracking.c:481
    #3 0x5621d13caa1e in umfMemoryProviderAlloc /home/runner/work/unified-memory-framework/unified-memory-framework/src/memory_provider.c:245
    #4 0x5621d13eaf23 in disjoint_pool_aligned_malloc /home/runner/work/unified-memory-framework/unified-memory-framework/src/pool/pool_disjoint.c:745
    #5 0x5621d13c7ea2 in umfPoolAlignedMalloc /home/runner/work/unified-memory-framework/unified-memory-framework/src/memory_pool.c:195
    #6 0x5621d12b6d99 in pow2AlignedAllocHelper(umf_memory_pool_t*) /home/runner/work/unified-memory-framework/unified-memory-framework/test/poolFixtures.hpp:190
    #7 0x5621d12b8cf1 in operator() /home/runner/work/unified-memory-framework/unified-memory-framework/test/poolFixtures.hpp:250
    #8 0x5621d12f1c43 in __invoke_impl<void, umfPoolTest_multiThreadedpow2AlignedAlloc_Test::TestBody()::<lambda(umf_memory_pool_handle_t)>, umf_memory_pool_t*> /usr/include/c++/11/bits/invoke.h:61
    #9 0x5621d12f11d0 in __invoke<umfPoolTest_multiThreadedpow2AlignedAlloc_Test::TestBody()::<lambda(umf_memory_pool_handle_t)>, umf_memory_pool_t*> /usr/include/c++/11/bits/invoke.h:96
    #10 0x5621d12f0270 in _M_invoke<0, 1> /usr/include/c++/11/bits/std_thread.h:259
    #11 0x5621d12ef8f1 in operator() /usr/include/c++/11/bits/std_thread.h:266
    #12 0x5621d12ef74e in _M_run /usr/include/c++/11/bits/std_thread.h:211
    #13 0x7f3ea00dc252  (/lib/x86_64-linux-gnu/libstdc++.so.6+0xdc252)
    #14 0x7f3e9f494ac2  (/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)
    #15 0x7f3e9f52684f  (/lib/x86_64-linux-gnu/libc.so.6+0x12684f)
2025-03-31T17:41:35.3852388Z
Address 0x7f3e9feac248 is a wild pointer.
SUMMARY: AddressSanitizer: unknown-crash /home/runner/work/unified-memory-framework/unified-memory-framework/src/utils/utils_concurrency.h:166 in utils_atomic_load_acquire_u64
Shadow bytes around the buggy address:
  0x0fe853fcd7f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0fe853fcd800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f7
  0x0fe853fcd810: f7 f7 f7 f7 f7 00 00 00 00 00 00 00 00 00 00 00
  0x0fe853fcd820: 00 00 00 00 00 00 00 00 00 00 f7 f7 f7 f7 f7 f7
  0x0fe853fcd830: 00 00 00 00 00 00 f7 f7 f7 f7 f7 f7 00 00 00 f7
=>0x0fe853fcd840: f7 f7 00 00 00 f7 f7 f7 f7[f7]f7 f7 f7 f7 f7 f7
  0x0fe853fcd850: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x0fe853fcd860: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x0fe853fcd870: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x0fe853fcd880: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x0fe853fcd890: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7

or ERROR: AddressSanitizer: use-after-poison:
https://github.com/ldorau/unified-memory-framework/actions/runs/14173082582/job/39701246433

[ RUN      ] mallocPoolTest/umfPoolTest.multiThreadedMallocFree/1
=================================================================
==1562263==ERROR: AddressSanitizer: use-after-poison on address 0x7fc68a62e218 at pc 0x5587ac5a19b8 bp 0x7fc67e7feb30 sp 0x7fc67e7feb20
READ of size 8 at 0x7fc68a62e218 thread T19
    #0 0x5587ac5a19b7 in utils_atomic_load_acquire_u64 /home/runner/work/unified-memory-framework/unified-memory-framework/src/utils/utils_concurrency.h:166
    #1 0x5587ac5a30bc in umfMemoryTrackerAdd /home/runner/work/unified-memory-framework/unified-memory-framework/src/provider/provider_tracking.c:202
    #2 0x5587ac5a5c50 in trackingAlloc /home/runner/work/unified-memory-framework/unified-memory-framework/src/provider/provider_tracking.c:481
    #3 0x5587ac59dcbe in umfMemoryProviderAlloc /home/runner/work/unified-memory-framework/unified-memory-framework/src/memory_provider.c:245
    #4 0x5587ac5c0f10 in proxy_aligned_malloc /home/runner/work/unified-memory-framework/unified-memory-framework/src/pool/pool_proxy.c:51
    #5 0x5587ac5c1036 in proxy_malloc /home/runner/work/unified-memory-framework/unified-memory-framework/src/pool/pool_proxy.c:64
    #6 0x5587ac59b010 in umfPoolMalloc /home/runner/work/unified-memory-framework/unified-memory-framework/src/memory_pool.c:189
    #7 0x5587ac409e51 in operator() /home/runner/work/unified-memory-framework/unified-memory-framework/test/poolFixtures.hpp:223
    #8 0x5587ac453b77 in __invoke_impl<void, umfPoolTest_multiThreadedMallocFree_Test::TestBody()::<lambda(size_t, umf_memory_pool_handle_t)>, long unsigned int, umf_memory_pool_t*> /usr/include/c++/11/bits/invoke.h:61
    #9 0x5587ac45309c in __invoke<umfPoolTest_multiThreadedMallocFree_Test::TestBody()::<lambda(size_t, umf_memory_pool_handle_t)>, long unsigned int, umf_memory_pool_t*> /usr/include/c++/11/bits/invoke.h:96
    #10 0x5587ac451e4d in _M_invoke<0, 1, 2> /usr/include/c++/11/bits/std_thread.h:259
    #11 0x5587ac4511ed in operator() /usr/include/c++/11/bits/std_thread.h:266
    #12 0x5587ac450fc2 in _M_run /usr/include/c++/11/bits/std_thread.h:211
    #13 0x7fc68a4dc252  (/lib/x86_64-linux-gnu/libstdc++.so.6+0xdc252)
    #14 0x7fc689894ac2  (/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)
    #15 0x7fc68992684f  (/lib/x86_64-linux-gnu/libc.so.6+0x12684f)
2025-03-31T19:11:14.9259311Z
Address 0x7fc68a62e218 is a wild pointer.
SUMMARY: AddressSanitizer: use-after-poison /home/runner/work/unified-memory-framework/unified-memory-framework/src/utils/utils_concurrency.h:166 in utils_atomic_load_acquire_u64
Shadow bytes around the buggy address:
  0x0ff9514bdbf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x0ff9514bdc00: 00 00 00 00 00 00 00 00 00 00 00 00 f7 f7 f7 f7
  0x0ff9514bdc10: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 00 00 00 00 00
  0x0ff9514bdc20: 00 f7 f7 f7 f7 f7 f7 f7 f7 f7 00 00 00 f7 f7 f7
  0x0ff9514bdc30: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
=>0x0ff9514bdc40: f7 f7 f7[f7]f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x0ff9514bdc50: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x0ff9514bdc60: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x0ff9514bdc70: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x0ff9514bdc80: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
  0x0ff9514bdc90: f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7 f7
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7

Environment Information

• UMF version (hash commit or a tag): commit e3d2929
• OS(es) version(s): Linux

Please provide a reproduction of the bug:

AddressSanitizer: unknown-crash:
https://github.com/ldorau/unified-memory-framework/actions/runs/14172971689/job/39700871936
https://github.com/ldorau/unified-memory-framework/actions/runs/14173082582/job/39701246468
https://github.com/ldorau/unified-memory-framework/actions/runs/14172971689/job/39700871892
https://github.com/ldorau/unified-memory-framework/actions/runs/14172971689/job/39700871926

or AddressSanitizer: use-after-poison:

https://github.com/ldorau/unified-memory-framework/actions/runs/14173082582/job/39701246433
https://github.com/ldorau/unified-memory-framework/actions/runs/14173082582/job/39701246427

How often bug is revealed:

very rare (when tests are looped)

Details

Additional information about Priority and Help Requested:

Requested priority: Medium

[ldorau]

This is the real scenario (see also: #1270 (comment)):
a) thread T2 allocates memory region R (for example: 0x100-0x300 - address 0x100 and size 0x200) and adds it to the tracker (this is the only entry in the tracker yet) and gets preempted ...
b) thread T1 allocates memory of size 0x500 and receives pointer P (address 0x400, size 0x500 - range 0x400-0x900) and tries to add it to the tracker, so it calls umfMemoryTrackerAdd(), so critnib_find(FIND_LE) finds and returns the only region R existing in the tracker R: 0x100-0x300 and T1 gets preempted ...
c) thread T2 removes the region R from the tracker, frees its memory and T2 gets preempted ...
d) thread T1 tries to read a size of the already freed region R and it crashes ...

[ldorau]

Fix for this issue: #1270