Add resident device change call #1517

lslusarczyk · 2025-09-11T12:30:55Z

UMF part of https://github.com/intel/llvm/blob/sycl/sycl/doc/extensions/supported/sycl_ext_oneapi_peer_access.asciidoc feature.

UR/llvm part is in intel/llvm#19257

include/umf/memory_pool_ops.h

src/libumf.map

src/memory_provider.c

src/provider/provider_level_zero.c

src/provider/provider_cuda.c

src/provider/provider_tracking.c

test/common/provider_null.c

src/provider/provider_level_zero.c

src/critnib/critnib.c

src/libumf.map

src/libumf.def

src/memory_provider.c

include/umf/memory_provider_ops.h

include/umf/providers/provider_level_zero.h

ldorau

14 / 16 files reviewed (will be continued)

include/umf/memory_provider_ops.h

…OINTER_TESTING_PROPS

lslusarczyk · 2025-10-01T08:37:14Z

@lukaszstolarczuk , @bratpiorka , @ldorau ,
CI passed, please check if I applied your comments correctly, resolve discussions which can be resolved and give approve if there is nothing left to do with respect to your comments

@lplewa , please do the same wrt your comments when you're back

test/test_valgrind.sh

test/common/level_zero_mocks.cpp

src/libumf.def

include/umf/memory_provider_ops.h

src/provider/provider_level_zero.c

lukaszstolarczuk · 2025-10-01T14:48:41Z

src/provider/provider_level_zero.c

+
+    uint32_t existing_peer_index = 0;
+    utils_write_lock(&ze_provider->resident_device_rwlock);
+    while (existing_peer_index < ze_provider->resident_device_count &&


you do the incrementation within the while loop and have extra && to mark the stop, I believe I agree with Łukasz - this would look better as for.

// you obviously don't have to init anything in for (e.g. for(; index < count; index++)

src/provider/provider_level_zero.c

bratpiorka · 2025-10-02T12:08:57Z

test/common/CMakeLists.txt

+    add_umf_library(
+        NAME umf_ze_loopback
+        TYPE SHARED
+        SRCS ze_loopback.h ze_loopback.cpp


.h is not needed?

bratpiorka · 2025-10-02T12:09:57Z

test/common/level_zero_mocks.cpp

+
+#include "umf/providers/provider_level_zero.h"
+#include "utils_load_library.h"
+#include <cstdlib>


#include <cstdlib> #include <umf/providers/provider_level_zero.h> #include "level_zero_mocks.h" #include "utils_load_library.h"

ok, applied

bratpiorka · 2025-10-02T12:13:49Z

test/common/level_zero_mocks.h

+#ifndef UMF_TEST_PROVIDER_LEVEL_ZERO_MOCKS_H
+#define UMF_TEST_PROVIDER_LEVEL_ZERO_MOCKS_H
+
+#include "utils_log.h"


#include <vector> #include <gmock/gmock.h> #include "utils_log.h" #include "ze_loopback.h"

ok, applied

bratpiorka · 2025-10-02T12:14:27Z

test/common/ze_loopback.cpp

+ *
+ */
+
+#include "ze_loopback.h"


#include <cstdlib> #include <iostream> #include "ze_loopback.h"

ok, applied

bratpiorka · 2025-10-02T12:16:20Z

test/pools/pool_residency.cpp

+// Under the Apache License v2.0 with LLVM Exceptions. See LICENSE.TXT.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+
+#include "../common/level_zero_mocks.h"


#include <umf/pools/pool_disjoint.h> #include <umf/providers/provider_level_zero.h> #include "gtest/gtest.h" #include "../common/level_zero_mocks.h" #include "pool.hpp"

ok, applied

lplewa

Please refactor commits structure.

commits in PR should be split to functional parts, with proper commit message.
Each commit should work (at least it should build - but we do not check it so exception happens). You should not split implementation, tests, documentation to separate commits.

If i were you i would create(This list is an inspiration for proper commits structure, you do not have split your PR to exactly this commits) : commit for adding iterate over critnib, commit for changes in asserts, maybe commit for this mock library, commit for allowing ovveride lib_name, commit for change in setResidentDevides, commit for main functionality of this PR....

include/umf/memory_pool_ops.h

lplewa · 2025-10-06T12:24:58Z

src/critnib/critnib.c

 void critnib_iter(critnib *c, uintptr_t min, uintptr_t max,
                  int (*func)(uintptr_t key, void *value, void *privdata),
                  void *privdata) {
+    bool wasIterating = false;


move lock inside if - then you can just add else to if instead extra variable. This will make it a lite bit easier to read.

no, because I want to access root in if (c->root) under the lock too

lplewa · 2025-10-06T12:46:26Z

src/provider/provider_level_zero.c

+
+    uint32_t existing_peer_index = 0;
+    utils_write_lock(&ze_provider->resident_device_rwlock);
+    while (existing_peer_index < ze_provider->resident_device_count &&


Yes i noticed that you need it outside of loop - this is why i write
for(index = 0; index < count; index++) if (device[index] == index) break;
not
for (int index = 0; index < count; index++) if (device[index] == index) break;

If something is for loop we should use for not while. If you need access to loop counter outside of function you can declare it before loop - this is a standard approach which should not confuse reader.

lplewa · 2025-10-06T12:57:25Z

src/provider/provider_level_zero.c

+        LOG_ERR("umfMemoryTrackerIterateAll did not manage to do some change "
+                "numFailed: %d, numSuccess: %d",
+                privData.success_changes, privData.failed_changes);
+        return UMF_RESULT_ERROR_MEMORY_PROVIDER_SPECIFIC;


When returining this error code get_last_native_error function should be able to return more precise error message - please add support for it.

ok, added store_last_native_error(result); in ze_memory_provider_resident_device_change_helper

lplewa · 2025-10-06T12:58:37Z

src/provider/provider_level_zero.c

+        .failed_changes = 0,
+    };
+
+    umf_result_t result = umfMemoryTrackerIterateAll(


I think we miss communicated - I will follow up on this later on.

lplewa · 2025-10-06T13:02:03Z

src/provider/provider_level_zero.c

+    return 0;
+}
+
+umf_result_t umfLevelZeroMemoryProviderResidentDeviceChange(


I would prefer to have add and remove function - i recall pbalcer already suggested it too.

I will split it to 3 functions - add, delete, and helper function to iterate over existing allocations (so we limit code duplication)

lslusarczyk · 2025-10-06T14:32:29Z

Please refactor commits structure.

commits in PR should be split to functional parts, with proper commit message. Each commit should work (at least it should build - but we do not check it so exception happens). You should not split implementation, tests, documentation to separate commits.

If i were you i would create(This list is an inspiration for proper commits structure, you do not have split your PR to exactly this commits) : commit for adding iterate over critnib, commit for changes in asserts, maybe commit for this mock library, commit for allowing ovveride lib_name, commit for change in setResidentDevides, commit for main functionality of this PR....

Refactor will be squashing all commits to one during merge.
Each commit will be working :)

lplewa · 2025-10-06T14:34:02Z

Please refactor commits structure.
commits in PR should be split to functional parts, with proper commit message. Each commit should work (at least it should build - but we do not check it so exception happens). You should not split implementation, tests, documentation to separate commits.
If i were you i would create(This list is an inspiration for proper commits structure, you do not have split your PR to exactly this commits) : commit for adding iterate over critnib, commit for changes in asserts, maybe commit for this mock library, commit for allowing ovveride lib_name, commit for change in setResidentDevides, commit for main functionality of this PR....

Refactor will be squashing all commits to one during merge. Each commit will be working :)

We do not do squash merge in this project

lslusarczyk requested a review from a team as a code owner September 11, 2025 12:30

bratpiorka requested changes Sep 11, 2025

View reviewed changes

bratpiorka requested review from lplewa and ldorau September 11, 2025 12:53