[OFFLOAD] Add plugin with support for Intel oneAPI Level Zero #158900
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
@llvm/pr-subscribers-offload Author: Alex Duran (adurang) ChangesAdd a new nextgen plugin that supports GPU devices through the Intel oneAPI Level Zero library. This PR also adds some new support to the PerThread auxiliary classes that is used by the plugin. Patch is 250.22 KiB, truncated to 20.00 KiB below, full version: https://github.com/llvm/llvm-project/pull/158900.diff 27 Files Affected:
diff --git a/offload/CMakeLists.txt b/offload/CMakeLists.txt
index b277380783500..8a704ab05eb53 100644
--- a/offload/CMakeLists.txt
+++ b/offload/CMakeLists.txt
@@ -150,9 +150,9 @@ if(DEFINED LIBOMPTARGET_BUILD_CUDA_PLUGIN OR
message(WARNING "Option removed, use 'LIBOMPTARGET_PLUGINS_TO_BUILD' instead")
endif()
-set(LIBOMPTARGET_ALL_PLUGIN_TARGETS amdgpu cuda host)
+set(LIBOMPTARGET_ALL_PLUGIN_TARGETS amdgpu cuda host level_zero)
set(LIBOMPTARGET_PLUGINS_TO_BUILD "all" CACHE STRING
- "Semicolon-separated list of plugins to use: cuda, amdgpu, host or \"all\".")
+ "Semicolon-separated list of plugins to use: cuda, amdgpu, level_zero, host or \"all\".")
if(LIBOMPTARGET_PLUGINS_TO_BUILD STREQUAL "all")
set(LIBOMPTARGET_PLUGINS_TO_BUILD ${LIBOMPTARGET_ALL_PLUGIN_TARGETS})
@@ -176,6 +176,19 @@ if(NOT (CMAKE_SYSTEM_PROCESSOR MATCHES "(x86_64)|(ppc64le)|(aarch64)$"
list(REMOVE_ITEM LIBOMPTARGET_PLUGINS_TO_BUILD "cuda")
endif()
endif()
+if(NOT (CMAKE_SYSTEM_PROCESSOR MATCHES "(x86_64)|(AMD64)$" AND
+ CMAKE_SYSTEM_NAME MATCHES "Linux|Windows"))
+ if("level_zero" IN_LIST LIBOMPTARGET_PLUGINS_TO_BUILD)
+ message(STATUS "Not building Level Zero plugin: it is only supported on "
+ "Linux/Windows x86_64, ppc64le, or aarch64 hosts")
+ list(REMOVE_ITEM LIBOMPTARGET_PLUGINS_TO_BUILD "level_zero")
+ endif()
+endif()
+if("level_zero" IN_LIST LIBOMPTARGET_PLUGINS_TO_BUILD AND
+ NOT LIBOMPTARGET_DEP_LEVEL_ZERO_FOUND)
+ message(STATUS "Not building Level Zero plugin: dependencies not found")
+ list(REMOVE_ITEM LIBOMPTARGET_PLUGINS_TO_BUILD "level_zero")
+endif()
message(STATUS "Building the offload library with support for "
"the \"${LIBOMPTARGET_PLUGINS_TO_BUILD}\" plugins")
diff --git a/offload/cmake/Modules/LibomptargetGetDependencies.cmake b/offload/cmake/Modules/LibomptargetGetDependencies.cmake
index 2a8bdebf2c1dd..0af0ae1ecdbec 100644
--- a/offload/cmake/Modules/LibomptargetGetDependencies.cmake
+++ b/offload/cmake/Modules/LibomptargetGetDependencies.cmake
@@ -89,4 +89,25 @@ if(LIBOMPTARGET_AMDGPU_ARCH)
endif()
endif()
+################################################################################
+# Looking for Level0
+################################################################################
+message(STATUS "Looking for Level0 includes.")
+find_path(LIBOMPTARGET_DEP_LEVEL_ZERO_INCLUDE_DIRS NAMES level_zero/ze_api.h)
+
+if(NOT LIBOMPTARGET_DEP_LEVEL_ZERO_INCLUDE_DIRS)
+ set(LIBOMPTARGET_DEP_LEVEL_ZERO_FOUND FALSE)
+ message(STATUS "Could NOT find Level Zero. Missing includes.")
+else()
+ message(STATUS "Level Zero include DIR: ${LIBOMPTARGET_DEP_LEVEL_ZERO_INCLUDE_DIRS}")
+ set(LIBOMPTARGET_DEP_LEVEL_ZERO_FOUND TRUE)
+ message(STATUS "Looking for Level Zero library.")
+ find_library(LIBOMPTARGET_DEP_LEVEL_ZERO_LIBRARIES NAMES ze_loader)
+ if(NOT LIBOMPTARGET_DEP_LEVEL_ZERO_LIBRARIES)
+ message(STATUS "Could NOT find Level Zero. Missing library.")
+ else()
+ message(STATUS "Level Zero library: ${LIBOMPTARGET_DEP_LEVEL_ZERO_LIBRARIES}")
+ endif()
+endif()
+
set(OPENMP_PTHREAD_LIB ${LLVM_PTHREAD_LIB})
diff --git a/offload/include/OpenMP/InteropAPI.h b/offload/include/OpenMP/InteropAPI.h
index 53ac4be2e2e98..2553bfa930784 100644
--- a/offload/include/OpenMP/InteropAPI.h
+++ b/offload/include/OpenMP/InteropAPI.h
@@ -160,17 +160,12 @@ struct InteropTableEntry {
Interops.push_back(obj);
}
- template <class ClearFuncTy> void clear(ClearFuncTy f) {
- for (auto &Obj : Interops) {
- f(Obj);
- }
- }
-
/// vector interface
int size() const { return Interops.size(); }
iterator begin() { return Interops.begin(); }
iterator end() { return Interops.end(); }
iterator erase(iterator it) { return Interops.erase(it); }
+ void clear() { Interops.clear(); }
};
struct InteropTblTy
diff --git a/offload/include/PerThreadTable.h b/offload/include/PerThreadTable.h
index 45b196171b4c8..0241370953c67 100644
--- a/offload/include/PerThreadTable.h
+++ b/offload/include/PerThreadTable.h
@@ -16,6 +16,60 @@
#include <list>
#include <memory>
#include <mutex>
+#include <type_traits>
+
+template <typename ObjectType> struct PerThread {
+ struct PerThreadData {
+ std::unique_ptr<ObjectType> ThEntry;
+ };
+
+ std::mutex Mtx;
+ std::list<std::shared_ptr<PerThreadData>> ThreadDataList;
+
+ // define default constructors, disable copy and move constructors
+ PerThread() = default;
+ PerThread(const PerThread &) = delete;
+ PerThread(PerThread &&) = delete;
+ PerThread &operator=(const PerThread &) = delete;
+ PerThread &operator=(PerThread &&) = delete;
+ ~PerThread() {
+ std::lock_guard<std::mutex> Lock(Mtx);
+ ThreadDataList.clear();
+ }
+
+private:
+ PerThreadData &getThreadData() {
+ static thread_local std::shared_ptr<PerThreadData> ThData = nullptr;
+ if (!ThData) {
+ ThData = std::make_shared<PerThreadData>();
+ std::lock_guard<std::mutex> Lock(Mtx);
+ ThreadDataList.push_back(ThData);
+ }
+ return *ThData;
+ }
+
+protected:
+ ObjectType &getThreadEntry() {
+ auto &ThData = getThreadData();
+ if (ThData.ThEntry)
+ return *ThData.ThEntry;
+ ThData.ThEntry = std::make_unique<ObjectType>();
+ return *ThData.ThEntry;
+ }
+
+public:
+ ObjectType &get() { return getThreadEntry(); }
+
+ template <class F> void clear(F f) {
+ std::lock_guard<std::mutex> Lock(Mtx);
+ for (auto ThData : ThreadDataList) {
+ if (!ThData->ThEntry)
+ continue;
+ f(*ThData->ThEntry);
+ }
+ ThreadDataList.clear();
+ }
+};
// Using an STL container (such as std::vector) indexed by thread ID has
// too many race conditions issues so we store each thread entry into a
@@ -23,10 +77,32 @@
// T is the container type used to store the objects, e.g., std::vector,
// std::set, etc. by each thread. O is the type of the stored objects e.g.,
// omp_interop_val_t *, ...
-
template <typename ContainerType, typename ObjectType> struct PerThreadTable {
using iterator = typename ContainerType::iterator;
+ template <typename, typename = std::void_t<>>
+ struct has_iterator : std::false_type {};
+ template <typename T>
+ struct has_iterator<T, std::void_t<typename T::iterator>> : std::true_type {};
+
+ template <typename T, typename = std::void_t<>>
+ struct has_clear : std::false_type {};
+ template <typename T>
+ struct has_clear<T, std::void_t<decltype(std::declval<T>().clear())>>
+ : std::true_type {};
+
+ template <typename T, typename = std::void_t<>>
+ struct has_clearAll : std::false_type {};
+ template <typename T>
+ struct has_clearAll<T, std::void_t<decltype(std::declval<T>().clearAll(1))>>
+ : std::true_type {};
+
+ template <typename, typename = std::void_t<>>
+ struct is_associative : std::false_type {};
+ template <typename T>
+ struct is_associative<T, std::void_t<typename T::mapped_type>>
+ : std::true_type {};
+
struct PerThreadData {
size_t NElements = 0;
std::unique_ptr<ContainerType> ThEntry;
@@ -71,6 +147,11 @@ template <typename ContainerType, typename ObjectType> struct PerThreadTable {
return ThData.NElements;
}
+ void setNElements(size_t Size) {
+ auto &NElements = getThreadNElements();
+ NElements = Size;
+ }
+
public:
void add(ObjectType obj) {
auto &Entry = getThreadEntry();
@@ -104,11 +185,81 @@ template <typename ContainerType, typename ObjectType> struct PerThreadTable {
for (auto ThData : ThreadDataList) {
if (!ThData->ThEntry || ThData->NElements == 0)
continue;
- ThData->ThEntry->clear(f);
+ if constexpr (has_clearAll<ContainerType>::value) {
+ ThData->ThEntry->clearAll(f);
+ } else if constexpr (has_iterator<ContainerType>::value &&
+ has_clear<ContainerType>::value) {
+ for (auto &Obj : *ThData->ThEntry) {
+ if constexpr (is_associative<ContainerType>::value) {
+ f(Obj.second);
+ } else {
+ f(Obj);
+ }
+ }
+ ThData->ThEntry->clear();
+ } else {
+ static_assert(true, "Container type not supported");
+ }
ThData->NElements = 0;
}
ThreadDataList.clear();
}
};
+template <typename T, typename = std::void_t<>> struct ContainerValueType {
+ using type = typename T::value_type;
+};
+template <typename T>
+struct ContainerValueType<T, std::void_t<typename T::mapped_type>> {
+ using type = typename T::mapped_type;
+};
+
+template <typename ContainerType, size_t reserveSize = 0>
+struct PerThreadContainer
+ : public PerThreadTable<ContainerType,
+ typename ContainerValueType<ContainerType>::type> {
+
+ // helpers
+ template <typename T, typename = std::void_t<>> struct indexType {
+ using type = typename T::size_type;
+ };
+ template <typename T> struct indexType<T, std::void_t<typename T::key_type>> {
+ using type = typename T::key_type;
+ };
+ template <typename T, typename = std::void_t<>>
+ struct has_resize : std::false_type {};
+ template <typename T>
+ struct has_resize<T, std::void_t<decltype(std::declval<T>().resize(1))>>
+ : std::true_type {};
+
+ template <typename T, typename = std::void_t<>>
+ struct has_reserve : std::false_type {};
+ template <typename T>
+ struct has_reserve<T, std::void_t<decltype(std::declval<T>().reserve(1))>>
+ : std::true_type {};
+
+ using IndexType = typename indexType<ContainerType>::type;
+ using ObjectType = typename ContainerValueType<ContainerType>::type;
+
+ // Get the object for the given index in the current thread
+ ObjectType &get(IndexType Index) {
+ auto &Entry = this->getThreadEntry();
+
+ // specialized code for vector-like containers
+ if constexpr (has_resize<ContainerType>::value) {
+ if (Index >= Entry.size()) {
+ if constexpr (has_reserve<ContainerType>::value && reserveSize > 0) {
+ if (Entry.capacity() < reserveSize)
+ Entry.reserve(reserveSize);
+ }
+ // If the index is out of bounds, try resize the container
+ Entry.resize(Index + 1);
+ }
+ }
+ ObjectType &Ret = Entry[Index];
+ this->setNElements(Entry.size());
+ return Ret;
+ }
+};
+
#endif
diff --git a/offload/plugins-nextgen/common/include/DLWrap.h b/offload/plugins-nextgen/common/include/DLWrap.h
index 8934e7e701021..95ce86e123cd3 100644
--- a/offload/plugins-nextgen/common/include/DLWrap.h
+++ b/offload/plugins-nextgen/common/include/DLWrap.h
@@ -282,5 +282,21 @@ template <size_t Requested, size_t Required> constexpr void verboseAssert() {
return dlwrap::SYM_USE##_Trait::get()(x0, x1, x2, x3, x4, x5, x6, x7, x8, \
x9, x10); \
}
+#define DLWRAP_INSTANTIATE_12(SYM_DEF, SYM_USE, T) \
+ T::ReturnType SYM_DEF(typename T::template arg<0>::type x0, \
+ typename T::template arg<1>::type x1, \
+ typename T::template arg<2>::type x2, \
+ typename T::template arg<3>::type x3, \
+ typename T::template arg<4>::type x4, \
+ typename T::template arg<5>::type x5, \
+ typename T::template arg<6>::type x6, \
+ typename T::template arg<7>::type x7, \
+ typename T::template arg<8>::type x8, \
+ typename T::template arg<9>::type x9, \
+ typename T::template arg<10>::type x10, \
+ typename T::template arg<11>::type x11) { \
+ return dlwrap::SYM_USE##_Trait::get()(x0, x1, x2, x3, x4, x5, x6, x7, x8, \
+ x9, x10, x11); \
+ }
#endif // OMPTARGET_SHARED_DLWRAP_H
diff --git a/offload/plugins-nextgen/level_zero/CMakeLists.txt b/offload/plugins-nextgen/level_zero/CMakeLists.txt
new file mode 100644
index 0000000000000..b9c8dd423c3ca
--- /dev/null
+++ b/offload/plugins-nextgen/level_zero/CMakeLists.txt
@@ -0,0 +1,69 @@
+if(NOT LIBOMPTARGET_DEP_LEVEL_ZERO_FOUND)
+return()
+endif()
+
+# Create the library and add the default arguments.
+add_target_library(omptarget.rtl.level_zero LEVEL_ZERO)
+
+set(LEVEL_ZERO_SRC_FILES
+ src/L0Context.cpp
+ src/L0Device.cpp
+ src/L0Kernel.cpp
+ src/L0Memory.cpp
+ src/L0Program.cpp
+ src/L0Plugin.cpp
+ src/L0Program.cpp
+ src/L0Options.cpp
+)
+list(APPEND LEVEL_ZERO_SRC_FILES
+ src/OmpWrapper.cpp
+)
+
+target_sources(omptarget.rtl.level_zero PRIVATE
+ ${LEVEL_ZERO_SRC_FILES}
+)
+
+target_include_directories(omptarget.rtl.level_zero PRIVATE
+ ${CMAKE_CURRENT_SOURCE_DIR}/include
+ ${CMAKE_CURRENT_SOURCE_DIR}/src
+)
+
+target_include_directories(omptarget.rtl.level_zero PRIVATE
+ ${LIBOMPTARGET_INCLUDE_DIR}
+ ${LIBOMPTARGET_DEP_LEVEL_ZERO_INCLUDE_DIRS}
+ ${LIBOMPTARGET_LLVM_INCLUDE_DIRS}
+ ${LIBOMPTARGET_OMP_HEADER_DIR}
+)
+
+if (EXISTS ${LIBOMPTARGET_DEP_LEVEL_ZERO_LIBRARIES} AND NOT "level_zero" IN_LIST LIBOMPTARGET_DLOPEN_PLUGINS)
+message(STATUS "Building Level Zero NG plugin linked against level_zero library")
+
+if(CMAKE_SYSTEM_NAME MATCHES "Linux")
+ target_link_libraries(omptarget.rtl.level_zero PRIVATE
+ ${LIBOMPTARGET_DEP_LEVEL_ZERO_LIBRARIES})
+elseif(CMAKE_SYSTEM_NAME MATCHES "Windows")
+ # Full path to the L0 library is recognized as a linker option, so we
+ # separate directory and file name
+ get_filename_component(LEVEL_ZERO_LIBRARY_PATH
+ ${LIBOMPTARGET_DEP_LEVEL_ZERO_LIBRARIES} DIRECTORY)
+ get_filename_component(LEVEL_ZERO_LIBRARY_NAME
+ ${LIBOMPTARGET_DEP_LEVEL_ZERO_LIBRARIES} NAME)
+ target_link_libraries(omptarget.rtl.level_zero PRIVATE
+ ${LEVEL_ZERO_LIBRARY_NAME} ${LIBOMP_LIB_FILE})
+ target_link_directories(omptarget.rtl.level_zero PRIVATE ${LEVEL_ZERO_LIBRARY_PATH})
+ target_link_options(omptarget.rtl.level_zero PRIVATE "LINKER:-def:${CMAKE_CURRENT_SOURCE_DIR}/src/rtl.def")
+ libomptarget_add_resource_file(omptarget.rtl.level_zero)
+else()
+ message(FATAL_ERROR "Missing platfrom support")
+endif()
+
+else()
+message(STATUS "Building Level Zero NG plugin for dlopened level_zero")
+get_filename_component(LEVEL_ZERO_LIBRARY_NAME ${LIBOMPTARGET_DEP_LEVEL_ZERO_LIBRARIES} NAME)
+if(CMAKE_SYSTEM_NAME MATCHES "Windows")
+ # Windows uses dll instead of lib files at runtime
+ string(REGEX REPLACE "lib$" "dll" LEVEL_ZERO_LIBRARY_NAME ${LEVEL_ZERO_LIBRARY_NAME})
+endif()
+target_compile_options(omptarget.rtl.level_zero PRIVATE "-DLEVEL_ZERO_LIBRARY=\"${LEVEL_ZERO_LIBRARY_NAME}\"")
+target_sources(omptarget.rtl.level_zero PRIVATE src/L0DynWrapper.cpp)
+endif()
diff --git a/offload/plugins-nextgen/level_zero/include/AsyncQueue.h b/offload/plugins-nextgen/level_zero/include/AsyncQueue.h
new file mode 100644
index 0000000000000..105f68205e402
--- /dev/null
+++ b/offload/plugins-nextgen/level_zero/include/AsyncQueue.h
@@ -0,0 +1,50 @@
+//===--- Level Zero Target RTL Implementation -----------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Async Queue wrapper for SPIR-V/Xe machine
+//
+//===----------------------------------------------------------------------===//
+
+#pragma once
+
+#include <vector>
+
+#include "L0Memory.h"
+
+namespace llvm {
+namespace omp {
+namespace target {
+namespace plugin {
+
+/// Abstract queue that supports asynchronous command submission
+struct AsyncQueueTy {
+ /// List of events attahced to submitted commands
+ std::vector<ze_event_handle_t> WaitEvents;
+ /// Pending staging buffer to host copies
+ std::list<std::tuple<void *, void *, size_t>> H2MList;
+ /// Pending USM memory copy commands that must wait for kernel completion
+ std::list<std::tuple<const void *, void *, size_t>> USM2MList;
+ /// Kernel event not signaled
+ ze_event_handle_t KernelEvent = nullptr;
+ /// Is this queue being used currently
+ bool InUse = false;
+ /// Clear data
+ void reset() {
+ WaitEvents.clear();
+ H2MList.clear();
+ USM2MList.clear();
+ KernelEvent = nullptr;
+ }
+};
+
+typedef ObjPool<AsyncQueueTy> AsyncQueuePoolTy;
+
+} // namespace plugin
+} // namespace target
+} // namespace omp
+} // namespace llvm
diff --git a/offload/plugins-nextgen/level_zero/include/L0Context.h b/offload/plugins-nextgen/level_zero/include/L0Context.h
new file mode 100644
index 0000000000000..b2b6def8101ca
--- /dev/null
+++ b/offload/plugins-nextgen/level_zero/include/L0Context.h
@@ -0,0 +1,138 @@
+//===--- Level Zero Target RTL Implementation -----------------------------===//
+//
+// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.
+// See https://llvm.org/LICENSE.txt for license information.
+// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception
+//
+//===----------------------------------------------------------------------===//
+//
+// Level Zero Context abstraction
+//
+//===----------------------------------------------------------------------===//
+
+#pragma once
+
+#include "L0Memory.h"
+#include "PerThreadTable.h"
+
+namespace llvm::omp::target::plugin {
+
+class LevelZeroPluginTy;
+
+class L0ContextTLSTy {
+ StagingBufferTy StagingBuffer;
+
+public:
+ auto &getStagingBuffer() { return StagingBuffer; }
+ const auto &getStagingBuffer() const { return StagingBuffer; }
+
+ void clear() { StagingBuffer.clear(); }
+};
+
+struct L0ContextTLSTableTy
+ : public PerThreadContainer<
+ std::unordered_map<ze_context_handle_t, L0ContextTLSTy>> {
+ void clear() {
+ PerThreadTable::clear([](L0ContextTLSTy &Entry) { Entry.clear(); });
+ }
+};
+
+/// Driver and context-specific resources. We assume a single context per
+/// driver.
+class L0ContextTy {
+ /// The plugin that created this context
+ LevelZeroPluginTy &Plugin;
+
+ /// Level Zero Driver handle
+ ze_driver_handle_t zeDriver = nullptr;
+
+ /// Common Level Zero context
+ ze_context_handle_t zeContext = nullptr;
+
+ /// API version supported by the Level Zero driver
+ ze_api_version_t APIVersion = ZE_API_VERSION_CURRENT;
+
+ /// Imported external pointers. Track this only for user-directed
+ /// imports/releases.
+ std::unordered_map<uintptr_t, size_t> ImportedPtrs;
+
+ /// Common event pool
+ EventPoolTy EventPool;
+
+ /// Host Memory allocator for this driver
+ MemAllocatorTy HostMemAllocator;
+
+public:
+ /// Named constants for checking the imported external pointer regions.
+ static constexpr int32_t ImportNotExist = -1;
+ static constexpr int32_t ImportUnknown = 0;
+ static constexpr int32_t ImportExist = 1;
+
+ /// Create context, initialize event pool and extension functions
+ L0ContextTy(LevelZeroPluginTy &Plugin, ze_driver_handle_t zeDriver,
+ int32_t DriverId);
+
+ L0ContextTy(const L0ContextTy &) = delete;
+ L0ContextTy(L0ContextTy &&) = delete;
+ L0ContextTy &operator=(const L0ContextTy &) = delete;
+ L0ContextTy &operator=(const L0ContextTy &&) = delete;
+
+ /// Release resources
+ ~L0ContextTy() {
+ EventPool.deinit();
+ HostMemAllocator.deinit();
+ if (zeContext)
+ CALL_ZE_RET_VOID(zeContextDestroy, zeContext);
+ }
+
+ auto &getPlugin() const { return Plugin; }
+
+ StagingBufferTy &getStagingBuffer();
+
+ /// Add imported external pointer region.
+ void addImported(void *Ptr, size_t Size) {
+ (void)ImportedPtrs.emplace((uintptr_t)Ptr, Size);
+ }
+
+ /// Remove imported external pointer region
+ void removeImported(void *Ptr) { (void)ImportedPtrs.erase((uintptr_t)Ptr); }
+
+ /// Check if imported regions contain the specified region.
+ int32_t checkImported(void *Ptr, size_t Size) const {
+ uintptr_t LB = (uintptr_t)Ptr;
+ uintptr_t UB = LB + Size;
+ // We do not expect a large number of user-directed imports, so use simple
+ // logic.
+ for (auto &I : ImportedPtrs) {
+ uintptr_t ILB = I.first;
+ uintptr_t IUB = ILB + I.seco...
[truncated]
|
|
✅ With the latest revision this PR passed the C/C++ code formatter. |
sarnex
reviewed
Sep 16, 2025
jhuber6
reviewed
Sep 16, 2025
Co-authored-by: Alexey Sachkov <[email protected]>
Co-authored-by: Alexey Sachkov <[email protected]>
Co-authored-by: Alexey Sachkov <[email protected]>
sarnex
reviewed
Sep 18, 2025
jhuber6
reviewed
Sep 18, 2025
| list(REMOVE_ITEM LIBOMPTARGET_PLUGINS_TO_BUILD "cuda") | ||
| endif() | ||
| endif() | ||
| if(NOT (CMAKE_SYSTEM_PROCESSOR MATCHES "(x86_64)|(AMD64)$" AND |
There was a problem hiding this comment.
Wouldn't it be better to do this check when we create the list rather than removing it later?
There was a problem hiding this comment.
All the other plugins are treated in the same way right now (e.g., 3 lines above). They are in the "candidate" list and if some condition is not met they're removed.
I don't disagree with your statement about do it earlier but we should be consistent, so maybe something to do subsequently?
RossBrunton
reviewed
Sep 19, 2025
RossBrunton
reviewed
Sep 19, 2025
kevinsala
reviewed
Sep 26, 2025
| // use its lock and be able to cache the group params and indirect flags | ||
| auto &KernelPR = const_cast<KernelPropertiesTy &>(getProperties()); | ||
| // Protect from kernel preparation to submission as kernels are shared. | ||
| std::unique_lock<std::mutex> KernelLock(KernelPR.Mtx); |
There was a problem hiding this comment.
This mutex section seems much more coarse-grained than I'd expect. I don't think the other plugins have such large exclusive section when launching a kernel.
|
By the way unless this passes all the existing tests it should be disabled by default. |
There's no testing in this PR. |
It probably doesn't enable it as a backend for the OpenMP tests, but I'd imagine it would show up in platform detection for the offloading ones. |
Sorry for my ignorance but do you mean the ones in "unittests"? |
I checked this, and they're not run. It would need to be explicitly add it in the CMakeLists.txt of unittests. |
|
Will this have any testing support and does Intel plan on hosting a bot to test it? Because I'm not aware of anyone who can test these changes locally. |
Eventually yes. But at the current failrate it would create more noise than anything I think. But it's in our own interest to enable testing so of course we want to get there when possible. |
|
Regarding testing:
You can still set up a buildbot in the staging area. It won't bother anybody, but we could start monitoring it and the progress. |
|
Are they any other comments, issues? |
pbalcer
reviewed
Oct 7, 2025
sarnex
reviewed
Oct 7, 2025
Co-authored-by: Nick Sarnie <[email protected]>
|
I've successfully launched a simple SYCL program with a kernel through liboffload using this L0 plugin. I had to do a few minor changes, but it mostly worked. The biggest problem is that the current Plugin Interface infrastructure assumes that program binaries are ELF images. This seems OpenMP-specific, and is not true for SPIR-V (nor PTX, but it was hacked around to produce CUBIN instead for the purpose of liboffload) produced by SYCL. Our existing runtime API simply treats the programs as opaque binary blobs, and passes them directly into the L0/cuda/hip driver. Given the existing liboffload API for launching a kernel, the metadata contained in the ELF images doesn't seem necessary to me. And if there's anything missing, we can just add the required information to the API calls themselves. To make liboffload more generic, I think it'd be best to move the ELF requirement up a layer so that OpenMP can continue working as is with elf images, and other higher-level languages (like SYCL) can use whatever they want. |
|
@jhuber6 @kevinsala is there anything else needed here? thanks |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add a new nextgen plugin that supports GPU devices through the Intel oneAPI Level Zero library.
This PR also adds some new support to the PerThread auxiliary classes that is used by the plugin.