-
Notifications
You must be signed in to change notification settings - Fork 14.9k
[Offload] Add device UID #164391
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: main
Are you sure you want to change the base?
[Offload] Add device UID #164391
Conversation
@llvm/pr-subscribers-offload @llvm/pr-subscribers-backend-amdgpu Author: Robert Imschweiler (ro-i) ChangesIntroduced in OpenMP 6.0, the device UID shall be a unique identifier of a device on a given system. (Not necessarily a UUID.) Since it is not guaranteed that the (U)UIDs defined by the device vendor libraries, such as HSA, do not overlap with those of other vendors, the device UIDs in offload are always combined with the offload plugin name. In case the vendor library does not specify any device UID for a given device, we fall back to the offload-internal device ID. Note that the "fake" multiple host devices do not get different UIDs because they are effectively the same device. Full diff: https://github.com/llvm/llvm-project/pull/164391.diff 10 Files Affected:
diff --git a/offload/liboffload/API/Device.td b/offload/liboffload/API/Device.td
index 5b54c79d83f9d..e9c154818c4a1 100644
--- a/offload/liboffload/API/Device.td
+++ b/offload/liboffload/API/Device.td
@@ -29,6 +29,7 @@ def ol_device_info_t : Enum {
TaggedEtor<"PLATFORM", "ol_platform_handle_t", "the platform associated with the device">,
TaggedEtor<"NAME", "char[]", "Device name">,
TaggedEtor<"PRODUCT_NAME", "char[]", "Device user-facing marketing name">,
+ TaggedEtor<"UID", "char[]", "Device UID">,
TaggedEtor<"VENDOR", "char[]", "Device vendor">,
TaggedEtor<"DRIVER_VERSION", "char[]", "Driver version">,
TaggedEtor<"MAX_WORK_GROUP_SIZE", "uint32_t", "Maximum total work group size in work items">,
diff --git a/offload/liboffload/src/OffloadImpl.cpp b/offload/liboffload/src/OffloadImpl.cpp
index 6d22faeb0e57e..ed306470a0d76 100644
--- a/offload/liboffload/src/OffloadImpl.cpp
+++ b/offload/liboffload/src/OffloadImpl.cpp
@@ -147,8 +147,8 @@ llvm::Error ol_platform_impl_t::init() {
if (llvm::Error Err = Plugin->initDevice(Id))
return Err;
- auto Device = &Plugin->getDevice(Id);
- auto Info = Device->obtainInfoImpl();
+ GenericDeviceTy *Device = &Plugin->getDevice(Id);
+ llvm::Expected<InfoTreeNode> Info = Device->obtainInfo();
if (llvm::Error Err = Info.takeError())
return Err;
Devices.emplace_back(std::make_unique<ol_device_impl_t>(Id, Device, *this,
@@ -467,6 +467,7 @@ Error olGetDeviceInfoImplDetail(ol_device_handle_t Device,
switch (PropName) {
case OL_DEVICE_INFO_NAME:
case OL_DEVICE_INFO_PRODUCT_NAME:
+ case OL_DEVICE_INFO_UID:
case OL_DEVICE_INFO_VENDOR:
case OL_DEVICE_INFO_DRIVER_VERSION: {
// String values
@@ -544,6 +545,8 @@ Error olGetDeviceInfoImplDetailHost(ol_device_handle_t Device,
return Info.writeString("Virtual Host Device");
case OL_DEVICE_INFO_PRODUCT_NAME:
return Info.writeString("Virtual Host Device");
+ case OL_DEVICE_INFO_UID:
+ return Info.writeString(GenericDeviceTy::getHostDeviceUid());
case OL_DEVICE_INFO_VENDOR:
return Info.writeString("Liboffload");
case OL_DEVICE_INFO_DRIVER_VERSION:
diff --git a/offload/plugins-nextgen/amdgpu/dynamic_hsa/hsa_ext_amd.h b/offload/plugins-nextgen/amdgpu/dynamic_hsa/hsa_ext_amd.h
index 29cfe78082dbb..ddfa65c76cf2d 100644
--- a/offload/plugins-nextgen/amdgpu/dynamic_hsa/hsa_ext_amd.h
+++ b/offload/plugins-nextgen/amdgpu/dynamic_hsa/hsa_ext_amd.h
@@ -72,6 +72,7 @@ typedef enum hsa_amd_agent_info_s {
HSA_AMD_AGENT_INFO_MAX_WAVES_PER_CU = 0xA00A,
HSA_AMD_AGENT_INFO_NUM_SIMDS_PER_CU = 0xA00B,
HSA_AMD_AGENT_INFO_COOPERATIVE_QUEUES = 0xA010,
+ HSA_AMD_AGENT_INFO_UUID = 0xA011,
HSA_AMD_AGENT_INFO_TIMESTAMP_FREQUENCY = 0xA016,
} hsa_amd_agent_info_t;
diff --git a/offload/plugins-nextgen/amdgpu/src/rtl.cpp b/offload/plugins-nextgen/amdgpu/src/rtl.cpp
index a7723b8598815..bcde829923b0d 100644
--- a/offload/plugins-nextgen/amdgpu/src/rtl.cpp
+++ b/offload/plugins-nextgen/amdgpu/src/rtl.cpp
@@ -2069,6 +2069,20 @@ struct AMDGPUDeviceTy : public GenericDeviceTy, AMDGenericDeviceTy {
return Err;
ComputeUnitKind = GPUName;
+ // From the ROCm HSA documentation:
+ // Query the UUID of the agent. The value is an Ascii string with a maximum
+ // of 21 chars including NUL. The string value consists of two parts: header
+ // and body. The header identifies the device type (GPU, CPU, DSP) while the
+ // body encodes the UUID as a 16 digit hex string.
+ //
+ // Agents that do not support UUID will return the string "GPU-XX" or
+ // "CPU-XX" or "DSP-XX" depending on their device type.
+ char Uuid[24];
+ if (auto Err = getDeviceAttr(HSA_AMD_AGENT_INFO_UUID, Uuid))
+ return Err;
+ if (strcmp(Uuid + 3, "-XX") != 0)
+ DeviceUid = std::string(Plugin.getName()) + "-" + std::string(Uuid);
+
// Get the wavefront size.
uint32_t WavefrontSize = 0;
if (auto Err = getDeviceAttr(HSA_AGENT_INFO_WAVEFRONT_SIZE, WavefrontSize))
diff --git a/offload/plugins-nextgen/common/include/PluginInterface.h b/offload/plugins-nextgen/common/include/PluginInterface.h
index 8c530bba3882c..6fe90d6ace1a0 100644
--- a/offload/plugins-nextgen/common/include/PluginInterface.h
+++ b/offload/plugins-nextgen/common/include/PluginInterface.h
@@ -791,6 +791,10 @@ struct GenericDeviceTy : public DeviceAllocatorTy {
/// this id is not unique between different plugins; they may overlap.
int32_t getDeviceId() const { return DeviceId; }
+ /// Get the unique identifier of the device.
+ const char *getDeviceUid() const { return DeviceUid.c_str(); }
+ static constexpr const char *getHostDeviceUid() { return "HOST"; }
+
/// Set the context of the device if needed, before calling device-specific
/// functions. Plugins may implement this function as a no-op if not needed.
virtual Error setContext() = 0;
@@ -989,9 +993,12 @@ struct GenericDeviceTy : public DeviceAllocatorTy {
Error syncEvent(void *EventPtr);
virtual Error syncEventImpl(void *EventPtr) = 0;
+ /// Obtain information about the device.
+ Expected<InfoTreeNode> obtainInfo();
+ virtual Expected<InfoTreeNode> obtainInfoImpl() = 0;
+
/// Print information about the device.
Error printInfo();
- virtual Expected<InfoTreeNode> obtainInfoImpl() = 0;
/// Return true if the device has work that is either queued or currently
/// running
@@ -1196,6 +1203,12 @@ struct GenericDeviceTy : public DeviceAllocatorTy {
/// global device id and is not the device id visible to the OpenMP user.
const int32_t DeviceId;
+ /// The unique identifier of the device.
+ /// Per default, the unique identifier of the device is set to the device id,
+ /// combined with the plugin name, since the offload device id may overlap
+ /// between different plugins.
+ std::string DeviceUid;
+
/// The default grid values used for this device.
llvm::omp::GV GridValues;
diff --git a/offload/plugins-nextgen/common/src/PluginInterface.cpp b/offload/plugins-nextgen/common/src/PluginInterface.cpp
index db43cbe49cc2b..cad91668c5e0b 100644
--- a/offload/plugins-nextgen/common/src/PluginInterface.cpp
+++ b/offload/plugins-nextgen/common/src/PluginInterface.cpp
@@ -715,6 +715,9 @@ GenericDeviceTy::GenericDeviceTy(GenericPluginTy &Plugin, int32_t DeviceId,
DeviceId(DeviceId), GridValues(OMPGridValues),
PeerAccesses(NumDevices, PeerAccessState::PENDING), PeerAccessesLock(),
PinnedAllocs(*this), RPCServer(nullptr) {
+ DeviceUid = std::string(Plugin.getName()) + "-" +
+ std::to_string(static_cast<uint64_t>(DeviceId));
+
#ifdef OMPT_SUPPORT
OmptInitialized.store(false);
// Bind the callbacks to this device's member functions
@@ -1524,11 +1527,18 @@ Error GenericDeviceTy::enqueueHostCall(void (*Callback)(void *), void *UserData,
return Err;
}
+Expected<InfoTreeNode> GenericDeviceTy::obtainInfo() {
+ Expected<InfoTreeNode> Info = obtainInfoImpl();
+ if (Info)
+ Info->add("UID", DeviceUid, "", DeviceInfo::UID);
+ return Info;
+}
+
Error GenericDeviceTy::printInfo() {
- auto Info = obtainInfoImpl();
+ Expected<InfoTreeNode> Info = obtainInfo();
// Get the vendor-specific info entries describing the device properties.
- if (auto Err = Info.takeError())
+ if (Error Err = Info.takeError())
return Err;
// Print all info entries.
diff --git a/offload/plugins-nextgen/host/src/rtl.cpp b/offload/plugins-nextgen/host/src/rtl.cpp
index eb4ecac9907a1..f75c9cd29b2ad 100644
--- a/offload/plugins-nextgen/host/src/rtl.cpp
+++ b/offload/plugins-nextgen/host/src/rtl.cpp
@@ -148,7 +148,9 @@ struct GenELF64DeviceTy : public GenericDeviceTy {
/// Create the device with a specific id.
GenELF64DeviceTy(GenericPluginTy &Plugin, int32_t DeviceId,
int32_t NumDevices)
- : GenericDeviceTy(Plugin, DeviceId, NumDevices, GenELF64GridValues) {}
+ : GenericDeviceTy(Plugin, DeviceId, NumDevices, GenELF64GridValues) {
+ DeviceUid = getHostDeviceUid();
+ }
~GenELF64DeviceTy() {}
diff --git a/offload/tools/deviceinfo/llvm-offload-device-info.cpp b/offload/tools/deviceinfo/llvm-offload-device-info.cpp
index 9b58d67f017ca..42ffb97d6d77c 100644
--- a/offload/tools/deviceinfo/llvm-offload-device-info.cpp
+++ b/offload/tools/deviceinfo/llvm-offload-device-info.cpp
@@ -176,6 +176,7 @@ ol_result_t printDevice(std::ostream &S, ol_device_handle_t D) {
printDeviceValue<const char *>(S, D, OL_DEVICE_INFO_NAME, "Name"));
OFFLOAD_ERR(printDeviceValue<const char *>(S, D, OL_DEVICE_INFO_PRODUCT_NAME,
"Product Name"));
+ OFFLOAD_ERR(printDeviceValue<const char *>(S, D, OL_DEVICE_INFO_UID, "UID"));
OFFLOAD_ERR(
printDeviceValue<ol_device_type_t>(S, D, OL_DEVICE_INFO_TYPE, "Type"));
OFFLOAD_ERR(printDeviceValue<const char *>(
diff --git a/offload/unittests/OffloadAPI/device/olGetDeviceInfo.cpp b/offload/unittests/OffloadAPI/device/olGetDeviceInfo.cpp
index 8cb0b8065c33e..30eafee026316 100644
--- a/offload/unittests/OffloadAPI/device/olGetDeviceInfo.cpp
+++ b/offload/unittests/OffloadAPI/device/olGetDeviceInfo.cpp
@@ -98,6 +98,16 @@ TEST_P(olGetDeviceInfoTest, SuccessProductName) {
ASSERT_EQ(std::strlen(Name.data()), Size - 1);
}
+TEST_P(olGetDeviceInfoTest, SuccessUID) {
+ size_t Size = 0;
+ ASSERT_SUCCESS(olGetDeviceInfoSize(Device, OL_DEVICE_INFO_UID, &Size));
+ ASSERT_GT(Size, 0ul);
+ std::vector<char> UID;
+ UID.resize(Size);
+ ASSERT_SUCCESS(olGetDeviceInfo(Device, OL_DEVICE_INFO_UID, Size, UID.data()));
+ ASSERT_EQ(std::strlen(UID.data()), Size - 1);
+}
+
TEST_P(olGetDeviceInfoTest, HostProductName) {
size_t Size = 0;
ASSERT_SUCCESS(olGetDeviceInfoSize(Host, OL_DEVICE_INFO_PRODUCT_NAME, &Size));
@@ -109,6 +119,16 @@ TEST_P(olGetDeviceInfoTest, HostProductName) {
ASSERT_EQ(std::strlen(Name.data()), Size - 1);
}
+TEST_P(olGetDeviceInfoTest, HostUID) {
+ size_t Size = 0;
+ ASSERT_SUCCESS(olGetDeviceInfoSize(Host, OL_DEVICE_INFO_UID, &Size));
+ ASSERT_GT(Size, 0ul);
+ std::vector<char> UID;
+ UID.resize(Size);
+ ASSERT_SUCCESS(olGetDeviceInfo(Host, OL_DEVICE_INFO_UID, Size, UID.data()));
+ ASSERT_EQ(std::strlen(UID.data()), Size - 1);
+}
+
TEST_P(olGetDeviceInfoTest, SuccessVendor) {
size_t Size = 0;
ASSERT_SUCCESS(olGetDeviceInfoSize(Device, OL_DEVICE_INFO_VENDOR, &Size));
diff --git a/offload/unittests/OffloadAPI/device/olGetDeviceInfoSize.cpp b/offload/unittests/OffloadAPI/device/olGetDeviceInfoSize.cpp
index c4a3c2d5e3c75..79a18c1d133dc 100644
--- a/offload/unittests/OffloadAPI/device/olGetDeviceInfoSize.cpp
+++ b/offload/unittests/OffloadAPI/device/olGetDeviceInfoSize.cpp
@@ -32,6 +32,7 @@ OL_DEVICE_INFO_SIZE_TEST_EQ(Platform, ol_platform_handle_t,
OL_DEVICE_INFO_PLATFORM);
OL_DEVICE_INFO_SIZE_TEST_NONZERO(Name, OL_DEVICE_INFO_NAME);
OL_DEVICE_INFO_SIZE_TEST_NONZERO(ProductName, OL_DEVICE_INFO_PRODUCT_NAME);
+OL_DEVICE_INFO_SIZE_TEST_NONZERO(UID, OL_DEVICE_INFO_UID);
OL_DEVICE_INFO_SIZE_TEST_NONZERO(Vendor, OL_DEVICE_INFO_VENDOR);
OL_DEVICE_INFO_SIZE_TEST_NONZERO(DriverVersion, OL_DEVICE_INFO_DRIVER_VERSION);
OL_DEVICE_INFO_SIZE_TEST_EQ(MaxWorkGroupSize, uint32_t,
|
@@ -791,6 +791,10 @@ struct GenericDeviceTy : public DeviceAllocatorTy { | |||
/// this id is not unique between different plugins; they may overlap. | |||
int32_t getDeviceId() const { return DeviceId; } | |||
|
|||
/// Get the unique identifier of the device. | |||
const char *getDeviceUid() const { return DeviceUid.c_str(); } |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is DeviceUid stored as a field in GenericDeviceTy? liboffload already stores the info tree (which contains copies of all the strings) in the device struct.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, but the info tree needs to be populated, which I do in obtainInfo() for the UID. And doesn't it make more sense to set the UID on creation of the object and not later while calling some method?
DeviceId(DeviceId), GridValues(OMPGridValues), | ||
PeerAccesses(NumDevices, PeerAccessState::PENDING), PeerAccessesLock(), | ||
PinnedAllocs(*this), RPCServer(nullptr) { | ||
DeviceUid = std::string(Plugin.getName()) + "-" + |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This does not seem like a UUID, what does OpenMP say about this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That it's not a uuid as in "universal" uid (that's why I only call it "uid", too). Specifically, OpenMP 6.0, Section 9.1 says:
The uid trait specifies a unique identifier string of the device, for which the accepted values are implementation defined.
So it's only an identifier unique to a device on a given system.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Well we should probably use UUIDs anyway if possible.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I do. This is just the default implementation. It is overwritten in the amdgpu rtl, for example.
(And in the cuda rtl, once I add cuDeviceGetUuid() as you suggested below. Just needed to check for a test system.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
CUDA has cuDeviceGetUuid() we should use that https://docs.nvidia.com/cuda/cuda-driver-api/group__CUDA__DEVICE.html#group__CUDA__DEVICE_1g987b46b884c101ed5be414ab4d9e60e4.
(I'm currently fixing the UID in the cuda plugin since I now can test it on an appropriate system.) |
std::string DeviceUid; | ||
/// Construct the device UID from the vendor (U)UID. | ||
void setDeviceUidFromVendorUid(StringRef VendorUid); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do we need a global for this? Figured we can just recompute it whenever we call this function, not like it's on a fast path.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hm, but who would own the string pointed to by the const char *
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The UUID is supposed to be a constant property of the device. Why do we need to set it and we can just return a std::string.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I just added this set function so that it can be consistently called from the initImpl method of the plugins.
Ok, I can also return std::string and thus remove the attribute
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, I can also return std::string and thus remove the attribute
But then, I have to do the hsa/cuda call every time. That doesn't seem appropriate?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I just added this set function so that it can be consistently called from the initImpl method of the plugins.
(and it's protected)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do we need to call it in the initimpl, do we need to use this before the devices are loaded?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I mean, I get the UUID in the initImpl, because that's where all the other info-retrieving calls to the vendor libs are.
I could also store that UUID as an attribute and construct the DeviceUid in getDeviceUid(). Would you feel better about that version compared to directly computing DeviceUid and just returning it in getDeviceUid() (aka the current state)?
TaggedEtor<"PLATFORM", "ol_platform_handle_t", "the platform associated with the device">, | ||
TaggedEtor<"NAME", "char[]", "Device name">, | ||
TaggedEtor<"PRODUCT_NAME", "char[]", "Device user-facing marketing name">, | ||
TaggedEtor<"UID", "char[]", "Device UID">, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should probably call this UUID.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
but it's not guaranteed to be a UUID (and doesn't need to be).
I don't think that all vendors can agree on UUIDs for their stuff?
std::string DeviceUid; | ||
/// Construct the device UID from the vendor (U)UID. | ||
void setDeviceUidFromVendorUid(StringRef VendorUid); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The UUID is supposed to be a constant property of the device. Why do we need to set it and we can just return a std::string.
Introduced in OpenMP 6.0, the device UID shall be a unique identifier of a device on a given system. (Not necessarily a UUID.) Since it is not guaranteed that the (U)UIDs defined by the device vendor libraries, such as HSA, do not overlap with those of other vendors, the device UIDs in offload are always combined with the offload plugin name. In case the vendor library does not specify any device UID for a given device, we fall back to the offload-internal device ID. Note that the "fake" multiple host devices do *not* get different UIDs because they are effectively the same device. The device UID can be retrieved using the `llvm-offload-device-info` tool.
fb02cf7
to
fff258a
Compare
(needed to resolve a merge conflict with main) |
Introduced in OpenMP 6.0, the device UID shall be a unique identifier of a device on a given system. (Not necessarily a UUID.) Since it is not guaranteed that the (U)UIDs defined by the device vendor libraries, such as HSA, do not overlap with those of other vendors, the device UIDs in offload are always combined with the offload plugin name. In case the vendor library does not specify any device UID for a given device, we fall back to the offload-internal device ID. Note that the "fake" multiple host devices do not get different UIDs because they are effectively the same device.
The device UID can be retrieved using the
llvm-offload-device-info
tool.