[L0] optimize # of event status queries through batching

Currently, for out-of-order immediate commandlists, each enqueued
operation creates an event on an internal vector to indicate
when that operation is completed. When that vector reaches
1k (by default) entries, each event is individually queried
for status and, if it completed, is cleaned up and removed.
This is causing latency spikes for enqueue operations and
reduces overall performance when using short-lived operations.

This patch introduces an event batching mechanism, where instead
of querying each event individually, the adapter will group
events in batches of 512 entries (by default) that will be
queried collectively with the use of a barrier.

Most of the credit goes to Michal Mrozek <[email protected]>
for profiling the problem and proposing a solution.

This new mechanism is disabled by default. To use it, this patch
is adding three new env variables:

- UR_L0_IMMEDIATE_COMMANDLISTS_BATCH_EVENT_COMPLETIONS. Enables
  event completion batching. Defaults to off.
- UR_L0_IMMEDIATE_COMMANDLISTS_BATCH_MAX. Controls the maximum
  number of batches for grouping the events. If there are no batches
  available, the adapter will fallback to querying events
  individually. Defaults to 10.
- UR_L0_IMMEDIATE_COMMANDLISTS_EVENTS_PER_BATCH. The number of
  events in each batch. Once a batch reaches this threshold,
  the adapter appends a barrier to the command list,
  and then another batch is selected to group incoming events.
  Defaults to 512.

Early results show appreciable reduction in enqueue tail latency,
but more experiments are needed.

Author: pbalcer

source/adapters/level_zero/context.cpp

@@ -13,6 +13,7 @@
 #include <mutex>
 #include <string.h>
 
+#include "adapters/level_zero/queue.hpp"
 #include "context.hpp"
 #include "ur_level_zero.hpp"
 
@@ -596,29 +597,6 @@ ur_context_handle_t_::decrementUnreleasedEventsInPool(ur_event_handle_t Event) {
   return UR_RESULT_SUCCESS;
 }
 
-// Get value of the threshold for number of events in immediate command lists.
-// If number of events in the immediate command list exceeds this threshold then
-// cleanup process for those events is executed.
-static const size_t ImmCmdListsEventCleanupThreshold = [] {
-  const char *UrRet =
-      std::getenv("UR_L0_IMMEDIATE_COMMANDLISTS_EVENT_CLEANUP_THRESHOLD");
-  const char *PiRet = std::getenv(
-      "SYCL_PI_LEVEL_ZERO_IMMEDIATE_COMMANDLISTS_EVENT_CLEANUP_THRESHOLD");
-  const char *ImmCmdListsEventCleanupThresholdStr =
-      UrRet ? UrRet : (PiRet ? PiRet : nullptr);
-  static constexpr int Default = 1000;
-  if (!ImmCmdListsEventCleanupThresholdStr)
-    return Default;
-
-  int Threshold = std::atoi(ImmCmdListsEventCleanupThresholdStr);
-
-  // Basically disable threshold if negative value is provided.
-  if (Threshold < 0)
-    return INT_MAX;
-
-  return Threshold;
-}();
-
 // Get value of the threshold for number of active command lists allowed before
 // we start heuristically cleaning them up.
 static const size_t CmdListsCleanupThreshold = [] {
@@ -648,8 +626,8 @@ ur_result_t ur_context_handle_t_::getAvailableCommandList(
   // Immediate commandlists have been pre-allocated and are always available.
   if (Queue->UsingImmCmdLists) {
     CommandList = Queue->getQueueGroup(UseCopyEngine).getImmCmdList();
-    if (CommandList->second.EventList.size() >
-        ImmCmdListsEventCleanupThreshold) {
+    if (CommandList->second.EventList.size() >=
+        Queue->getImmdCmmdListsEventCleanupThreshold()) {
       std::vector<ur_event_handle_t> EventListToCleanup;
       Queue->resetCommandList(CommandList, false, EventListToCleanup);
       CleanupEventListFromResetCmdList(EventListToCleanup, true);
@@ -743,11 +721,13 @@ ur_result_t ur_context_handle_t_::getAvailableCommandList(
         ZE2UR_CALL(zeFenceCreate, (ZeCommandQueue, &ZeFenceDesc, &ZeFence));
         ZeStruct<ze_command_queue_desc_t> ZeQueueDesc;
         ZeQueueDesc.ordinal = QueueGroupOrdinal;
+
         CommandList =
             Queue->CommandListMap
                 .emplace(ZeCommandList,
-                         ur_command_list_info_t{ZeFence, true, false,
-                                                ZeCommandQueue, ZeQueueDesc})
+                         ur_command_list_info_t(ZeFence, true, false,
+                                                ZeCommandQueue, ZeQueueDesc,
+                                                Queue->useCompletionBatching()))
                 .first;
       }
       ZeCommandListCache.erase(ZeCommandListIt);

source/adapters/level_zero/event.cpp

@@ -11,6 +11,7 @@
 #include <algorithm>
 #include <climits>
 #include <mutex>
+#include <optional>
 #include <string.h>
 
 #include "command_buffer.hpp"
@@ -1129,6 +1130,7 @@ ur_result_t ur_event_handle_t_::reset() {
   RefCountExternal = 0;
   RefCount.reset();
   CommandList = std::nullopt;
+  completionBatch = std::nullopt;
 
   if (!isHostVisible())
     HostVisibleEvent = nullptr;

source/adapters/level_zero/event.hpp

@@ -222,6 +222,10 @@ struct ur_event_handle_t_ : _ur_object {
 
   // Get the host-visible event or create one and enqueue its signal.
   ur_result_t getOrCreateHostVisibleEvent(ze_event_handle_t &HostVisibleEvent);
+
+  // completion batch for this event. Only used for out-of-order immediate
+  // command lists.
+  std::optional<ur_completion_batch_it> completionBatch;
 };
 
 // Helper function to implement zeHostSynchronize.