[SYCL] Support negative filters for ONEAPI_DEVICE_SELECTOR (#7309)

This PR aims to add support for negative filters for the
ONEAPI_DEVICE_SELECTOR variable to provide the user with a more flexible
way of specifying which devices should and should not be available for
usage. For example, ONEAPI_DEVICE_SELECTOR='opencl:*;**!opencl:gpu**'
considers all opencl backend devices except for those that are of the
gpu type.

Author: lbushi25

sycl/doc/EnvironmentVariables.md

@@ -34,7 +34,11 @@ With no environment variables set to say otherwise, all platforms and devices pr
 The syntax of this environment variable follows this BNF grammar:
 ```
 ONEAPI_DEVICE_SELECTOR = <selector-string>
-<selector-string> ::= <term>[;<term>...]
+<selector-string> ::= { <accept-filters> | <discard-filters> | <accept-filters>;<discard-filters> }
+<accept-filters> ::= <accept-filter>[;<accept-filter>...]
+<discard-filters> ::= <discard-filter>[;<discard-filter>...] 
+<accept-filter> ::= <term>
+<discard-filter> ::= !<term>
 <term> ::= <backend>:<devices>
 <backend> ::= { * | level_zero | opencl | cuda | hip | esimd_emulator }  // case insensitive
 <devices> ::= <device>[,<device>...]
@@ -52,6 +56,13 @@ The device indices are zero-based and are unique only within a backend. Therefor
 
 Additionally, if a sub-device is chosen (via numeric index or wildcard), then an additional layer of partitioning can be specified. In other words, a sub-sub-device can be selected. Like sub-devices, this is done with a period ( `.` ) and a sub-sub-device specifier which is a wildcard symbol ( `*` ) or a numeric index.  Example `ONEAPI_DEVICE_SELECTOR=level_zero:0.*.*` would partition device 0 into sub-devices and then partition each of those into sub-sub-devices. The range of grandchild sub-sub-devices would be the final devices available to the app, neither device 0, nor its child partitions would be in that list. 
 
+Lastly, a filter in the grammar can be thought of as a term in conjuction with an action that is taken on all devices that are selected by the term. The action can be an accept action or a discard action. Based on the action, a filter can be an accept filter or a discard filter.
+The string `<term>` represents an accept filter and the string `!<term>` represents a discard filter. The underlying term is the same but they perform different actions on the matching devices list.
+For example, `!opencl:*` discards all devices of the opencl backend from the list of available devices. The discarding filters, if there are any, must all appear at the end of the selector string. 
+When one or more filters accept a device and one or more filters discard the device, the latter have priority and the device is ultimately not made available to the user. This allows the user to provide selector strings such as `*:gpu;!cuda:*` that accepts all gpu devices except those with a CUDA backend.
+Furthermore, if the value of this environment variable only has discarding filters, an accepting filter that matches all devices, but not sub-devices and sub-sub-devices, will be implicitly included in the 
+environment variable to allow the user to specify only the list of devices that must not be made available. Therefore, `!*:cpu` will accept all devices except those that are of the cpu type and `opencl:*;!*:cpu`
+will accept all devices of the opencl backend exept those that are of the opencl backend and of the cpu type. It is legal to have a rejection filter even if it specifies devices have already been omitted by previous filters in the selection string. Doing so has no effect; the rejected devices are still omitted.
 
 The following examples further illustrate the usage of this environment variable:
 
@@ -66,13 +77,14 @@ The following examples further illustrate the usage of this environment variable
 | `ONEAPI_DEVICE_SELECTOR=opencl:0.*` | All the sub-devices from the OpenCL device with index 0 are exposed as SYCL root devices. No other devices are available. |
 | `ONEAPI_DEVICE_SELECTOR=opencl:0.2` | The third sub-device (2 in zero-based counting) of the OpenCL device with index 0 will be the sole device available.  |
 | `ONEAPI_DEVICE_SELECTOR=level_zero:*,*.*` | Exposes Level Zero devices to the application in two different ways. Each device (aka "card") is exposed as a SYCL root device and each sub-device is also exposed as a SYCL root device.|
-
+| `ONEAPI_DEVICE_SELECTOR="opencl:*;!opencl:0"` | All OpenCL devices except for the device with index 0 are available. |
+| `ONEAPI_DEVICE_SELECTOR="!*:cpu"` | All devices except for CPU devices are available. |
 
 Notes:
 - The backend argument is always required. An error will be thrown if it is absent.
 - Additionally, the backend MUST be followed by colon ( `:` ) and at least one device specifier of some sort, else an error is thrown.
 - For sub-devices and sub-sub-devices, the parent device must support partitioning (`info::partition_property::partition_by_affinity_domain` and `info::partition_affinity_domain::next_partitionable`. See the SYCL 2020 specification for a precise definition.) For Intel GPUs, the sub-device and sub-sub-device syntax can be used to expose tiles or CCSs to the SYCL application as root devices.  The exact mapping between sub-device, sub-sub-device, tiles, and CCSs is specific to the hardware.
-- The semi-colon character ( `;` ) is treated specially by many shells, so you may need to enclose the string in quotes if the selection string contains this character. 
+- The semi-colon character ( `;` ) and the exclamation mark character ( `!`  ) are treated specially by many shells, so you may need to enclose the string in quotes if the selection string contains these characters. 
 
 
 

sycl/include/sycl/detail/device_filter.hpp

@@ -29,7 +29,8 @@ std::ostream &operator<<(std::ostream &os, std::optional<T> const &opt) {
 }
 
 // the ONEAPI_DEVICE_SELECTOR string gets broken down into these targets
-// will will match devices.
+// will will match devices. If the target is negative, such as !opencl:*
+// then matching devices will not be made available to the user.
 struct ods_target {
 public:
   std::optional<backend> Backend;
@@ -44,6 +45,8 @@ struct ods_target {
   bool HasSubSubDeviceWildCard = false; // two levels of sub-devices.
   std::optional<unsigned> SubSubDeviceNum;
 
+  bool IsNegativeTarget = false; // used to represent negative filters.
+
   ods_target(backend be) { Backend = be; };
   ods_target(){};
   friend std::ostream &operator<<(std::ostream &Out, const ods_target &Target);

sycl/source/detail/device_filter.cpp

@@ -175,6 +175,7 @@ Parse_ONEAPI_DEVICE_SELECTOR(const std::string &envStr) {
   }
 
   std::vector<std::string_view> Entries = tokenize(envStr, ";");
+  unsigned int negative_filters = 0;
   // Each entry: "level_zero:gpu" or "opencl:0.0,0.1" or "opencl:*" but NOT just
   // "opencl".
   for (const auto Entry : Entries) {
@@ -190,6 +191,21 @@ Parse_ONEAPI_DEVICE_SELECTOR(const std::string &envStr) {
       std::vector<std::string_view> Targets = tokenize(Pair[1], ",");
       for (auto TargetStr : Targets) {
         ods_target DeviceTarget(be);
+        if (Entry[0] == '!') { // negative filter
+          DeviceTarget.IsNegativeTarget = true;
+          ++negative_filters;
+        } else { // positive filter
+          // no need to set IsNegativeTarget=false because it is so by default.
+          // ensure that no negative filter has been seen because all
+          // negative filters must come after all positive filters
+          if (negative_filters > 0) {
+            std::stringstream ss;
+            ss << "All negative(discarding) filters must appear after all "
+                  "positive(accepting) filters!";
+            throw sycl::exception(sycl::make_error_code(errc::invalid),
+                                  ss.str());
+          }
+        }
         Parse_ODS_Device(DeviceTarget, TargetStr);
         Result.push_back(DeviceTarget);
       }
@@ -201,6 +217,20 @@ Parse_ONEAPI_DEVICE_SELECTOR(const std::string &envStr) {
     }
   }
 
+  // This if statement handles the special case when the filter list
+  // contains at least one negative filter but no positive filters.
+  // This means that no devices will be available at all and so its as if
+  // the filter list was empty because the negative filters do not have any
+  // any effect. Hoewever, it is desirable to be able to set the
+  // ONEAPI_DEVICE_SELECTOR=!*:gpu to consider all devices except gpu
+  // devices so that we must implicitly add an acceptall target to the
+  // list of targets to make this work. So the result will be as if
+  // the filter string had the *:* string in it.
+  if (!Result.empty() && negative_filters == Result.size()) {
+    ods_target acceptAll{backend::all};
+    acceptAll.DeviceType = info::device_type::all;
+    Result.push_back(acceptAll);
+  }
   return Result;
 }
 

sycl/source/detail/platform_impl.cpp

@@ -149,18 +149,48 @@ std::vector<platform> platform_impl::get_platforms() {
 // ONEAPI_DEVICE_SELECTOR This function matches devices in the order of backend,
 // device_type, and device_num. The device_filter and ods_target structs pun for
 // each other, as do device_filter_list and ods_target_list.
+// Since ONEAPI_DEVICE_SELECTOR admits negative filters, we use type traits
+// to distinguish the case where we are working with ONEAPI_DEVICE_SELECTOR
+// in the places where the functionality diverges between these two
+// environment variables.
 template <typename ListT, typename FilterT>
 static int filterDeviceFilter(std::vector<RT::PiDevice> &PiDevices,
                               RT::PiPlatform Platform, ListT *FilterList) {
 
+  constexpr bool is_ods_target = std::is_same_v<FilterT, ods_target>;
+  // There are some differences in implementation between SYCL_DEVICE_FILTER
+  // and ONEAPI_DEVICE_SELECTOR so we use if constexpr to select the
+  // appropriate execution path if we are dealing with the latter variable.
+
+  if constexpr (is_ods_target) {
+
+    // Since we are working with ods_target filters ,which can be negative,
+    // we sort the filters so that all the negative filters appear before
+    // all the positive filters.  This enables us to have the full list of
+    // blacklisted devices by the time we get to the positive filters
+    // so that if a positive filter matches a blacklisted device we do
+    // not add it to the list of available devices.
+    std::sort(FilterList->get().begin(), FilterList->get().end(),
+              [](const ods_target &filter1, const ods_target &filter2) {
+                if (filter2.IsNegativeTarget)
+                  return false;
+                return true;
+              });
+  }
+
+  // this map keeps track of devices discarded by negative filters, it is only
+  // used in the ONEAPI_DEVICE_SELECTOR implemenation. It cannot be placed
+  // in the if statement above because it will then be out of scope in the rest
+  // of the function
+  std::map<RT::PiDevice *, bool> Blacklist;
+
   std::vector<plugin> &Plugins = RT::initialize();
   auto It =
       std::find_if(Plugins.begin(), Plugins.end(), [Platform](plugin &Plugin) {
         return Plugin.containsPiPlatform(Platform);
       });
   if (It == Plugins.end())
     return -1;
-
   plugin &Plugin = *It;
   backend Backend = Plugin.getBackend();
   int InsertIDx = 0;
@@ -188,12 +218,37 @@ static int filterDeviceFilter(std::vector<RT::PiDevice> &PiDevices,
         if (FilterDevType == info::device_type::all) {
           // Last, match the device_num entry
           if (!Filter.DeviceNum || DeviceNum == Filter.DeviceNum.value()) {
-            PiDevices[InsertIDx++] = Device;
+            if constexpr (is_ods_target) {      // dealing with ODS filters
+              if (!Blacklist[&Device]) {        // ensure it is not blacklisted
+                if (!Filter.IsNegativeTarget) { // is filter positive?
+                  PiDevices[InsertIDx++] = Device;
+                } else {
+                  // Filter is negative and the device matches the filter so
+                  // blacklist the device.
+                  Blacklist[&Device] = true;
+                }
+              }
+            } else { // dealing with SYCL_DEVICE_FILTER
+              PiDevices[InsertIDx++] = Device;
+            }
             break;
           }
+
         } else if (FilterDevType == DeviceType) {
           if (!Filter.DeviceNum || DeviceNum == Filter.DeviceNum.value()) {
-            PiDevices[InsertIDx++] = Device;
+            if constexpr (is_ods_target) {
+              if (!Blacklist[&Device]) {
+                if (!Filter.IsNegativeTarget) {
+                  PiDevices[InsertIDx++] = Device;
+                } else {
+                  // Filter is negative and the device matches the filter so
+                  // blacklist the device.
+                  Blacklist[&Device] = true;
+                }
+              }
+            } else {
+              PiDevices[InsertIDx++] = Device;
+            }
             break;
           }
         }