twister: Platform key for test suites

Adds an option to inform twister a testsuite should only be built and
run for platforms with unique sets of attributes. This enables
for example keying on unique (arch, simulation) platforms to run the test
suite on.

The most common usage may be test suites configured to run once per
(arch, simulation) pair as being enough. Additional information about
platforms may enable running a test once per hardware IP block or once
per soc family or soc avoiding duplicated effort in building and running
tests when once suffices.

Signed-off-by: Tom Burdick <[email protected]>

Author: teburd

doc/develop/test/twister.rst

@@ -388,6 +388,25 @@ harness: <string>
     Usually pertains to external dependency domains but can be anything such as
     console, sensor, net, keyboard, Bluetooth or pytest.
 
+platform_key: <list of platform attributes>
+    Often a test needs to only be built and run once to qualify as passing.
+    Imagine a library of code that depends on the platform architecture where
+    passing the test on a single platform for each arch is enough to qualify the
+    tests and code as passing. The platform_key attribute enables doing just
+    that.
+
+    For example to key on (arch, simulation) to ensure a test is run once
+    per arch and simulation (as would be most common)::
+
+      platform_key:
+        - arch
+        - simulation
+
+    Adding platform (board) attributes to include things such as soc name,
+    soc family, and perhaps sets of IP blocks implementing each peripheral
+    interface would enable other interesting uses. For example, this could enable
+    building and running SPI tests once for eacn unique IP block.
+
 harness_config: <harness configuration options>
     Extra harness configuration options to be used to select a board and/or
     for handling generic Console with regex matching. Config can announce

scripts/pylib/twister/twisterlib/config_parser.py

@@ -33,6 +33,7 @@ class TwisterConfigParser:
                        "platform_type": {"type": "list", "default": []},
                        "platform_exclude": {"type": "set"},
                        "platform_allow": {"type": "set"},
+                       "platform_key": {"type": "list", "default": []},
                        "toolchain_exclude": {"type": "set"},
                        "toolchain_allow": {"type": "set"},
                        "filter": {"type": "str"},

scripts/pylib/twister/twisterlib/environment.py

@@ -361,6 +361,9 @@ def parse_arguments(args):
         help="Upon test failure, print relevant log data to stdout "
              "instead of just a path to it.")
 
+    parser.add_argument("--ignore-platform-key", action="store_true",
+                        help="Do not filter based on platform key")
+
     parser.add_argument(
         "-j", "--jobs", type=int,
         help="Number of jobs for building, defaults to number of CPU threads, "

scripts/pylib/twister/twisterlib/testplan.py

@@ -570,6 +570,7 @@ def apply_filters(self, **kwargs):
         runnable = (self.options.device_testing or self.options.filter == 'runnable')
         force_toolchain = self.options.force_toolchain
         force_platform = self.options.force_platform
+        ignore_platform_key = self.options.ignore_platform_key
         emu_filter = self.options.emulation_only
 
         logger.debug("platform filter: " + str(platform_filter))
@@ -616,6 +617,8 @@ def apply_filters(self, **kwargs):
 
         logger.info("Building initial testsuite list...")
 
+        keyed_tests = {}
+
         for ts_name, ts in self.testsuites.items():
 
             if ts.build_on_all and not platform_filter:
@@ -642,6 +645,7 @@ def apply_filters(self, **kwargs):
                     platform_scope = list(filter(lambda item: item.name in ts.platform_allow, \
                                              self.platforms))
 
+
             # list of instances per testsuite, aka configurations.
             instance_list = []
             for plat in platform_scope:
@@ -746,6 +750,31 @@ def apply_filters(self, **kwargs):
                 if plat.only_tags and not set(plat.only_tags) & ts.tags:
                     instance.add_filter("Excluded tags per platform (only_tags)", Filters.PLATFORM)
 
+                # platform_key is a list of unique platform attributes that form a unique key a test
+                # will match against to determine if it should be scheduled to run. A key containing a
+                # field name that the platform does not have will filter the platform.
+                #
+                # A simple example is keying on arch and simulation to run a test once per unique (arch, simulation) platform.
+                if not ignore_platform_key and hasattr(ts, 'platform_key') and len(ts.platform_key) > 0:
+                    # form a key by sorting the key fields first, then fetching the key fields from plat if they exist
+                    # if a field does not exist the test is still scheduled on that platform as its undeterminable.
+                    key_fields = sorted(set(ts.platform_key))
+                    key = [getattr(plat, key_field) for key_field in key_fields]
+                    has_all_fields = True
+                    for key_field in key_fields:
+                        if key_field is None:
+                            has_all_fields = False
+                    if has_all_fields:
+                        key.append(ts.name)
+                        key = tuple(key)
+                        keyed_test = keyed_tests.get(key)
+                        if keyed_test is not None:
+                            instance.add_filter(f"Excluded test already covered by platform_key {key} given fields {key_fields}", Filters.TESTSUITE)
+                        else:
+                            keyed_tests[key] = {'plat': plat, 'ts': ts}
+                    else:
+                        instance.add_filter(f"Excluded platform missing key fields demanded by test {key_fields}", Filters.PLATFORM)
+
                 test_configuration = ".".join([instance.platform.name,
                                                instance.testsuite.id])
                 # skip quarantined tests

scripts/schemas/twister/testsuite-schema.yaml

@@ -115,6 +115,12 @@ mapping:
         required: false
         sequence:
           - type: str
+      "platform_key":
+        required: false
+        type: seq
+        matching: "all"
+        sequence:
+          - type: str
       "tags":
         type: str
         required: false
@@ -272,6 +278,12 @@ mapping:
             required: false
             sequence:
               - type: str
+          "platform_key":
+            required: false
+            type: seq
+            matching: "all"
+            sequence:
+              - type: str
           "tags":
             type: str
             required: false