yet another intermediate state, still not good

Signed-off-by: Matthias Büchse <[email protected]>

Author: mbuechse

Tests/scs-compatible-iaas.yaml

@@ -4,144 +4,245 @@
 # - 2025-05-22 pruned old content; affected versions: v1, v2, v3-orig, v5
 
 
+# #### (TESTCASE) CATEGORIES ####
+# each category loosely corresponds to a standard
+# you can do one category for each version of the standard, but these can't share testcases!
+# so this works well for OpenStack Powered Compute
+# apart from that, put all versions into one category
+
+- id: category/opc-v2022.11
+  name: OpenStack Powered Compute v2022.11
+  url: https://opendev.org/openinfra/interop/src/branch/master/guidelines/2022.11.json
+- id: category/scs-0100
+  name: Flavor naming
+  url: https://docs.scs.community/standards/iaas/scs-0100
+- id: category/scs-0101
+  name: Entropy
+  url: https://docs.scs.community/standards/iaas/scs-0101
+- id: category/scs-0102
+  name: Image metadata
+  url: https://docs.scs.community/standards/iaas/scs-0102
+- id: category/scs-0103
+  name: Standard flavors
+  url: https://docs.scs.community/standards/iaas/scs-0103
+- id: category/scs-0104
+  name: Standard images
+  url: https://docs.scs.community/standards/iaas/scs-0104
+  # parameters:
+  #   image_spec: address (URL) of an image-spec (YAML) file
+- id: category/scs-0114
+  name: Volume Types
+  url: https://docs.scs.community/standards/iaas/scs-0114
+- id: category/scs-0115
+  name: Default rules for security groups
+  url: https://docs.scs.community/standards/iaas/scs-0115
+- id: category/scs-0116
+  name: Key manager
+  url: https://docs.scs.community/standards/iaas/scs-0116
+- id: category/scs-0117
+  name: Volume backup
+  url: https://docs.scs.community/standards/iaas/scs-0117
+- id: category/scs-0121
+  name: Availability Zones
+  url: https://docs.scs.community/standards/iaas/scs-0121
+- id: category/scs-0123
+  name: Mandatory and Supported IaaS Services
+  url: https://docs.scs.community/standards/iaas/scs-0123
+- id: category/scs-0302
+  name: Domain Manager Role
+  url: https://docs.scs.community/standards/iam/scs-0302
+
+
 # #### TESTCASES ####
+
 - id: testcase/flavor-name-syntax-check-v2-3
+  category: category/scs-0100
   tags: [scs-0100-v3.1]
   description: >
     Flavor names comply with syntax described in
     <https://docs.scs.community/standards/scs-0100-v3-flavor-naming> (identical with v2)
 - id: testcase/flavor-name-semantics-check
+  category: category/scs-0100
   tags: [scs-0100-v3.1]
   description: >
     Flavor names tell the truth according to
     <https://docs.scs.community/standards/scs-0100-v3-flavor-naming> (subsuming v1 and v2)
 - id: testcase/entropy-check-flavor-properties
+  category: category/scs-0101
   tags: [scs-0101-v1-rec]
   description: >
     Must have all flavor properties recommended in
     <https://docs.scs.community/standards/scs-0101-w1-entropy-implementation-testing#warnings>
 - id: testcase/entropy-check-image-properties
+  category: category/scs-0101
   tags: [scs-0101-v1-rec]
   description: >
     Must have all image properties recommended in
     <https://docs.scs.community/standards/scs-0101-w1-entropy-implementation-testing#warnings>
 - id: testcase/entropy-check-rngd
+  category: category/scs-0101
   tags: [scs-0101-v1]
   description: >
     Images of the test sample must have the service `rngd`; see
     <https://docs.scs.community/standards/scs-0101-w1-entropy-implementation-testing#errors>
 - id: testcase/entropy-check-entropy-avail
+  category: category/scs-0101
   tags: [scs-0101-v1]
   description: >
     A test instance must have the correct `entropy_avail`; see
     <https://docs.scs.community/standards/scs-0101-w1-entropy-implementation-testing#errors>
 - id: testcase/entropy-check-fips-test
+  category: category/scs-0101
   tags: [scs-0101-v1]
   description: >
     A test instance must pass the "FIPS test"; see
     <https://docs.scs.community/standards/scs-0101-w1-entropy-implementation-testing#errors>
 - id: testcase/image-metadata-check
+  category: category/scs-0102
   tags: [scs-0102-v1]
   description: >
     Must fulfill all requirements of <https://docs.scs.community/standards/scs-0102-v1-image-metadata>
 # required flavors as per scs-0103-v1
 - id: testcase/check-flavor-scs-1v-4
+  category: category/scs-0103
   tags: [scs-0103-v1]
 - id: testcase/check-flavor-scs-2v-8
+  category: category/scs-0103
   tags: [scs-0103-v1]
 - id: testcase/check-flavor-scs-4v-16
+  category: category/scs-0103
   tags: [scs-0103-v1]
 - id: testcase/check-flavor-scs-8v-32
+  category: category/scs-0103
   tags: [scs-0103-v1]
 - id: testcase/check-flavor-scs-1v-2
+  category: category/scs-0103
   tags: [scs-0103-v1]
 - id: testcase/check-flavor-scs-2v-4
+  category: category/scs-0103
   tags: [scs-0103-v1]
 - id: testcase/check-flavor-scs-4v-8
+  category: category/scs-0103
   tags: [scs-0103-v1]
 - id: testcase/check-flavor-scs-8v-16
+  category: category/scs-0103
   tags: [scs-0103-v1]
 - id: testcase/check-flavor-scs-16v-32
+  category: category/scs-0103
   tags: [scs-0103-v1]
 - id: testcase/check-flavor-scs-1v-8
+  category: category/scs-0103
   tags: [scs-0103-v1]
 - id: testcase/check-flavor-scs-2v-16
+  category: category/scs-0103
   tags: [scs-0103-v1]
 - id: testcase/check-flavor-scs-4v-32
+  category: category/scs-0103
   tags: [scs-0103-v1]
 - id: testcase/check-flavor-scs-1l-1
+  category: category/scs-0103
   tags: [scs-0103-v1]
 - id: testcase/check-flavor-scs-2v-4-20s
+  category: category/scs-0103
   tags: [scs-0103-v1]
 - id: testcase/check-flavor-scs-4v-16-100s
+  category: category/scs-0103
   tags: [scs-0103-v1]
 # recommended flavors as per scs-0103-v1
 - id: testcase/check-flavor-scs-1v-4-10
+  category: category/scs-0103
   tags: [scs-0103-v1-rec]
 - id: testcase/check-flavor-scs-2v-8-20
+  category: category/scs-0103
   tags: [scs-0103-v1-rec]
 - id: testcase/check-flavor-scs-4v-16-50
+  category: category/scs-0103
   tags: [scs-0103-v1-rec]
 - id: testcase/check-flavor-scs-8v-32-100
+  category: category/scs-0103
   tags: [scs-0103-v1-rec]
 - id: testcase/check-flavor-scs-1v-2-5
+  category: category/scs-0103
   tags: [scs-0103-v1-rec]
 - id: testcase/check-flavor-scs-2v-4-10
+  category: category/scs-0103
   tags: [scs-0103-v1-rec]
 - id: testcase/check-flavor-scs-4v-8-20
+  category: category/scs-0103
   tags: [scs-0103-v1-rec]
 - id: testcase/check-flavor-scs-8v-16-50
+  category: category/scs-0103
   tags: [scs-0103-v1-rec]
 - id: testcase/check-flavor-scs-16v-32-100
+  category: category/scs-0103
   tags: [scs-0103-v1-rec]
 - id: testcase/check-flavor-scs-1v-8-20
+  category: category/scs-0103
   tags: [scs-0103-v1-rec]
 - id: testcase/check-flavor-scs-2v-16-50
+  category: category/scs-0103
   tags: [scs-0103-v1-rec]
 - id: testcase/check-flavor-scs-4v-32-100
+  category: category/scs-0103
   tags: [scs-0103-v1-rec]
 - id: testcase/check-flavor-scs-1l-1-5
+  category: category/scs-0103
   tags: [scs-0103-v1-rec]
 # # image sources
 # ## capi
 - id: testcase/check-image-source-capi-2
+  category: category/scs-0104
   tags: [scs-0104-v1-2]
 - id: testcase/check-image-source-capi-1
+  category: category/scs-0104
   tags: [scs-0104-v1-1]
 # ## Ubuntu
 - id: testcase/check-image-source-ubuntu-24.04
+  category: category/scs-0104
   tags: [scs-0104-v1-2]
 - id: testcase/check-image-source-ubuntu-22.04
+  category: category/scs-0104
   tags: [scs-0104-v1-1, scs-0104-v1-2]
 - id: testcase/check-image-source-ubuntu-20.04
+  category: category/scs-0104
   tags: [scs-0104-v1-1, scs-0104-v1-2]
 # ## Debian
 - id: testcase/check-image-source-debian-12
+  category: category/scs-0104
   tags: [scs-0104-v1-1, scs-0104-v1-2]
 - id: testcase/check-image-source-debian-11
+  category: category/scs-0104
   tags: [scs-0104-v1-1, scs-0104-v1-2]
 - id: testcase/check-image-source-debian-10
+  category: category/scs-0104
   tags: [scs-0104-v1-1, scs-0104-v1-2]
 # # image presence
 # ## capi
 - id: testcase/check-image-capi-2
+  category: category/scs-0104
   tags: [scs-0104-v1-2]
 - id: testcase/check-image-capi-1
+  category: category/scs-0104
   tags: [scs-0104-v1-1]
 # ## Ubuntu
 - id: testcase/check-image-ubuntu-24.04
+  category: category/scs-0104
   tags: [scs-0104-v1-2]
 - id: testcase/check-image-ubuntu-22.04
+  category: category/scs-0104
   tags: [scs-0104-v1-1]
 # ## Debian
 - id: testcase/check-image-debian-12
+  category: category/scs-0104
   tags: [scs-0104-v1-2]
 # # volume types
 - id: testcase/volume-type-encrypted
+  category: category/scs-0114
   tags: [scs-0114-v1-rec]
   description: >
     An encrypted volume type can be discovered.
 - id: testcase/volume-type-replicated
+  category: category/scs-0114
   tags: [scs-0114-v1-rec]
   description: >
     A replicated volume type can be discovered.
@@ -150,52 +251,87 @@
 # see the correspondence between the test script and the standards requirements and recommendations
 # (the latter probably simply aren't checked?)
 - id: testcase/security-groups-default-rules-check
+  category: category/scs-0115
   tags: [scs-0115-v1]
   description: >
     Must fulfill all requirements of <https://docs.scs.community/standards/scs-0115-v1-default-rules-for-security-groups>
 # # key manager
 - id: testcase/key-manager-presence-check
+  category: category/scs-0116
   tags: [scs-0116-v1-rec]
   description: >
     Key manager service is discoverable.
 - id: testcase/key-manager-permissions-check
+  category: category/scs-0116
   tags: [scs-0116-v1]
   description: >
     Key manager is usable with member role.
 # # volume backup
 - id: testcase/volume-backup-check
+  category: category/scs-0117
   tags: [scs-0117-v1]
   description: >
     Volume backup works.
 # # availability zones
 - id: testcase/availability-zones-check
+  category: category/scs-0121
   tags: [scs-0121-v1,manual]
   description: >
     Note: manual check! Must fulfill all requirements of <https://docs.scs.community/standards/scs-0121-v1-Availability-Zones-Standard>
 # # service APIs
 - id: testcase/service-apis-check
+  category: category/scs-0123
   tags: [scs-0123-v1]
   description: >
     All required services (besides block-storage) are discoverable; as listed in <https://docs.scs.community/standards/scs-0123-v1-mandatory-and-supported-IaaS-services>.
 - id: testcase/storage-apis-check
+  category: category/scs-0123
   tags: [scs-0123-v1]
   description: >
     The block-storage API is discoverable; as listed in <https://docs.scs.community/standards/scs-0123-v1-mandatory-and-supported-IaaS-services>.
 - id: testcase/swift-s3-check
+  category: category/scs-0123
   tags: [scs-0123-v1]
   description: >
     The object-storage API is compatible with S3; as described in <https://docs.scs.community/standards/scs-0123-v1-mandatory-and-supported-IaaS-services>.
   type: swift-s3-check
 - id: testcase/service-apis-docs-check
+  category: category/scs-0123
   tags: [scs-0123-v1,manual]
   description: >
     Note: manual check! Must fulfill documentation requirements of <https://docs.scs.community/standards/scs-0123-v1-mandatory-and-supported-IaaS-services>.
 - id: testcase/domain-manager-check
+  category: category/scs-0302
   tags: [scs-0302-v1,manual]
   description: >
     Note: manual check! Must fulfill all requirements of <https://docs.scs.community/standards/scs-0302-v1-domain-manager-role>
 
+
 # #### TARGETS ####
+# we want to enable a visualization like the following
+#                   v5.1          v6
+# category 1
+# - testcase 1.1    required      required
+# - testcase 1.2    recommended   required
+# - testcase 1.3    --            recommended
+# etc.
+#
+# these categories alone would be too rigid though: if we have many testcase, it may be desireable to
+# highlight certain parts in a way that is orthogonal to categories.
+#
+# therefore we use two concepts:
+# - categories for sorting the testcase just like above, and
+# - targets for certain aspects of interest (such as security) that don't cover all testcases
+#
+# the former would be defined with the testcases
+# the latter would be more or less like the targets we currently have
+#
+# and then we still need to specify for each version: which testcases are required/recommended
+# we can still use targets for that:
+# - any testcase in `main` is required (it could be hidden from the list of targets if categories are used)
+# - any testcase not in main, but in another target is `recommended`
+# - if there is a target `recommended` we could hide that from the official list of targets same as `main`
+
 - id: target/scs-compatible-iaas-main-v5.1
   selectors:
   - scs-0100-v3.1