Record aborted testcases, part 1 (#975)

Harden scripts for all testcases that are currently being found in scs-compatible-iaas.yaml.
Future work includes to

- adapt `Tests/iaas/secure-connections` in the same manner and include testcases into yaml
- also harden scripts for testcases in scs-compatible-kaas.yaml

Signed-off-by: Matthias Büchse <[email protected]>
Co-authored-by: Marvin Frommhold <[email protected]>

Author: mbuechse

Standards/scs-0100-w1-flavor-naming-implementation-testing.md

@@ -167,9 +167,12 @@ None so far.
 
 ### Implementation
 
-The script [`flavor-names-openstack.py`](https://github.com/SovereignCloudStack/standards/tree/main/Tests/iaas/flavor-naming/flavor-names-openstack.py)
-talks to the OpenStack API of the cloud specified by the `OS_CLOUD` environment and queries properties and
-checks the names for standards compliance.
+We implemented two testcases, paralleling the two items in the "Errors" section above:
+
+- `scs-0100-syntax-check` ensures that any name starting with `SCS-` adheres to the standard;
+- `scs-0100-semantics-check` ensures that any such name is telling the truth as specified in the standard.
+
+These testcases can be checked using [`openstack_test.py`](https://github.com/SovereignCloudStack/standards/tree/main/Tests/iaas/openstack_test.py).
 
 ## Manual tests
 

Standards/scs-0101-w1-entropy-implementation-testing.md

@@ -60,8 +60,16 @@ as ensured by the image metadata standard.
 
 ### Implementation
 
-The script [`entropy-check.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/entropy/entropy-check.py)
-connects to OpenStack and performs the checks described in this section.
+We implemented the following testcases that reflect the items in the above section
+on automated tests:
+
+- `scs-0101-image-property`,
+- `scs-0101-flavor-property`,
+- `scs-0101-entropy-avail`,
+- `scs-0101-rngd`,
+- `scs-0101-fips-test` (covers both the error and warning case).
+
+These testcases can be checked using [`openstack_test.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py).
 
 ## Manual tests
 

Standards/scs-0102-v1-image-metadata.md

@@ -239,9 +239,3 @@ A boolean property that is not present is considered to be `false`.
   contact for issues with this image. Note that this field must only be set if the
   service provider does provide support for this image included in the image/flavor
   pricing (but it might be provided by a contracted 3rd party, e.g. the OS vendor).
-
-### Conformance Tests
-
-The script `image-md-check.py` retrieves the
-image list from a configured cloud and checks each image for the
-completeness and consistency of mandatory properties.

Standards/scs-0102-w1-image-metadata-implementation-testing.md

@@ -16,16 +16,35 @@ for these images.
 
 ## Automated tests
 
-### Images sample
-
-Some checks need to be performed on a live instance. All publicly available images on this instance
-will be checked for either only the mandatory properties or possibly also the recommended ones.
-Additionally, a user can also decide to test their private images, although this isn't a necessity.
-
-### Implementation
-
-The script [`image-md-check.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/image-metadata/image-md-check.py)
-connects to OpenStack and performs the checks described in this section.
+We implemented a host of testcases to reflect the requirements and recommendations of the standard. The following
+testcases ensure that fields have proper values:
+
+- `scs-0102-prop-architecture`,
+- `scs-0102-prop-hash_algo`,
+- `scs-0102-prop-min_disk`,
+- `scs-0102-prop-min_ram`,
+- `scs-0102-prop-os_version`,
+- `scs-0102-prop-os_distro`,
+- `scs-0102-prop-hw_disk_bus`,
+- `scs-0102-prop-hypervisor_type`,
+- `scs-0102-prop-hw_rng_model`,
+- `scs-0102-prop-image_build_date`,
+- `scs-0102-prop-image_original_user`,
+- `scs-0102-prop-image_source`,
+- `scs-0102-prop-image_description`,
+- `scs-0102-prop-replace_frequency`,
+- `scs-0102-prop-provided_until`,
+- `scs-0102-prop-uuid_validity`,
+- `scs-0102-prop-hotfix_hours`.
+
+The property `patchlevel` is not tested because it is entirely optional.
+
+The following testcase ensures that each image is as recent as claimed by its `replace_frequency`:
+
+- `scs-0102-image-recency`
+
+The script [`openstack_test.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py)
+can be used to check these testcases.
 
 ## Manual tests
 

Standards/scs-0103-v1-standard-flavors.md

@@ -156,27 +156,6 @@ with `block_device_mapping_v2`, e.g.
 to create a bootable 12G cinder volume from image `IMGUUID` that gets tied to the VM
 instance life cycle.)
 
-## Conformance Tests
-
-The script [`flavors-openstack.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/standard-flavors/flavors-openstack.py)
-will read the lists of mandatory and recommended flavors
-from a yaml file provided as command-line argument, connect to an OpenStack installation,
-and check whether the flavors are present and their extra_specs are correct.
-
-Missing flavors will be reported on various logging channels: error for mandatory, warning for
-recommended flavors. Incorrect extra_specs will be reported as error in any case.
-The return code will be non-zero if the test could not be performed or if any error was
-reported.
-
-The script does not check whether a name given via the extra_spec `scs:name-vN` is indeed valid according
-to any major version of the SCS standard on flavor naming.
-
-## Operational tooling
-
-The [openstack-flavor-manager](https://github.com/osism/openstack-flavor-manager) is able to
-create all standard, mandatory SCS flavors for you. It takes input that can be generated by
-`flavor-manager-input.py`.
-
 ## Previous standard versions
 
 The list of standard flavors used to be part of the flavor naming standard up until

Standards/scs-0103-w1-standard-flavors-implementation.md

@@ -0,0 +1,27 @@
+---
+title: "SCS Standard Flavors: Implementation and Testing Notes"
+type: Supplement
+track: IaaS
+status: Draft
+supplements:
+  - scs-0103-v1-standard-flavors.md
+---
+
+## Operational tooling
+
+The [openstack-flavor-manager](https://github.com/osism/openstack-flavor-manager) is able to
+create all standard, mandatory SCS flavors for you. It takes input that can be generated by
+[`flavor-manager-input.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/scs_0100_flavor_naming/flavor-manager-input.py).
+
+## Automated tests
+
+We implemented a set of testcases corresponding 1:1 to the standard flavors:
+
+- `scs-0103-flavor-X` with varying `X`: ensures that flavor `SCS-X` is present and has the
+  required `extra_specs`.
+
+_NOTE_: We still need to add testcases to ensure that the `extra_specs` of non-standard
+flavors are correct as well.
+
+The testcases can be run using the script
+[`openstack_test.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py).

Standards/scs-0104-v1-standard-images.md

@@ -128,18 +128,3 @@ next version. This standard makes no statement as to what is supposed to happen
 corresponding images in a live cloud environment. It is recommended to keep the
 once-mandatory images in the live environment. As for new environments, it is up to the
 operator whether to provide any or all of these images, as stated above.
-
-## Conformance Tests
-
-The script `images-openstack.py` will read the lists of mandatory and recommended images
-from a yaml file provided as command-line argument, connect to an OpenStack installation,
-and check whether the images are present. Missing images will be reported on various
-logging channels: error for mandatory, info for recommended images. Additionally, images
-whose `image_source` does not conform with the specifications will be reported on the
-error channel. The return code will be non-zero if the test could not be performed or
-if any errors have been reported.
-
-## Operational tooling
-
-The [openstack-image-manager](https://github.com/osism/openstack-image-manager) is able to
-create all standard, mandatory SCS images for you given image definitions from a YAML file.

Standards/scs-0104-w1-standard-images-implementation.md

@@ -1,5 +1,5 @@
 ---
-title: "SCS Standard Images: Implementation Notes"
+title: "SCS Standard Images: Implementation and Testing Notes"
 type: Supplement
 track: IaaS
 status: Draft
@@ -59,3 +59,20 @@ Run the test script on your environment and check the error messages :)
 The [openstack-image-manager](https://github.com/osism/openstack-image-manager) is able to
 create all standard, mandatory SCS images for you given image definitions from a YAML file.
 Please see [its documentation](https://docs.scs.community/docs/iaas/components/image-manager/) for details.
+
+## Automated tests
+
+We implemented testcases reflecting the information given in the YAML file(s). Be advised that
+the approach with the YAML file has considerable drawbacks, and it will be abandoned in
+future versions of the standard in favor of a more classical approach. The testcases already
+anticipate this future development.
+
+There are two classes of testcases:
+
+- `scs-0104-source-X` with varying `X`:
+  these ensure that certain images have the correct `image_source`;
+- `scs-0104-image-X` with varying `X`:
+  these ensure that certain images can be found in the list of public images.
+
+The testcases can be run using the script
+[`openstack_test.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py).

Standards/scs-0114-v1-volume-type-standard.md

@@ -124,15 +124,3 @@ It should look like the following part:
 ## Related Documents
 
 - corresponding [decision record document](https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0111-v1-volume-type-decisions.md)
-
-## Conformance Tests
-
-The script `/Tests/iaas/volume-types/volume-types-check.py` connects to an OpenStack environment and tests
-the following:
-
-- for each volume type: if its description starts with `[scs:....]`, then this prefix is a feature list
-  (sorted, each entry at most once), and each entry is one of the possible features described here,
-- the recommended volume types are present (otherwise, a WARNING is produced).
-
-The return code is zero precisely when the test could be performed and the conditions are satisfied.
-Otherwise, detailed errors and warnings are output to stderr.

Standards/scs-0114-w1-volume-type-implementation.md

@@ -0,0 +1,20 @@
+---
+title: "SCS Volume Types: Testing Notes"
+type: Supplement
+track: IaaS
+status: Draft
+supplements:
+  - scs-0114-v1-volume-type-standard.md
+---
+
+## Automated tests
+
+We implemented the following testcases:
+
+- `scs-0114-syntax-check` ensures that, for every volume type description,
+  the list of aspects, if present, is formatted according to the standard.
+- `scs-0114-encrypted-type` ensures that a volume type featuring encryption is present.
+- `scs-0114-replicated-type` ensures that a volume type featuring replication is present.
+
+The testcases can be run using the script
+[`openstack_test.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py).