adapt testing notes, contd.

Signed-off-by: Matthias Büchse <[email protected]>

Author: mbuechse

Standards/scs-0103-v1-standard-flavors.md

@@ -156,27 +156,6 @@ with `block_device_mapping_v2`, e.g.
 to create a bootable 12G cinder volume from image `IMGUUID` that gets tied to the VM
 instance life cycle.)
 
-## Conformance Tests
-
-The script [`flavors-openstack.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/standard-flavors/flavors-openstack.py)
-will read the lists of mandatory and recommended flavors
-from a yaml file provided as command-line argument, connect to an OpenStack installation,
-and check whether the flavors are present and their extra_specs are correct.
-
-Missing flavors will be reported on various logging channels: error for mandatory, warning for
-recommended flavors. Incorrect extra_specs will be reported as error in any case.
-The return code will be non-zero if the test could not be performed or if any error was
-reported.
-
-The script does not check whether a name given via the extra_spec `scs:name-vN` is indeed valid according
-to any major version of the SCS standard on flavor naming.
-
-## Operational tooling
-
-The [openstack-flavor-manager](https://github.com/osism/openstack-flavor-manager) is able to
-create all standard, mandatory SCS flavors for you. It takes input that can be generated by
-`flavor-manager-input.py`.
-
 ## Previous standard versions
 
 The list of standard flavors used to be part of the flavor naming standard up until

Standards/scs-0103-w1-standard-flavors-implementation.md

@@ -0,0 +1,27 @@
+---
+title: "SCS Standard Flavors: Implementation Notes"
+type: Supplement
+track: IaaS
+status: Draft
+supplements:
+  - scs-0103-v1-standard-flavors.md
+---
+
+## Operational tooling
+
+The [openstack-flavor-manager](https://github.com/osism/openstack-flavor-manager) is able to
+create all standard, mandatory SCS flavors for you. It takes input that can be generated by
+[`flavor-manager-input.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/scs_0100_flavor_naming/flavor-manager-input.py).
+
+## Automated tests
+
+We implemented a set of testcases corresponding 1:1 to the standard flavors:
+
+- `scs_0103_flavor_X` with varying `X`: ensures that flavor `SCS-X` is present and has the
+  required `extra_specs`.
+
+_NOTE_: We still need to add testcases to ensure that the `extra_specs` of non-standard
+flavors are correct as well.
+
+The testcases can be run using the script
+[`openstack_test.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py).

Standards/scs-0104-v1-standard-images.md

@@ -128,18 +128,3 @@ next version. This standard makes no statement as to what is supposed to happen
 corresponding images in a live cloud environment. It is recommended to keep the
 once-mandatory images in the live environment. As for new environments, it is up to the
 operator whether to provide any or all of these images, as stated above.
-
-## Conformance Tests
-
-The script `images-openstack.py` will read the lists of mandatory and recommended images
-from a yaml file provided as command-line argument, connect to an OpenStack installation,
-and check whether the images are present. Missing images will be reported on various
-logging channels: error for mandatory, info for recommended images. Additionally, images
-whose `image_source` does not conform with the specifications will be reported on the
-error channel. The return code will be non-zero if the test could not be performed or
-if any errors have been reported.
-
-## Operational tooling
-
-The [openstack-image-manager](https://github.com/osism/openstack-image-manager) is able to
-create all standard, mandatory SCS images for you given image definitions from a YAML file.

Standards/scs-0104-w1-standard-images-implementation.md

@@ -59,3 +59,20 @@ Run the test script on your environment and check the error messages :)
 The [openstack-image-manager](https://github.com/osism/openstack-image-manager) is able to
 create all standard, mandatory SCS images for you given image definitions from a YAML file.
 Please see [its documentation](https://docs.scs.community/docs/iaas/components/image-manager/) for details.
+
+## Automated tests
+
+We implemented testcases reflecting the information given in the YAML file(s). Be advised that
+the approach with the YAML file has considerable drawbacks, and it will be abandoned in
+future versions of the standard in favor of a more classical approach. The testcases already
+anticipate this future development.
+
+There are two classes of testcases:
+
+- `scs_0104_source_X` with varying `X`:
+  these ensure that certain images have the correct `image_source`;
+- `scs_0104_image_X` with varying `X`:
+  these ensure that certain images can be found in the list of public images.
+
+The testcases can be run using the script
+[`openstack_test.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py).