adapt testing notes, contd.

Signed-off-by: Matthias Büchse <[email protected]>

Author: mbuechse

Standards/scs-0101-w1-entropy-implementation-testing.md

@@ -63,11 +63,11 @@ as ensured by the image metadata standard.
 We implemented the following testcases that reflect the items in the above section
 on automated tests:
 
-- `scs_0101_image_property`,
-- `scs_0101_flavor_property`,
-- `scs_0101_entropy_avail`,
-- `scs_0101_rngd`,
-- `scs_0101_fips_test` (covers both the error and warning case).
+- `scs-0101-image-property`,
+- `scs-0101-flavor-property`,
+- `scs-0101-entropy-avail`,
+- `scs-0101-rngd`,
+- `scs-0101-fips-test` (covers both the error and warning case).
 
 These testcases can be checked using [`openstack_test.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py).
 

Standards/scs-0102-w1-image-metadata-implementation-testing.md

@@ -19,29 +19,29 @@ for these images.
 We implemented a host of testcases to reflect the requirements and recommendations of the standard. The following
 testcases ensure that fields have proper values:
 
-- `scs_0102_prop_architecture`,
-- `scs_0102_prop_hash_algo`,
-- `scs_0102_prop_min_disk`,
-- `scs_0102_prop_min_ram`,
-- `scs_0102_prop_os_version`,
-- `scs_0102_prop_os_distro`,
-- `scs_0102_prop_hw_disk_bus`,
-- `scs_0102_prop_hypervisor_type`,
-- `scs_0102_prop_hw_rng_model`,
-- `scs_0102_prop_image_build_date`,
-- `scs_0102_prop_image_original_user`,
-- `scs_0102_prop_image_source`,
-- `scs_0102_prop_image_description`,
-- `scs_0102_prop_replace_frequency`,
-- `scs_0102_prop_provided_until`,
-- `scs_0102_prop_uuid_validity`,
-- `scs_0102_prop_hotfix_hours`.
+- `scs-0102-prop-architecture`,
+- `scs-0102-prop-hash_algo`,
+- `scs-0102-prop-min_disk`,
+- `scs-0102-prop-min_ram`,
+- `scs-0102-prop-os_version`,
+- `scs-0102-prop-os_distro`,
+- `scs-0102-prop-hw_disk_bus`,
+- `scs-0102-prop-hypervisor_type`,
+- `scs-0102-prop-hw_rng_model`,
+- `scs-0102-prop-image_build_date`,
+- `scs-0102-prop-image_original_user`,
+- `scs-0102-prop-image_source`,
+- `scs-0102-prop-image_description`,
+- `scs-0102-prop-replace_frequency`,
+- `scs-0102-prop-provided_until`,
+- `scs-0102-prop-uuid_validity`,
+- `scs-0102-prop-hotfix_hours`.
 
 The property `patchlevel` is not tested because it is entirely optional.
 
 The following testcase ensures that each image is as recent as claimed by its `replace_frequency`:
 
-- `scs_0102_image_recency`
+- `scs-0102-image-recency`
 
 The script [`openstack_test.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py)
 can be used to check these testcases.

Standards/scs-0103-w1-standard-flavors-implementation.md

@@ -1,5 +1,5 @@
 ---
-title: "SCS Standard Flavors: Implementation Notes"
+title: "SCS Standard Flavors: Implementation and Testing Notes"
 type: Supplement
 track: IaaS
 status: Draft
@@ -17,7 +17,7 @@ create all standard, mandatory SCS flavors for you. It takes input that can be g
 
 We implemented a set of testcases corresponding 1:1 to the standard flavors:
 
-- `scs_0103_flavor_X` with varying `X`: ensures that flavor `SCS-X` is present and has the
+- `scs-0103-flavor-X` with varying `X`: ensures that flavor `SCS-X` is present and has the
   required `extra_specs`.
 
 _NOTE_: We still need to add testcases to ensure that the `extra_specs` of non-standard

Standards/scs-0104-w1-standard-images-implementation.md

@@ -1,5 +1,5 @@
 ---
-title: "SCS Standard Images: Implementation Notes"
+title: "SCS Standard Images: Implementation and Testing Notes"
 type: Supplement
 track: IaaS
 status: Draft
@@ -69,9 +69,9 @@ anticipate this future development.
 
 There are two classes of testcases:
 
-- `scs_0104_source_X` with varying `X`:
+- `scs-0104-source-X` with varying `X`:
   these ensure that certain images have the correct `image_source`;
-- `scs_0104_image_X` with varying `X`:
+- `scs-0104-image-X` with varying `X`:
   these ensure that certain images can be found in the list of public images.
 
 The testcases can be run using the script

Standards/scs-0114-v1-volume-type-standard.md

@@ -124,15 +124,3 @@ It should look like the following part:
 ## Related Documents
 
 - corresponding [decision record document](https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0111-v1-volume-type-decisions.md)
-
-## Conformance Tests
-
-The script `/Tests/iaas/volume-types/volume-types-check.py` connects to an OpenStack environment and tests
-the following:
-
-- for each volume type: if its description starts with `[scs:....]`, then this prefix is a feature list
-  (sorted, each entry at most once), and each entry is one of the possible features described here,
-- the recommended volume types are present (otherwise, a WARNING is produced).
-
-The return code is zero precisely when the test could be performed and the conditions are satisfied.
-Otherwise, detailed errors and warnings are output to stderr.

Standards/scs-0114-w1-volume-type-implementation.md

@@ -0,0 +1,20 @@
+---
+title: "SCS Volume Types: Testing Notes"
+type: Supplement
+track: IaaS
+status: Draft
+supplements:
+  - scs-0114-v1-volume-type-standard.md
+---
+
+## Automated tests
+
+We implemented the following testcases:
+
+- `scs-0114-syntax-check` ensures that, for every volume type description,
+  the list of aspects, if present, is formatted according to the standard.
+- `scs-0114-encrypted-type` ensures that a volume type featuring encryption is present.
+- `scs-0114-replicated-type` ensures that a volume type featuring replication is present.
+
+The testcases can be run using the script
+[`openstack_test.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py).

Standards/scs-0115-v1-default-rules-for-security-groups.md

@@ -134,8 +134,3 @@ $ openstack default security group rule list
 The spec for introducing configurability for the default Security Groups Rules can be found [here](https://specs.openstack.org/openstack/neutron-specs/specs/2023.2/configurable-default-sg-rules.html).
 
 More about Security Groups as a resource in OpenStack can be found [here](https://docs.openstack.org/nova/latest/user/security-groups.html).
-
-## Conformance Tests
-
-The conformance tests should check for the absence of any ingress traffic rules except traffic from the same Security Group in the `openstack default security group rule list`.
-As having egress rules is allowed by this standard, but not forced and can be set in various ways, the tests should check for presence of any egress rules.

Standards/scs-0115-w1-security-groups-implementation.md

@@ -0,0 +1,22 @@
+---
+title: "Default Rules for Security Groups: Implementation and Testing Notes"
+type: Supplement
+track: IaaS
+status: Draft
+supplements:
+  - scs-0115-v1-default-rules-for-security-groups.md
+---
+
+## Automated tests
+
+We implemented a single test case,
+
+- `scs-0115-default-rules`,
+
+which ensures
+
+1. the absence of any ingress traffic rules except traffic from the same Security Group in the `openstack default security group rule list`,
+2. the presence of any egress traffic rules.
+
+The testcase can be run using the script
+[`openstack_test.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py).

Standards/scs-0116-w1-key-manager-implementation-testing.md

@@ -41,13 +41,13 @@ This can be done with a small change in the policy.yaml file. The `creator` has
 
 ## Automated Tests
 
-The check for the presence of a Key Manager is done with a test script, that checks the presence of a Key Manager service in the catalog endpoint of Openstack.
-This check can eventually be moved to the checks for the mandatory an supported service/API list, in case of a promotion of the Key Manager to the mandatory list.
+We implemented the following testcases, in accordance with the standard:
 
-### Implementation
+- `scs-0116-presence` ensures that a service of type "key-manager" occurs in the service catalog;
+- `scs-0116-permissions` ensures that a regular user has suitable access to the key-manager API.
 
-The script [`check-for-key-manager.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/key-manager/check-for-key-manager.py)
-connects to OpenStack and performs the checks described in this section.
+The testcases can be run using the script
+[`openstack_test.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/openstack_test.py).
 
 ## Manual Tests
 

Standards/scs-0117-v1-volume-backup-service.md

@@ -84,15 +84,3 @@ The volume backup target storage SHOULD be a separate storage system from the on
 
 - [OpenStack Block Storage v3 Backup API reference](https://docs.openstack.org/api-ref/block-storage/v3/index.html#backups-backups)
 - [OpenStack Volume Backup Drivers](https://docs.openstack.org/cinder/latest/configuration/block-storage/backup-drivers.html)
-
-## Conformance Tests
-
-Conformance tests include using the `/v3/{project_id}/backups` Block Storage API endpoint to create a volume and a backup of it as a non-admin user and subsequently restore the backup on a new volume while verifying the success of each operation.
-These tests verify the mandatory part of the standard: providing the Volume Backup API.
-
-There is a test suite in [`volume-backup-tester.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/volume-backup/volume-backup-tester.py).
-The test suite connects to the OpenStack API and executes basic operations using the volume backup API to verify that the functionality requested by the standard is available.
-Please consult the associated [README.md](https://github.com/SovereignCloudStack/standards/blob/main/Tests/iaas/volume-backup/README.md) for detailed setup and testing instructions.
-
-Note that these tests don't verify the optional part of the standard: providing a separate storage backend for Cinder volume backups.
-This cannot be checked from outside of the infrastructure as it is an architectural property of the infrastructure itself and transparent to customers.