Merge branch 'main' into 526-refine-cve-check-in-scs-0210-v2-test-script

Author: piobig2871

…-v1-security-of-iaas-service-software.md …-v1-security-of-iaas-service-software.mdStandards/scs-XXXX-v1-security-of-iaas-service-software.md renamed to Standards/scs-0124-v1-security-of-iaas-service-software.md Standards/scs-XXXX-v1-security-of-iaas-service-software.md renamed to Standards/scs-0124-v1-security-of-iaas-service-software.md

@@ -4,7 +4,7 @@ type: Supplement
 track: IaaS
 status: Draft
 supplements:
-  - scs-XXXX-v1-security-of-iaas-service-software.md
+  - scs-0124-v1-security-of-iaas-service-software.md
 ---
 
 ## Testing or Detecting security updates in software

…-w1-security-of-iaas-service-software.md …-w1-security-of-iaas-service-software.mdStandards/scs-XXXX-w1-security-of-iaas-service-software.md renamed to Standards/scs-0124-w1-security-of-iaas-service-software.md Standards/scs-XXXX-w1-security-of-iaas-service-software.md renamed to Standards/scs-0124-w1-security-of-iaas-service-software.md

@@ -84,4 +84,3 @@ If the standard is used by a provider, the following decisions are binding and v
 [k8s-ha]: https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/
 [k8s-large-clusters]: https://kubernetes.io/docs/setup/best-practices/cluster-large/
 [scs-0213-v1]: https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0213-v1-k8s-nodes-anti-affinity.md
-[k8s-labels-docs]: https://kubernetes.io/docs/reference/labels-annotations-taints/#topologykubernetesiozone

Standards/scs-0125-v1-secure-connections.md

@@ -1,7 +1,8 @@
 ---
 title: Kubernetes Node Distribution and Availability
 type: Standard
-status: Draft
+status: Stable
+stabilized_at: 2024-11-21
 replaces: scs-0214-v1-k8s-node-distribution.md
 track: KaaS
 ---
@@ -100,23 +101,6 @@ These labels MUST be kept up to date with the current state of the deployment.
   The field gets autopopulated most of the time by either the kubelet or external mechanisms
   like the cloud controller.
 
-- `topology.scs.community/host-id`
-
-  This is an SCS-specific label; it MUST contain the hostID of the physical machine running
-  the hypervisor (NOT: the hostID of a virtual machine). Here, the hostID is an arbitrary identifier,
-  which need not contain the actual hostname, but it should nonetheless be unique to the host.
-  This helps identify the distribution over underlying physical machines,
-  which would be masked if VM hostIDs were used.
-
-## Conformance Tests
-
-The script `k8s-node-distribution-check.py` checks the nodes available with a user-provided
-kubeconfig file. Based on the labels `topology.scs.community/host-id`,
-`topology.kubernetes.io/zone`, `topology.kubernetes.io/region` and `node-role.kubernetes.io/control-plane`,
-the script then determines whether the nodes are distributed according to this standard.
-If this isn't the case, the script produces an error.
-It also produces warnings and informational outputs, e.g., if labels don't seem to be set.
-
 ## Previous standard versions
 
 This is version 2 of the standard; it extends [version 1](scs-0214-v1-k8s-node-distribution.md) with the

Standards/scs-0214-v1-k8s-node-distribution.md

@@ -16,25 +16,15 @@ Worker nodes can also be distributed over "failure zones", but this isn't a requ
 Distribution must be shown through labelling, so that users can access these information.
 
 Node distribution metadata is provided through the usage of the labels
-`topology.kubernetes.io/region`, `topology.kubernetes.io/zone` and
-`topology.scs.community/host-id` respectively.
-
-At the moment, not all labels are set automatically by most K8s cluster utilities, which incurs
-additional setup and maintenance costs.
+`topology.kubernetes.io/region` and `topology.kubernetes.io/zone`.
 
 ## Automated tests
 
-### Notes
-
-The test for the [SCS K8s Node Distribution and Availability](https://github.com/SovereignCloudStack/standards/blob/main/Standards/scs-0214-v2-k8s-node-distribution.md)
-checks if control-plane nodes are distributed over different failure zones (distributed into
-physical machines, zones and regions) by observing their labels defined by the standard.
-
-### Implementation
+Currently, automated testing is not readily possible because we cannot access information about
+the underlying host of a node (as opposed to its region and zone). Therefore, the test will only output
+a tentative result.
 
-The script [`k8s_node_distribution_check.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/kaas/k8s-node-distribution/k8s_node_distribution_check.py)
-connects to an existing K8s cluster and checks if a distribution can be detected with the labels
-set for the nodes of this cluster.
+The current implementation can be found in the script [`k8s_node_distribution_check.py`](https://github.com/SovereignCloudStack/standards/blob/main/Tests/kaas/k8s-node-distribution/k8s_node_distribution_check.py).
 
 ## Manual tests
 

Standards/scs-0214-v2-k8s-node-distribution.md

@@ -1,7 +1,8 @@
 ---
 title: KaaS Networking Standard
 type: Standard
-status: Draft
+status: Stable
+stabilized_at: 2024-11-21
 track: KaaS
 ---
 

Standards/scs-0214-w1-k8s-node-distribution-implementation-testing.md

@@ -1,2 +1,3 @@
 htmlcov/
 .coverage
+.secret

Standards/scs-0219-v1-kaas-networking.md

@@ -0,0 +1,90 @@
+#!/usr/bin/env python3
+# vim: set ts=4 sw=4 et:
+#
+# add_subject.py
+#
+# (c) Matthias Büchse <[email protected]>
+# SPDX-License-Identifier: Apache-2.0
+import base64
+import getpass
+import os
+import os.path
+import re
+import shutil
+import signal
+import subprocess
+import sys
+
+try:
+    from passlib.context import CryptContext
+    import argon2  # noqa:F401
+except ImportError:
+    print('Missing passlib and/or argon2. Please do:\npip install passlib argon2_cffi', file=sys.stderr)
+    sys.exit(1)
+
+# see ../compliance-monitor/monitor.py
+CRYPTCTX = CryptContext(schemes=('argon2', 'bcrypt'), deprecated='auto')
+SSH_KEYGEN = shutil.which('ssh-keygen')
+SUBJECT_RE = re.compile(r"[a-zA-Z0-9_\-]+")
+
+
+def main(argv, cwd):
+    if len(argv) != 1:
+        raise RuntimeError("Need to supply precisely one argument: name of subject")
+    subject = argv[0]
+    print(f"Attempt to add subject {subject!r}")
+    keyfile_path = os.path.join(cwd, '.secret', 'keyfile')
+    tokenfile_path = os.path.join(cwd, '.secret', 'tokenfile')
+    if os.path.exists(keyfile_path):
+        raise RuntimeError(f"Keyfile {keyfile_path} already present. Please proceed manually")
+    if os.path.exists(tokenfile_path):
+        raise RuntimeError(f"Tokenfile {tokenfile_path} already present. Please proceed manually")
+    if not SUBJECT_RE.fullmatch(subject):
+        raise RuntimeError(f"Subject name {subject!r} using disallowed characters")
+    sanitized_subject = subject.replace('-', '_')
+    print("Creating API key...")
+    while True:
+        password = getpass.getpass("Enter passphrase: ")
+        if password == getpass.getpass("Repeat passphrase: "):
+            break
+        print("No match. Try again...")
+    token = base64.b64encode(f"{subject}:{password}".encode('utf-8'))
+    hash_ = CRYPTCTX.hash(password)
+    with open(tokenfile_path, "wb") as fileobj:
+        os.fchmod(fileobj.fileno(), 0o600)
+        fileobj.write(token)
+    print("Creating key file using `ssh-keygen`...")
+    subprocess.check_call([SSH_KEYGEN, '-t', 'ed25519', '-C', sanitized_subject, '-f', keyfile_path, '-N', '', '-q'])
+    with open(keyfile_path + '.pub', "r") as fileobj:
+        pubkey_components = fileobj.readline().split()
+    print(f'''
+The following SECRET files have been created:
+
+  - {keyfile_path}
+  - {tokenfile_path}
+
+They are required for submitting test reports. You MUST keep them secure and safe.
+
+Insert the following snippet into compliance-monitor/bootstrap.yaml:
+
+  - subject: {subject}
+    api_keys:
+      - "{hash_}"
+    keys:
+      - public_key: "{pubkey_components[1]}"
+        public_key_type: "{pubkey_components[0]}"
+        public_key_name: "primary"
+
+Make sure to submit a pull request with the changed file. Otherwise, the reports cannot be submitted.
+''', end='')
+
+
+if __name__ == "__main__":
+    try:
+        sys.exit(main(sys.argv[1:], cwd=os.path.dirname(sys.argv[0]) or os.getcwd()) or 0)
+    except RuntimeError as e:
+        print(str(e), file=sys.stderr)
+        sys.exit(1)
+    except KeyboardInterrupt:
+        print("Interrupted", file=sys.stderr)
+        sys.exit(128 + signal.SIGINT)

Tests/.gitignore

@@ -1,3 +1,4 @@
+#!/usr/bin/env python3
 """Mandatory APIs checker
 This script retrieves the endpoint catalog from Keystone using the OpenStack
 SDK and checks whether all mandatory APi endpoints, are present.
@@ -26,54 +27,30 @@
 block_storage_service = ["volume", "volumev3", "block-storage"]
 
 
-def connect(cloud_name: str) -> openstack.connection.Connection:
-    """Create a connection to an OpenStack cloud
-    :param string cloud_name:
-        The name of the configuration to load from clouds.yaml.
-    :returns: openstack.connnection.Connection
-    """
-    return openstack.connect(
-        cloud=cloud_name,
-    )
-
-
-def check_presence_of_mandatory_services(cloud_name: str, s3_credentials=None):
-    try:
-        connection = connect(cloud_name)
-        services = connection.service_catalog
-    except Exception as e:
-        print(str(e))
-        raise Exception(
-            f"Connection to cloud '{cloud_name}' was not successfully. "
-            f"The Catalog endpoint could not be accessed. "
-            f"Please check your cloud connection and authorization."
-        )
+def check_presence_of_mandatory_services(conn: openstack.connection.Connection, s3_credentials=None):
+    services = conn.service_catalog
 
     if s3_credentials:
         mandatory_services.remove("object-store")
     for svc in services:
         svc_type = svc['type']
         if svc_type in mandatory_services:
             mandatory_services.remove(svc_type)
-            continue
-        if svc_type in block_storage_service:
+        elif svc_type in block_storage_service:
             block_storage_service.remove(svc_type)
 
     bs_service_not_present = 0
     if len(block_storage_service) == 3:
         # neither block-storage nor volume nor volumev3 is present
         # we must assume, that there is no volume service
-        logger.error("FAIL: No block-storage (volume) endpoint found.")
+        logger.error("No block-storage (volume) endpoint found.")
         mandatory_services.append(block_storage_service[0])
         bs_service_not_present = 1
-    if not mandatory_services:
-        # every mandatory service API had an endpoint
-        return 0 + bs_service_not_present
-    else:
-        # there were multiple mandatory APIs not found
-        logger.error(f"FAIL: The following endpoints are missing: "
-                     f"{mandatory_services}")
-        return len(mandatory_services) + bs_service_not_present
+    if mandatory_services:
+        # some mandatory APIs were not found
+        logger.error(f"The following endpoints are missing: "
+                     f"{', '.join(mandatory_services)}.")
+    return len(mandatory_services) + bs_service_not_present
 
 
 def list_containers(conn):
@@ -167,8 +144,8 @@ def s3_from_ostack(creds, conn, endpoint):
         # pass
 
 
-def check_for_s3_and_swift(cloud_name: str, s3_credentials=None):
-    # If we get credentials we assume, that there is no Swift and only test s3
+def check_for_s3_and_swift(conn: openstack.connection.Connection, s3_credentials=None):
+    # If we get credentials, we assume that there is no Swift and only test s3
     if s3_credentials:
         try:
             s3 = s3_conn(s3_credentials)
@@ -183,58 +160,46 @@ def check_for_s3_and_swift(cloud_name: str, s3_credentials=None):
         if s3_buckets == [TESTCONTNAME]:
             del_bucket(s3, TESTCONTNAME)
         # everything worked, and we don't need to test for Swift:
-        print("SUCCESS: S3 exists")
+        logger.info("SUCCESS: S3 exists")
         return 0
     # there were no credentials given, so we assume s3 is accessable via
     # the service catalog and Swift might exist too
-    try:
-        connection = connect(cloud_name)
-        connection.authorize()
-    except Exception as e:
-        print(str(e))
-        raise Exception(
-            f"Connection to cloud '{cloud_name}' was not successfully. "
-            f"The Catalog endpoint could not be accessed. "
-            f"Please check your cloud connection and authorization."
-        )
     s3_creds = {}
     try:
-        endpoint = connection.object_store.get_endpoint()
-    except Exception as e:
-        logger.error(
-            f"FAIL: No object store endpoint found in cloud "
-            f"'{cloud_name}'. No testing for the s3 service possible. "
-            f"Details: %s", e
+        endpoint = conn.object_store.get_endpoint()
+    except Exception:
+        logger.exception(
+            "No object store endpoint found. No testing for the s3 service possible."
         )
         return 1
     # Get S3 endpoint (swift) and ec2 creds from OpenStack (keystone)
-    s3_from_ostack(s3_creds, connection, endpoint)
+    s3_from_ostack(s3_creds, conn, endpoint)
     # Overrides (var names are from libs3, in case you wonder)
     s3_from_env(s3_creds, "HOST", "S3_HOSTNAME", "https://")
     s3_from_env(s3_creds, "AK", "S3_ACCESS_KEY_ID")
     s3_from_env(s3_creds, "SK", "S3_SECRET_ACCESS_KEY")
 
-    s3 = s3_conn(s3_creds, connection)
+    s3 = s3_conn(s3_creds, conn)
     s3_buckets = list_s3_buckets(s3)
     if not s3_buckets:
         s3_buckets = create_bucket(s3, TESTCONTNAME)
         assert s3_buckets
 
     # If we got till here, s3 is working, now swift
-    swift_containers = list_containers(connection)
+    swift_containers = list_containers(conn)
     # if not swift_containers:
-    #    swift_containers = create_container(connection, TESTCONTNAME)
+    #    swift_containers = create_container(conn, TESTCONTNAME)
     result = 0
     if Counter(s3_buckets) != Counter(swift_containers):
-        print("WARNING: S3 buckets and Swift Containers differ:\n"
-              f"S3: {sorted(s3_buckets)}\nSW: {sorted(swift_containers)}")
+        logger.warning("S3 buckets and Swift Containers differ:\n"
+                       f"S3: {sorted(s3_buckets)}\nSW: {sorted(swift_containers)}")
         result = 1
     else:
-        print("SUCCESS: S3 and Swift exist and agree")
+        logger.info("SUCCESS: S3 and Swift exist and agree")
     # Clean up
     # FIXME: Cleanup created EC2 credential
     # if swift_containers == [TESTCONTNAME]:
-    #    del_container(connection, TESTCONTNAME)
+    #    del_container(conn, TESTCONTNAME)
     # Cleanup created S3 bucket
     if s3_buckets == [TESTCONTNAME]:
         del_bucket(s3, TESTCONTNAME)
@@ -266,34 +231,47 @@ def main():
         help="Enable OpenStack SDK debug logging"
     )
     args = parser.parse_args()
+    logging.basicConfig(
+        format="%(levelname)s: %(message)s",
+        level=logging.DEBUG if args.debug else logging.INFO,
+    )
     openstack.enable_logging(debug=args.debug)
 
     # parse cloud name for lookup in clouds.yaml
-    cloud = os.environ.get("OS_CLOUD", None)
-    if args.os_cloud:
-        cloud = args.os_cloud
-    assert cloud, (
-        "You need to have the OS_CLOUD environment variable set to your cloud "
-        "name or pass it via --os-cloud"
-    )
+    cloud = args.os_cloud or os.environ.get("OS_CLOUD", None)
+    if not cloud:
+        raise RuntimeError(
+            "You need to have the OS_CLOUD environment variable set to your "
+            "cloud name or pass it via --os-cloud"
+        )
 
     s3_credentials = None
     if args.s3_endpoint:
         if (not args.s3_access) or (not args.s3_access_secret):
-            print("WARNING: test for external s3 needs access key and access secret.")
+            logger.warning("test for external s3 needs access key and access secret.")
         s3_credentials = {
             "AK": args.s3_access,
             "SK": args.s3_access_secret,
             "HOST": args.s3_endpoint
         }
     elif args.s3_access or args.s3_access_secret:
-        print("WARNING: access to s3 was given, but no endpoint provided.")
+        logger.warning("access to s3 was given, but no endpoint provided.")
 
-    result = check_presence_of_mandatory_services(cloud, s3_credentials)
-    result = result + check_for_s3_and_swift(cloud, s3_credentials)
+    with openstack.connect(cloud) as conn:
+        result = check_presence_of_mandatory_services(conn, s3_credentials)
+        result += check_for_s3_and_swift(conn, s3_credentials)
+
+    print('service-apis-check: ' + ('PASS', 'FAIL')[min(1, result)])
 
     return result
 
 
 if __name__ == "__main__":
-    main()
+    try:
+        sys.exit(main())
+    except SystemExit:
+        raise
+    except BaseException as exc:
+        logging.debug("traceback", exc_info=True)
+        logging.critical(str(exc))
+        sys.exit(1)