documentation

Author: backstreetkiwi

README.md

@@ -9,10 +9,16 @@ This catalog contains the config data for the tests run in https://testing.stack
 * [platforms.yaml](catalog/platforms.yaml) contains the definition of the test system vendors and platforms
 * [operator-tests.yaml](catalog/operator-tests.yaml) defines the integration tests for our operators
 
+See this [Nuclino page](https://app.nuclino.com/Stackable/Engineering/Configuring-Test-Jobs-4e993d84-19b4-4081-846d-738f9f38573d) for further information.
+
 ## Apps
 
-Under [apps/](apps/), we maintain a bunch of Dockerized applications for the maintenance of Jenkins and to run the tests.
+Under [apps/](apps/README.md), we maintain a bunch of Dockerized applications for the maintenance of Jenkins and to run the tests.
 
 ## Seed Job
 
 The [seed job pipeline](jenkins/jobbuilder.groovy) to populate the Jenkins
+
+## Tools
+
+The [tools folder](tools/) contains Dockerized tools to be run in the maintenance jobs. They are not strictly related to testing, but found their home here because this is the only Jenkins we have.

apps/README.md

@@ -0,0 +1,26 @@
+# testing.stackable.tech apps
+
+For running our testing platform, we need to perform complex jobs in Jenkins. These jobs are coded in Dockerized apps. As they have common tasks, we maintain a shared codebase for these apps and build different Docker images for them.
+
+## The Apps
+
+We have these two apps:
+
+* **Jenkins Job Builder** is a "seed job" which (re)creates all the jobs in our Jenkins. 
+* **Operator Test Runner** is an app which creates/terminates K8s clusters and runs operator integration tests in them.
+
+## Project Structure
+
+The code is documented inline, here's the project structure:
+
+* [docker/](docker/) contains the Docker build resources (`Dockerfile` + ...) for the apps
+* [jjb/](jjb/) contains the templates for the jobs for the **Jenkins Job Builder** app.
+* [src/](src/) contains the Python sources.
+* [src/modules/](src/modules/) contains the common Python modules of this project
+* [jenkins-job-builer.py](src/jenkins-job-builder.py) is the main program of the **Jenkins Job Builder** app.
+* [operator-test-runner.py](src/operator-test-runner.py) is the main program of the **Operator Test Runner** app.
+* [build.sh](build.sh) lets you build the Docker images locally.
+
+## Build process
+
+The [seed job](../jenkins/jobbuilder.groovy) which is created during Jenkins setup and executed every 5 minutes does build the apps as Docker images inside the Jenkins. They are tagged as `latest` and used by the Jenkins jobs, so every code change pushed to the `main` branch is reflected in near-real-time. No external Docker image registry like Nexus or Harbor is needed here.

apps/src/jenkins-job-builder.py

@@ -1,30 +1,40 @@
+"""
+    Main module of the Jenkins Job builder application
+
+    See https://jenkins-job-builder.readthedocs.io/en/latest/
+"""
 import os
 from jinja2 import Template
 from subprocess import PIPE, Popen
 import json
 
 import modules.catalog as catalog
 
+# values of the params (given as env vars during Docker run)
 param_jenkins_url = None
 param_jenkins_username = None
 param_jenkins_password = None
 
+# keys for the env vars
 PARAM_KEY_JENKINS_URL = 'JENKINS_URL'
 PARAM_KEY_JENKINS_USERNAME = 'JENKINS_USERNAME'
 PARAM_KEY_JENKINS_PASSWORD = 'JENKINS_PASSWORD'
 
 SLACK_CHANNEL = '#team-testing'
-
 CLUSTER_LOGGING_ENDPOINT = 'https://search.t2.stackable.tech'
 OPENSEARCH_DASHBOARDS_URL = 'https://logs.t2.stackable.tech'
 
 def get_platform_metadata():
+    """
+        Creates a dict containing display name and version list for a given platform.
+    """
     return { p['id']: { 'name': p['name'], 'versions': [ v for v in p['versions']] } for p in catalog.platforms }
 
-
 def init():
     """
         Initializes this app, checks if all params are provided as environment variables.
+
+        Returns True if initialization succeeded, False otherwise.
     """
     global param_jenkins_url
     global param_jenkins_username
@@ -48,24 +58,30 @@ def init():
 
     return True
 
-
 def generate_jjb_config():
+    """
+        Creates the config file for the JJB util using Jinja templating.
+    """
     with open("/jjb/jjb.conf.j2", "r") as t:
         template = Template(t.read())
         with open("/jjb/jjb.conf", "w") as f:
             f.write(template.render(os.environ))
             f.close()
 
-
 def generate_jjb_file_maintenance_jobs():
+    """
+        Creates the JJB job definition file defining the maintenance jobs using Jinja templating.
+    """
     with open("/jjb/maintenance.j2", "r") as t:
         template = Template(t.read())
         with open("/jjb/maintenance.yaml", "w") as f:
             f.write(template.render(os.environ))
             f.close()
 
-
 def generate_jjb_file_operator_weekly_test_jobs(platform_metadata):
+    """
+        Creates the JJB job definition files defining the weekly operator test jobs using Jinja templating.
+    """
     with open("/jjb/trigger-weekly-tests.groovy.j2", "r") as t:
         template = Template(t.read())
         with open("/jjb/trigger-weekly-tests.groovy", "w") as f:
@@ -79,6 +95,9 @@ def generate_jjb_file_operator_weekly_test_jobs(platform_metadata):
 
 
 def generate_jjb_file_operator_custom_test_jobs(platform_metadata, operator_versions):
+    """
+        Creates the JJB job definition file defining the custom operator test jobs using Jinja templating.
+    """
     with open("/jjb/operator_custom_tests.j2", "r") as t:
         template = Template(t.read())
         with open("/jjb/operator_custom_tests.yaml", "w") as f:
@@ -87,6 +106,9 @@ def generate_jjb_file_operator_custom_test_jobs(platform_metadata, operator_vers
 
 
 def execute_jjb():
+    """
+        Executes JJB for all job definition files
+    """
     os.system(f"jenkins-jobs --conf /jjb/jjb.conf update /jjb/maintenance.yaml")
     os.system(f"jenkins-jobs --conf /jjb/jjb.conf update /jjb/operator_weekly_tests.yaml")
     os.system(f"jenkins-jobs --conf /jjb/jjb.conf update /jjb/operator_custom_tests.yaml")

apps/src/modules/catalog.py

@@ -1,20 +1,20 @@
-# import sys
-# import os
-# import os.path
-import hiyapyco
-# from jinja2 import Template
-
-# platform_name = None
-# k8s_version = None
-# metadata_annotations = {}
+"""
+    This module enables access to the catalog
+"""
 
-# catalog_platforms = None
+import hiyapyco
 
+# variables holding the catalog data
 providers = []
 platforms = []
 operator_tests = []
 
 def read_providers(logger):
+    """
+        Reads the list of cluster providers from platforms.yaml
+
+        logger              logger (String-consuming function)
+    """
     global providers
     platforms_yaml = hiyapyco.load("/platforms.yaml")
     if not 'providers' in platforms_yaml:
@@ -24,10 +24,15 @@ def read_providers(logger):
     if len(providers) == 0:
         logger('platforms.yaml does not contain providers.')
         return False
-    # TODO More checks to make sure the catalog file is usable
+    # TODO More syntax checks to make sure the catalog file is usable
     return True
 
 def read_platforms(logger):
+    """
+        Reads the list of cluster platforms from platforms.yaml
+
+        logger              logger (String-consuming function)
+    """
     global platforms
     platforms = hiyapyco.load("/platforms.yaml")['platforms']
     platforms_yaml = hiyapyco.load("/platforms.yaml")
@@ -38,21 +43,29 @@ def read_platforms(logger):
     if len(platforms) == 0:
         logger('platforms.yaml does not contain platforms.')
         return False
-    # TODO More checks to make sure the catalog file is usable
+    # TODO More syntax checks to make sure the catalog file is usable
     return True
 
-
 def read_operator_tests(logger):
+    """
+        Reads the list of operator test definitions from operator-tests.yaml
+
+        logger              logger (String-consuming function)
+    """
     global operator_tests
     operator_tests = hiyapyco.load("/operator-tests.yaml")
     if len(operator_tests) == 0:
         logger('operator-tests.yaml does not contain any tests.')
         return False
-    # TODO More checks to make sure the catalog file is usable
+    # TODO More syntax checks to make sure the catalog file is usable
     return True
 
-
 def read_catalog(logger):
+    """
+        Read all catalogs
+
+        logger              logger (String-consuming function)
+    """
     if not read_providers(logger):
         return False
     if not read_platforms(logger):
@@ -64,14 +77,19 @@ def read_catalog(logger):
     logger(f"Read {len(operator_tests)} operator tests: [{','.join([ot['id'] for ot in operator_tests])}]")
     return True
 
-
 def get_platform(platform):
+    """
+        Get the platform matching the platform string.
+        If the platform string matches an ID, that platform is chosen.
+        Otherwise, we search for a matching name attribute.
+
+        platform        string which identifies a platform
+    """
     matching_platform = next(filter(lambda p: p['id']==platform, platforms), None)
     if not matching_platform:
         matching_platform = next(filter(lambda p: p['name']==platform, platforms), None)
     return matching_platform
 
-
 def get_spec_for_operator_test(operator_test, platform, logger):
     """
         Reads the cluster spec for the given operator/platform combination.

apps/src/modules/cluster.py

@@ -1,3 +1,9 @@
+"""
+    This module is a facade to the cluster providers.
+
+    Currently, we offer clusters for replicated.com and IONOS.
+"""
+
 import modules.provider_ionos as ionos
 import modules.provider_replicated as replicated
 
@@ -21,10 +27,13 @@ def create_cluster(provider_id, id, spec, platform_version, cluster_info_file, l
     if 'ionos' == provider_id:
         return ionos.create_cluster(id, spec, platform_version, cluster_info_file, logger)
 
-
 def terminate_cluster(provider_id, cluster, logger):
     """
         Terminates the given cluster. (Blocking operation)
+
+        provider_id         ID of the cloud provider / vendor
+        cluster             vendor-specific cluster object which was previously returned by create_cluster()
+        logger              logger (String-consuming function)
     """
     if 'replicated' == provider_id:
         return replicated.terminate_cluster(cluster, logger)

apps/src/modules/cluster_logging.py

@@ -1,3 +1,11 @@
+"""
+    This module installs the "cluster logging" on a test cluster.
+
+    "cluster logging" refers to the ability of a cluster to forward the logs
+    (K8s Events, Stackable Operator, Stackable Products) to our centralized
+    log index.
+"""
+
 from modules.command import run_command
 
 def install_cluster_logging(cluster_id, endpoint, username, password, logger):
@@ -54,4 +62,3 @@ def install_cluster_logging(cluster_id, endpoint, username, password, logger):
         return False
 
     return True
-    

apps/src/modules/command.py

@@ -1,12 +1,28 @@
+"""
+    This util module helps you to run system processes
+"""
+
 from subprocess import PIPE, TimeoutExpired, Popen
 from time import sleep
 
-
 def output_to_string_array(byte_stream):
+    """
+        Splits the binary output of the command into an array of lines/strings
+    """
     return [line.strip() for line in byte_stream.decode("utf-8").splitlines()]
 
-
 def run_command(command, description, timeout=60, retries=1, delay=60):
+    """
+        Execute command.
+
+        command             command
+        description         command (short) description
+        timeout             time [seconds], after which system call is killed
+        retries             number of tries to execute command if it returns <> 0
+        delay               time [seconds] between two (re)tries.
+
+        Returns tuple (exit_code, output)
+    """
     while retries > 0:
         exit_code, output = _run_command(command, description, timeout)
         if exit_code == 0:
@@ -15,11 +31,15 @@ def run_command(command, description, timeout=60, retries=1, delay=60):
         sleep(delay)
     return exit_code, output
 
-
 def _run_command(command, description, timeout=60):
     """
-        Execute command.
-        Returns tuple (exit_code, output as multi-line)
+        Execute command (module-internal)
+
+        command             command
+        description         command (short) description
+        timeout             time [seconds], after which system call is killed
+
+        Returns tuple (exit_code, output)
     """
     proc = Popen(['/bin/bash', '-c', command], stdout=PIPE, stderr=PIPE)
     try: