fix: Compose now uses pull

Adds log to test group compose actions
Add ability to add a delay to test-group compose startup

Author: Alan Christie

README.rst

@@ -100,6 +100,12 @@ You can display the utility's help with::
 
     jote --help
 
+Jote container network
+----------------------
+
+``jote`` tests are executed on the network ``data-manager_jote``. This is
+defined in the docker-compose file that it generates to run your tests.
+
 Built-in variables
 ------------------
 
@@ -190,6 +196,81 @@ You should try and avoid creating too many long-running tests. If you cannot,
 consider whether it's a appropriate to use ``run-level`` to avoid ``jote``
 running them by default.
 
+Test groups
+-----------
+
+Tests are normally executed and the environment torn-down between them.
+If you have tests that depend on the results from a prior test you can run
+tests as a **group**, which preserves the project directory between the tests.
+
+To run a sequence of test (as a **group**) you need to define a ``test-group``
+in your Job Definition file and then refer to that group in your test. Here,
+we define a test group called ``experiment-a``, at the top of the
+definition file::
+
+    test-groups:
+    - name: experiment-a
+
+
+We then place a test in that group with a ``run-group`` declaration
+in the corresponding test block::
+
+    jobs:
+      job-a:
+        [...]
+        tests:
+          test-1:
+            run-groups:
+            - name: experiment-a
+              ordinal: 1
+
+We need to provide an ``ordinal`` value. This numeric value (from 1 ..N)
+puts the test in a specific position in the test sequence. When tests are
+placed in a ``run-group`` you have to order your tests so that ``a`` follows
+``b``. This is done with unique ordinals for each test in each group. A test
+with ordinal ``1`` will run before a test with ordinal ``2``.
+
+You can run just the tests for a specific group by using  the ``--run-group``
+option::
+
+    jote --run-group experiment-a
+
+Running additional containers (group testing)
+---------------------------------------------
+
+Test groups provide an ability to launch additional support containers during
+testing. You might want to start a background database for example, that can
+be used by tests in your ``test-group``. To take advantage of this feature
+you just need to provide a ``docker-compose`` file (in the Job definition
+``data-manager`` directory) and name that file in you r``test-groups``
+declaration.
+
+Here we declare a docker-compose file called
+``docker-compose-experiment-a.yaml``::
+
+     test-groups:
+    - name: experiment-a
+      compose:
+        file: docker-compose-experiment-a.yaml
+
+The compose filename must begin ``docker-compose`` and end ``.yaml``.
+
+The compose file is run before any tests in the corresponding test group
+have been run and will be stopped after the last test in the group.
+
+The compose file you provide is run in a *detached* state so ``jote`` does
+not wait for the containers to start (or initialise). As the first test
+in the test group can begin very soon after the compose file is started
+you can minimise the risk that your containers are not ready for the tests
+by adding a fixed delay between ``jote`` starting the compose file and
+running the first test::
+
+     test-groups:
+    - name: experiment-a
+      compose:
+        file: docker-compose-experiment-a.yaml
+        delay-seconds: 10
+
 Nextflow test execution
 -----------------------
 
@@ -208,7 +289,7 @@ constraints.
 
 When running nextflow jobs ``jote`` writes a ``nextflow.config`` to the
 test's simulated project directory prior to executing the command, and
-this is the curent-workign directory when the test starts.
+this is the current-working directory when the test starts.
 ``jote`` *will not* let you have a nextflow config in your home directory
 as any settings found there would be merged with the file ``jote`` writes,
 potentially disturbing the execution behaviour.

requirements.txt

@@ -1,5 +1,5 @@
 docker-compose == 1.29.2
-im-data-manager-job-decoder == 1.12.2
+im-data-manager-job-decoder == 1.12.3
 munch == 2.5.0
 pyyaml == 5.4.1
 yamllint == 1.28.0

src/jote/compose.py

@@ -270,10 +270,31 @@ def delete(self) -> None:
         print("# Compose: Deleted")
 
     @staticmethod
-    def run_group_compose_file(compose_file: str) -> bool:
-        """Runs a group compose file in a detached state.
-        The file is expected to be resident in the 'data-manager' directory."""
+    def run_group_compose_file(compose_file: str, delay_seconds: int = 0) -> bool:
+        """Starts a group compose file in a detached state.
+        The file is expected to be a compose file in the 'data-manager' directory.
+        We pull the continer imag to reduce the 'docker-compose up' time
+        and then optionally wait for a period of seconds.
+        """
+
+        print("# Compose: Starting test group containers...")
+
+        # Runs a group compose file in a detached state.
+        # The file is expected to be resident in the 'data-manager' directory.
         try:
+            # Pre-pull the docker-compose images.
+            # This saves start-up execution time.
+            _ = subprocess.run(
+                [
+                    "docker-compose",
+                    "-f",
+                    os.path.join("data-manager", compose_file),
+                    "pull",
+                ],
+                capture_output=False,
+                check=False,
+            )
+
             # Bring the group-test compose file up.
             # By using '-p' ('--project-name')
             # we set the prefix for the network name and services from this container
@@ -295,16 +316,22 @@ def run_group_compose_file(compose_file: str) -> bool:
         except:  # pylint: disable=bare-except
             return False
 
-        # Wait for 10 seconds.
-        # Giving the container
-        time.sleep(10)
+        # Wait for a period of seconds after launching?
+        if delay_seconds:
+            print(f"# Compose: Post-bring-up test group sleep ({delay_seconds})...")
+            time.sleep(delay_seconds)
 
+        print("# Compose: Started test group containers")
         return True
 
     @staticmethod
     def stop_group_compose_file(compose_file: str) -> bool:
-        """Runs a group compose file in a detached state.
-        The file is expected to be ..."""
+        """Stops a group compose file.
+        The file is expected to be a compose file in the 'data-manager' directory.
+        """
+
+        print("# Compose: Stopping test group containers...")
+
         try:
             # Bring the compose file down...
             _ = subprocess.run(
@@ -322,4 +349,6 @@ def stop_group_compose_file(compose_file: str) -> bool:
         except:  # pylint: disable=bare-except
             return False
 
+        print("# Compose: Stopped test group containers")
+
         return True

src/jote/jote.py

@@ -965,15 +965,17 @@ def _run_grouped_tests(
                 # an 'ordinal', 'job name', 'job test' and the 'job' definition
 
                 # Start the group compose file?
-                if (
-                    index == 0
-                    and "compose-file" in grouped_test[0]
-                    and not args.dry_run
-                ):
-                    group_compose_file = grouped_test[0]["compose-file"]
+                if index == 0 and "compose" in grouped_test[0] and not args.dry_run:
+                    group_compose_file = grouped_test[0].compose.file
                     assert group_compose_file
+                    # Optional post-compose (up) delay?
+                    delay_seconds: int = 0
+                    if "delay-seconds" in grouped_test[0].compose:
+                        delay_seconds = grouped_test[0].compose["delay-seconds"]
+                    # Bring 'up' the group compose file...
                     g_compose_result: bool = Compose.run_group_compose_file(
-                        group_compose_file
+                        group_compose_file,
+                        delay_seconds,
                     )
                     if not g_compose_result:
                         print("! FAILURE")