Added documentation to all of the test files. The documentation for the

clitest.py file could probably be better, but that can happen when/if it
is ever actually put in it's own repository to make a little test
framework. Each test file has some information on how test are made for
its specific needs. The general info was added to the test_word.py file.
And of course the information on each actual operation being tested
available in the project README.

Removed old test files that were no longer in use. Also removed the main
functions from the individual test files. They are somewhat useful if
you want to run an individual test suite, but so far I have never done
it and it reduces a point of maintenance when the testing classes and
functions change.

Also made some adjustment to the makefile because the default .c rule
was compiling with the release flags.

Author: strinsberg

Makefile

@@ -1,22 +1,23 @@
 CC= gcc
-CC_FLAGS = -o lt64 -O3 -I include
-CC_DEBUG_FLAGS= -o lt64-debug -Og -D DEBUG -I include
-CC_TEST_FLAGS= -o lt64-test -Og -D TEST -I include
+CC_FLAGS = -I include
+CC_RELEASE_FLAGS = -o lt64 -O3
+CC_DEBUG_FLAGS= -o lt64-debug -Og -D DEBUG
+CC_TEST_FLAGS= -o lt64-test -Og -D TEST
 
 %.o: %.cpp
 	$(CC) $(CC_FLAGS) -c $< -o $@
 
 .PHONY: release
 release: src/main.c
-	$(CC) src/* $(CC_FLAGS)
+	$(CC) src/* $(CC_FLAGS) $(CC_RELEASE_FLAGS)
 
 .PHONY: debug
 debug: src/main.c
-	$(CC) src/* $(CC_DEBUG_FLAGS)
+	$(CC) src/* $(CC_FLAGS) $(CC_DEBUG_FLAGS)
 
 .PHONY: tests
 tests: src/main.c
-	$(CC) src/* $(CC_TEST_FLAGS)
+	$(CC) src/* $(CC_FLAGS) $(CC_TEST_FLAGS)
 	python3 test/runner.py
 	rm test.vm lt64-test
 

test/clitest.py

@@ -1,3 +1,26 @@
+"""
+TestSuite and Test objects and some helper functions for running simple tests
+on command line programs.
+
+Tries to do a lot and not too much while providing simple clean output for
+test results. Uses ansi color codes to color the output. This will make test
+output hard to read on terminals that do not support them.
+
+Lots of different kind of tests that feed a command line program some input and
+expect some output can be run with these tools. However, it is likely that some
+things will need to be adjusted. The test suite has been written with a lot of
+small methods that are called by other methods. This gives it a bit of a
+template pattern to replace any little peice as needed. This may be to read
+input or output differently or to perform a different check to determine
+success or failure or to add new data and operations while keeping the same
+general pattern. The helper functions are also separate and not TestSuite
+methods to allow their use in different ways without always subclassing
+TestSuite. To see some project specific uses see the lieutenant-64 repository.
+
+TODO Make this it's own project and repo. Then add some examples of how it
+could be used to test certain kinds of program and maybe some simple extensions
+to cover common use cases not covered by the base class.
+"""
 import subprocess as sp
 
 ### Exit codes ###
@@ -11,6 +34,15 @@
 
 ### Functions ################################################################
 def display_totals(failed, total, reprint_failed=True, show_fail_info=True):
+    """Display the results of a test run to stdout.
+
+    failed is a list of failed Test objects. If there are any failed tests,
+    and reprint_failed is true, they will have their individual results
+    printed with all their info. If show_fail_info is false then the short
+    output for each failed test result will be reprinted.
+
+    total is the number of tests that were run.
+    """
     print()
     print()
     if len(failed):
@@ -25,6 +57,14 @@ def display_totals(failed, total, reprint_failed=True, show_fail_info=True):
         print(f"{GREEN}****{NORMAL} {total} Tests Passed {GREEN}****{NORMAL}")
 
 def display_test_results(test, passed, num, show_fail_info=True):
+    """Displays a line with the test name and whether it passed or failed.
+
+    If show_fail_info is true more detailed output will be given with any error
+    text and exit code followed by the expected output and the actual output.
+
+    num is the number of the test in it's test suite and will be printed before
+    the test name.
+    """
     print(f"{f'{num}:':<3} {test.name:<70}", end="")
     if passed:
         print(f"[{GREEN}PASS{NORMAL}]")
@@ -41,6 +81,14 @@ def display_test_results(test, passed, num, show_fail_info=True):
             print()
 
 def compile_program(command_list):
+    """Given a list of the compilation command and it's arguments will run
+    a subprocess to compile a program.
+
+    Captures error output and the exit code and prints them to stderr if the
+    compilation fails.
+
+    Retruns True on successful compilation and False otherwise.
+    """
     command = " ".join(command_list)
     print(f"Compiling: {command}")
     comp = sp.run(command_list, capture_output=True)
@@ -53,6 +101,11 @@ def compile_program(command_list):
     return True
 
 def run_test_suites(test_suites):
+    """Runs each test suite in a list of tests suites.
+
+    Sets all test output for the running of the tests to False so that only
+    the test names and results are printed during the run of each suite.
+    At the end reprints each failed test with all of it's verbose info."""
     failed = []
     total = 0
     for suite in test_suites:
@@ -69,6 +122,42 @@ def run_test_suites(test_suites):
 
 # TODO implement behaviour for show_line_diff=True
 class TestSuite:
+    """A collection of tests for a command line program that will be run
+    as a group.
+
+    Takes a test suite name, a command for the command line program to run for
+    each test, and a list of tests.
+
+    The input to the program defaults to stdin, but program_source can be
+    passed a file name to read the input to pass to the program under test's
+    subprocess. In a similar way program output can specify where to expect
+    the actual output. If it is stdout the programs output will be read from
+    stdout. If it is a filename that file will be read instead.
+
+    input_is_file tells the test suite that the program under test expects to
+    read from a file instead of being given input. If this is true then
+    program source must be set as the filename and any program input will be
+    written to that file so the program under test can use it to get test
+    input.
+
+    expected_is_file tells the test suite that the expected output needs to be
+    read from a file. This might be handy if the expected output is very large
+    and it is impractical to specify it in a string when creating the Test
+    object.
+    
+    print_end_info will tell the suite that when it is finished running it
+    should display some information about the run of the suite. If this is
+    true and reprint_failed is true then each failed test will be reprinted
+    after the tests have all finished running.
+
+    show_fail_info specifies that when a test fails a verbose output of
+    failure information should be printed. This might contain error information
+    and will always contain the expected output vs the actual output.
+
+    show_line_diff(unimplemented) is indended to give a view of the difference
+    between the expected and actual instead of printing them completely.
+    However it has not been setup yet.
+    """
     def __init__(self, name, command,
                  tests=[], compile_command=None,
                  program_source="stdin", program_output="stdout",
@@ -90,6 +179,9 @@ def __init__(self, name, command,
         self.failed=[]
 
     def run(self):
+        """Runs each test in order and displays the results for each run
+        and the totals if print_end_info is true.
+        """
         print()
         print(f"======== {self.name} ========")
         if self.compile_command:
@@ -102,6 +194,8 @@ def run(self):
                            self.reprint_failed, self.show_fail_info)
 
     def execute(self):
+        """Executes each test and prints the individual test info
+        according to the printing flags that are set."""
         for i, t in enumerate(self.tests):
             if self.program_source != "stdin":
                 self.write_program_input(t)
@@ -118,24 +212,41 @@ def execute(self):
                 self.failed.append((i+1, t))
 
     def write_program_input(self, test):
-        print("base write")
+        """Writes a test's program input to a file so that it can be read
+        by the program under test. Can be overridden if a specific Test type
+        might need a different output type. I.e. writing a binary file."""
         with open(self.program_source, 'w+') as f:
             f.write(test.input)
 
     def get_test_input(self, test):
+        """Returns the test input. If the test expects input from a file it
+        will be read in and returned. Otherwise the test input will be
+        returned as a byte string. The reason for this is that Subprocess.run
+        expects program input as a byte string. It can also be overriden if
+        getting or creating the test input from a test is more complex. I.e.
+        test input is given as a hex string and needs to be converted to the
+        appropriate representation before being returned as a byte string."""
         if self.input_is_file:
             with open(test.input, 'rb') as f:
                 return f.read()
         else:
             return test.input().encode('utf-8')
 
     def check(self, test, proc):
+        """Checks to see if the process result output matches the expected
+        output. To pass a test must have matching expected and actual
+        output as well as return 0 for the exit code."""
         actual = self.get_actual(test, proc)
         expected = self.get_expected(test, proc)
         test.exit_code = proc.returncode
         return actual == expected and test.exit_code == EXIT_SUCCESS
 
     def get_actual(self, test, proc):
+        """Returns the string of actual output after stripping whitespace
+        from the ends. If the output is expected to be in a file it reads it
+        from, otherwise the output captured to stdout by the process run is
+        returned as a utf-8 string. Can be overriden to provide different
+        behaviour such as returning an ascii string."""
         if self.program_output != "stdout":
             with open(self.program_output, 'r'):
                 test.actual = f.read().strip()
@@ -144,6 +255,9 @@ def get_actual(self, test, proc):
         return test.actual
 
     def get_expected(self, test, proc):
+        """Retrns the string of expected output. As get_actual it will read
+        from a file or from the test's expected propery. Strips all leading
+        and trailing whitespace. Can be also be overriden."""
         if self.expected_is_file:
             with open(test.expected, 'r'):
                 test.expected = f.read().strip()
@@ -152,6 +266,20 @@ def get_expected(self, test, proc):
         return test.expected
 
 class Test:
+    """A simple data object for a test used to test the running of
+    a cli program.
+
+    name is the name that will be printed when showing test results.
+
+    input_ and expected are strings for data to pass to the program
+    under test and what to compare its output to.
+
+    show_fail_info tells display functions whether or not to show detailed
+    information on test failure.
+
+    members actual, stderr, and exit_code are provided to add information to
+    a test after running it to save for processes that may need it but run
+    in a different place than where the test was executed."""
     def __init__(self, name, input_, expected, show_fail_info=True):
         self.name = name
         self.input = input_

test/runner.py

@@ -1,28 +1,38 @@
+"""
+Test runner for the lieutenant-64 virtual machine.
+
+Assembles and runs all test suites and their tests. Calling
+`python3 runner.py` will execute all tests for the VM. The only thing it
+requires is that the lt64-test binary has been created. To both create the
+binary and run the tests and clean up afterward run `make tests`.
+
+For more information on the tests see the individual test files.
+"""
 from clitest import *
 import vmtest
 import test_word_ops as word
 import test_dword_ops as dword
 import test_movement_ops as move
 import test_io_ops as io
-import test_bit_ops as bits
 import test_errors as errors
 import sys
 
 def make_vm_suite(name, tests, prog_name):
+    """Helper to more easily create a VmTests object."""
     return vmtest.VmTests(name,
                           prog_name,
                           tests=tests,
                           program_source=vmtest.INPUT_FILE)
 
+# The name of the program under test
 prog_name = vmtest.TEST_NAME
 
+# The various test suites to use to test the VM
 suites = [
     make_vm_suite("Handles word ops", word.tests, prog_name),
     make_vm_suite("Handles dword ops", dword.tests, prog_name),
-    #make_vm_suite("Handles qword ops", qword.tests, prog_name),
     make_vm_suite("Handles movement ops", move.tests, prog_name),
     make_vm_suite("Handles print ops", io.tests, prog_name),
-    #make_vm_suite("Handles bit ops", bits.tests, prog_name),
     vmtest.VmIoTests("Handles read ops",
                      prog_name,
                      tests=io.read_tests,
@@ -33,5 +43,6 @@ def make_vm_suite(name, tests, prog_name):
                         program_source=vmtest.INPUT_FILE),
 ]
 
+# Execute each given test suite
 run_test_suites(suites)