Refactor Tool Meister module, add Tool unittests

Refactor the ToolMeister class and the Tool class hierarchy to construct
Tool objects once.  We were running the constructor for each Tool object
once to run the `install()` method, and then again for each "init" and
"start" operation.

Now the Tool object constructor runs once, and all code used to
determine if a tool is installed correctly results in the `install()`
method.

We also take the opportunity to properly assert the expected state each
method of the various Tool classes expect to encounter.

The Tool Meister constructor now validates that the `pbench_install_dir`
and `tmp_dir` arguments are directories that exist, raising an exception
otherwise.

The `.start()` methods for all Tool object sub-classes has been changed
to raise an exception if the `tool_dir` argument is not a real
directory.

Along the way, we also add type hints for all Tool Meister methods and
its constructor, correcting other type hints, and correcting the return
values of a few methods.

Other refactorings:

 * We also add docstrings to the Tool class hierarchy where missing

 * Use named tuples for installation results and tool meister params

 * Refactor {`start`|`init`}`_tools` to share code

 * Refactor the `ToolMeister` class's `start_tools()` and `init_tools()`
   methods to share as much code as possible

 * Rework cleanup of `self._running_tools`

Lastly, we have added unit tests for the Tool class hierarchy and some
helper functions.

Author: portante

agent/util-scripts/gold/test-client-tool-meister/test-53.txt

@@ -184,12 +184,12 @@ WARNING pbench-tool-meister end_tools -- testhost.example.com: unexpected temp f
 INFO pbench-tool-meister sysinfo -- pbench-sysinfo-dump -- /var/tmp/pbench-test-utils/opt/pbench-agent/unittest-scripts/pbench-sysinfo-dump /var/tmp/pbench-test-utils/pbench/mock-run/sysinfo/end/testhost.example.com block,security_mitigations,sos parallel
 INFO pbench-tool-meister sysinfo -- testhost.example.com: sysinfo send (no-op) default /var/tmp/pbench-test-utils/pbench/mock-run/sysinfo/end/testhost.example.com
 INFO pbench-tool-meister __exit__ -- testhost.example.com: terminating
-INFO pbench-tool-meister.logger-start log_subprocess_output -- mpstat: running "/var/tmp/pbench-test-utils/opt/pbench-agent/tool-scripts/datalog/mpstat-datalog 42 --options=forty-two "
-INFO pbench-tool-meister.logger-stop log_subprocess_output -- mpstat: stopping
-INFO pbench-tool-meister.logger-stop log_subprocess_output -- mpstat: post-processing following stop
-INFO pbench-tool-meister.logger-start log_subprocess_output -- mpstat: running "/var/tmp/pbench-test-utils/opt/pbench-agent/tool-scripts/datalog/mpstat-datalog 42 --options=forty-two "
-INFO pbench-tool-meister.logger-stop log_subprocess_output -- mpstat: stopping
-INFO pbench-tool-meister.logger-stop log_subprocess_output -- mpstat: post-processing following stop
+INFO pbench-tool-meister.logger-start log_raw_io_output -- mpstat: running "/var/tmp/pbench-test-utils/opt/pbench-agent/tool-scripts/datalog/mpstat-datalog 42 --options=forty-two "
+INFO pbench-tool-meister.logger-stop log_raw_io_output -- mpstat: stopping
+INFO pbench-tool-meister.logger-stop log_raw_io_output -- mpstat: post-processing following stop
+INFO pbench-tool-meister.logger-start log_raw_io_output -- mpstat: running "/var/tmp/pbench-test-utils/opt/pbench-agent/tool-scripts/datalog/mpstat-datalog 42 --options=forty-two "
+INFO pbench-tool-meister.logger-stop log_raw_io_output -- mpstat: stopping
+INFO pbench-tool-meister.logger-stop log_raw_io_output -- mpstat: post-processing following stop
 --- mock-run/tm/tm-default-testhost.example.com.err file contents
 +++ mock-run/tm/tm-default-testhost.example.com.out file contents
 --- mock-run/tm/tm-default-testhost.example.com.out file contents
@@ -214,12 +214,12 @@ testhost.example.com WARNING pbench-tool-meister end_tools -- testhost.example.c
 testhost.example.com INFO pbench-tool-meister sysinfo -- pbench-sysinfo-dump -- /var/tmp/pbench-test-utils/opt/pbench-agent/unittest-scripts/pbench-sysinfo-dump /var/tmp/pbench-test-utils/pbench/mock-run/sysinfo/end/testhost.example.com block,security_mitigations,sos parallel
 testhost.example.com INFO pbench-tool-meister sysinfo -- testhost.example.com: sysinfo send (no-op) default /var/tmp/pbench-test-utils/pbench/mock-run/sysinfo/end/testhost.example.com
 testhost.example.com INFO pbench-tool-meister __exit__ -- testhost.example.com: terminating
-testhost.example.com INFO pbench-tool-meister.logger-start log_subprocess_output -- mpstat: running "/var/tmp/pbench-test-utils/opt/pbench-agent/tool-scripts/datalog/mpstat-datalog 42 --options=forty-two "
-testhost.example.com INFO pbench-tool-meister.logger-stop log_subprocess_output -- mpstat: stopping
-testhost.example.com INFO pbench-tool-meister.logger-stop log_subprocess_output -- mpstat: post-processing following stop
-testhost.example.com INFO pbench-tool-meister.logger-start log_subprocess_output -- mpstat: running "/var/tmp/pbench-test-utils/opt/pbench-agent/tool-scripts/datalog/mpstat-datalog 42 --options=forty-two "
-testhost.example.com INFO pbench-tool-meister.logger-stop log_subprocess_output -- mpstat: stopping
-testhost.example.com INFO pbench-tool-meister.logger-stop log_subprocess_output -- mpstat: post-processing following stop
+testhost.example.com INFO pbench-tool-meister.logger-start log_raw_io_output -- mpstat: running "/var/tmp/pbench-test-utils/opt/pbench-agent/tool-scripts/datalog/mpstat-datalog 42 --options=forty-two "
+testhost.example.com INFO pbench-tool-meister.logger-stop log_raw_io_output -- mpstat: stopping
+testhost.example.com INFO pbench-tool-meister.logger-stop log_raw_io_output -- mpstat: post-processing following stop
+testhost.example.com INFO pbench-tool-meister.logger-start log_raw_io_output -- mpstat: running "/var/tmp/pbench-test-utils/opt/pbench-agent/tool-scripts/datalog/mpstat-datalog 42 --options=forty-two "
+testhost.example.com INFO pbench-tool-meister.logger-stop log_raw_io_output -- mpstat: stopping
+testhost.example.com INFO pbench-tool-meister.logger-stop log_raw_io_output -- mpstat: post-processing following stop
 --- mock-run/tm/tm.logs file contents
 +++ tools-default/prometheus/prom.log file contents
 --- tools-default/prometheus/prom.log file contents

agent/util-scripts/gold/test-client-tool-meister/test-56.txt

@@ -169,12 +169,12 @@ INFO pbench-tool-meister _send_directory -- remote-a.example.com: PUT tool-data
 INFO pbench-tool-meister sysinfo -- pbench-sysinfo-dump -- /var/tmp/pbench-test-utils/opt/pbench-agent/unittest-scripts/pbench-sysinfo-dump /var/tmp/pbench-test-utils/pbench/tmp/tm.lite.NNNNN.nnnnnnnn/remote-a.example.com block,security_mitigations,sos parallel
 INFO pbench-tool-meister _send_directory -- remote-a.example.com: PUT sysinfo-data completed lite /var/tmp/pbench-test-utils/pbench/tmp/tm.lite.NNNNN.nnnnnnnn/remote-a.example.com
 INFO pbench-tool-meister __exit__ -- remote-a.example.com: terminating
-INFO pbench-tool-meister.logger-start log_subprocess_output -- mpstat: running "/var/tmp/pbench-test-utils/opt/pbench-agent/tool-scripts/datalog/mpstat-datalog 42 --options=forty-two "
-INFO pbench-tool-meister.logger-stop log_subprocess_output -- mpstat: stopping
-INFO pbench-tool-meister.logger-stop log_subprocess_output -- mpstat: post-processing following stop
-INFO pbench-tool-meister.logger-start log_subprocess_output -- mpstat: running "/var/tmp/pbench-test-utils/opt/pbench-agent/tool-scripts/datalog/mpstat-datalog 42 --options=forty-two "
-INFO pbench-tool-meister.logger-stop log_subprocess_output -- mpstat: stopping
-INFO pbench-tool-meister.logger-stop log_subprocess_output -- mpstat: post-processing following stop
+INFO pbench-tool-meister.logger-start log_raw_io_output -- mpstat: running "/var/tmp/pbench-test-utils/opt/pbench-agent/tool-scripts/datalog/mpstat-datalog 42 --options=forty-two "
+INFO pbench-tool-meister.logger-stop log_raw_io_output -- mpstat: stopping
+INFO pbench-tool-meister.logger-stop log_raw_io_output -- mpstat: post-processing following stop
+INFO pbench-tool-meister.logger-start log_raw_io_output -- mpstat: running "/var/tmp/pbench-test-utils/opt/pbench-agent/tool-scripts/datalog/mpstat-datalog 42 --options=forty-two "
+INFO pbench-tool-meister.logger-stop log_raw_io_output -- mpstat: stopping
+INFO pbench-tool-meister.logger-stop log_raw_io_output -- mpstat: post-processing following stop
 === /var/tmp/pbench-test-utils/pbench/tmp/tm-lite-remote-a.example.com.out:
 === /var/tmp/pbench-test-utils/pbench/tmp/tm-lite-remote-b.example.com.err:
 INFO pbench-tool-meister sysinfo -- pbench-sysinfo-dump -- /var/tmp/pbench-test-utils/opt/pbench-agent/unittest-scripts/pbench-sysinfo-dump /var/tmp/pbench-test-utils/pbench/tmp/tm.lite.NNNNN.nnnnnnnn/blue:remote-b.example.com block,security_mitigations,sos parallel
@@ -196,12 +196,12 @@ WARNING pbench-tool-meister end_tools -- remote-b.example.com: unexpected temp f
 INFO pbench-tool-meister sysinfo -- pbench-sysinfo-dump -- /var/tmp/pbench-test-utils/opt/pbench-agent/unittest-scripts/pbench-sysinfo-dump /var/tmp/pbench-test-utils/pbench/tmp/tm.lite.NNNNN.nnnnnnnn/blue:remote-b.example.com block,security_mitigations,sos parallel
 INFO pbench-tool-meister _send_directory -- remote-b.example.com: PUT sysinfo-data completed lite /var/tmp/pbench-test-utils/pbench/tmp/tm.lite.NNNNN.nnnnnnnn/blue:remote-b.example.com
 INFO pbench-tool-meister __exit__ -- remote-b.example.com: terminating
-INFO pbench-tool-meister.logger-start log_subprocess_output -- mpstat: running "/var/tmp/pbench-test-utils/opt/pbench-agent/tool-scripts/datalog/mpstat-datalog 42 --options=forty-two "
-INFO pbench-tool-meister.logger-stop log_subprocess_output -- mpstat: stopping
-INFO pbench-tool-meister.logger-stop log_subprocess_output -- mpstat: post-processing following stop
-INFO pbench-tool-meister.logger-start log_subprocess_output -- mpstat: running "/var/tmp/pbench-test-utils/opt/pbench-agent/tool-scripts/datalog/mpstat-datalog 42 --options=forty-two "
-INFO pbench-tool-meister.logger-stop log_subprocess_output -- mpstat: stopping
-INFO pbench-tool-meister.logger-stop log_subprocess_output -- mpstat: post-processing following stop
+INFO pbench-tool-meister.logger-start log_raw_io_output -- mpstat: running "/var/tmp/pbench-test-utils/opt/pbench-agent/tool-scripts/datalog/mpstat-datalog 42 --options=forty-two "
+INFO pbench-tool-meister.logger-stop log_raw_io_output -- mpstat: stopping
+INFO pbench-tool-meister.logger-stop log_raw_io_output -- mpstat: post-processing following stop
+INFO pbench-tool-meister.logger-start log_raw_io_output -- mpstat: running "/var/tmp/pbench-test-utils/opt/pbench-agent/tool-scripts/datalog/mpstat-datalog 42 --options=forty-two "
+INFO pbench-tool-meister.logger-stop log_raw_io_output -- mpstat: stopping
+INFO pbench-tool-meister.logger-stop log_raw_io_output -- mpstat: post-processing following stop
 === /var/tmp/pbench-test-utils/pbench/tmp/tm-lite-remote-b.example.com.out:
 === /var/tmp/pbench-test-utils/pbench/tmp/tm-lite-remote-c.example.com.err:
 INFO pbench-tool-meister sysinfo -- pbench-sysinfo-dump -- /var/tmp/pbench-test-utils/opt/pbench-agent/unittest-scripts/pbench-sysinfo-dump /var/tmp/pbench-test-utils/pbench/tmp/tm.lite.NNNNN.nnnnnnnn/red:remote-c.example.com block,security_mitigations,sos parallel
@@ -446,12 +446,12 @@ WARNING pbench-tool-meister end_tools -- testhost.example.com: unexpected temp f
 INFO pbench-tool-meister sysinfo -- pbench-sysinfo-dump -- /var/tmp/pbench-test-utils/opt/pbench-agent/unittest-scripts/pbench-sysinfo-dump /var/tmp/pbench-test-utils/pbench/mock-run/sysinfo/end/testhost.example.com block,security_mitigations,sos parallel
 INFO pbench-tool-meister sysinfo -- testhost.example.com: sysinfo send (no-op) lite /var/tmp/pbench-test-utils/pbench/mock-run/sysinfo/end/testhost.example.com
 INFO pbench-tool-meister __exit__ -- testhost.example.com: terminating
-INFO pbench-tool-meister.logger-start log_subprocess_output -- mpstat: running "/var/tmp/pbench-test-utils/opt/pbench-agent/tool-scripts/datalog/mpstat-datalog 42 --options=forty-two "
-INFO pbench-tool-meister.logger-stop log_subprocess_output -- mpstat: stopping
-INFO pbench-tool-meister.logger-stop log_subprocess_output -- mpstat: post-processing following stop
-INFO pbench-tool-meister.logger-start log_subprocess_output -- mpstat: running "/var/tmp/pbench-test-utils/opt/pbench-agent/tool-scripts/datalog/mpstat-datalog 42 --options=forty-two "
-INFO pbench-tool-meister.logger-stop log_subprocess_output -- mpstat: stopping
-INFO pbench-tool-meister.logger-stop log_subprocess_output -- mpstat: post-processing following stop
+INFO pbench-tool-meister.logger-start log_raw_io_output -- mpstat: running "/var/tmp/pbench-test-utils/opt/pbench-agent/tool-scripts/datalog/mpstat-datalog 42 --options=forty-two "
+INFO pbench-tool-meister.logger-stop log_raw_io_output -- mpstat: stopping
+INFO pbench-tool-meister.logger-stop log_raw_io_output -- mpstat: post-processing following stop
+INFO pbench-tool-meister.logger-start log_raw_io_output -- mpstat: running "/var/tmp/pbench-test-utils/opt/pbench-agent/tool-scripts/datalog/mpstat-datalog 42 --options=forty-two "
+INFO pbench-tool-meister.logger-stop log_raw_io_output -- mpstat: stopping
+INFO pbench-tool-meister.logger-stop log_raw_io_output -- mpstat: post-processing following stop
 --- mock-run/tm/tm-lite-testhost.example.com.err file contents
 +++ mock-run/tm/tm-lite-testhost.example.com.out file contents
 --- mock-run/tm/tm-lite-testhost.example.com.out file contents
@@ -522,24 +522,24 @@ testhost.example.com WARNING pbench-tool-meister end_tools -- testhost.example.c
 testhost.example.com INFO pbench-tool-meister sysinfo -- pbench-sysinfo-dump -- /var/tmp/pbench-test-utils/opt/pbench-agent/unittest-scripts/pbench-sysinfo-dump /var/tmp/pbench-test-utils/pbench/mock-run/sysinfo/end/testhost.example.com block,security_mitigations,sos parallel
 testhost.example.com INFO pbench-tool-meister sysinfo -- testhost.example.com: sysinfo send (no-op) lite /var/tmp/pbench-test-utils/pbench/mock-run/sysinfo/end/testhost.example.com
 testhost.example.com INFO pbench-tool-meister __exit__ -- testhost.example.com: terminating
-remote-a.example.com INFO pbench-tool-meister.logger-start log_subprocess_output -- mpstat: running "/var/tmp/pbench-test-utils/opt/pbench-agent/tool-scripts/datalog/mpstat-datalog 42 --options=forty-two "
-remote-a.example.com INFO pbench-tool-meister.logger-stop log_subprocess_output -- mpstat: stopping
-remote-a.example.com INFO pbench-tool-meister.logger-stop log_subprocess_output -- mpstat: post-processing following stop
-remote-a.example.com INFO pbench-tool-meister.logger-start log_subprocess_output -- mpstat: running "/var/tmp/pbench-test-utils/opt/pbench-agent/tool-scripts/datalog/mpstat-datalog 42 --options=forty-two "
-remote-a.example.com INFO pbench-tool-meister.logger-stop log_subprocess_output -- mpstat: stopping
-remote-a.example.com INFO pbench-tool-meister.logger-stop log_subprocess_output -- mpstat: post-processing following stop
-remote-b.example.com INFO pbench-tool-meister.logger-start log_subprocess_output -- mpstat: running "/var/tmp/pbench-test-utils/opt/pbench-agent/tool-scripts/datalog/mpstat-datalog 42 --options=forty-two "
-remote-b.example.com INFO pbench-tool-meister.logger-stop log_subprocess_output -- mpstat: stopping
-remote-b.example.com INFO pbench-tool-meister.logger-stop log_subprocess_output -- mpstat: post-processing following stop
-remote-b.example.com INFO pbench-tool-meister.logger-start log_subprocess_output -- mpstat: running "/var/tmp/pbench-test-utils/opt/pbench-agent/tool-scripts/datalog/mpstat-datalog 42 --options=forty-two "
-remote-b.example.com INFO pbench-tool-meister.logger-stop log_subprocess_output -- mpstat: stopping
-remote-b.example.com INFO pbench-tool-meister.logger-stop log_subprocess_output -- mpstat: post-processing following stop
-testhost.example.com INFO pbench-tool-meister.logger-start log_subprocess_output -- mpstat: running "/var/tmp/pbench-test-utils/opt/pbench-agent/tool-scripts/datalog/mpstat-datalog 42 --options=forty-two "
-testhost.example.com INFO pbench-tool-meister.logger-stop log_subprocess_output -- mpstat: stopping
-testhost.example.com INFO pbench-tool-meister.logger-stop log_subprocess_output -- mpstat: post-processing following stop
-testhost.example.com INFO pbench-tool-meister.logger-start log_subprocess_output -- mpstat: running "/var/tmp/pbench-test-utils/opt/pbench-agent/tool-scripts/datalog/mpstat-datalog 42 --options=forty-two "
-testhost.example.com INFO pbench-tool-meister.logger-stop log_subprocess_output -- mpstat: stopping
-testhost.example.com INFO pbench-tool-meister.logger-stop log_subprocess_output -- mpstat: post-processing following stop
+remote-a.example.com INFO pbench-tool-meister.logger-start log_raw_io_output -- mpstat: running "/var/tmp/pbench-test-utils/opt/pbench-agent/tool-scripts/datalog/mpstat-datalog 42 --options=forty-two "
+remote-a.example.com INFO pbench-tool-meister.logger-stop log_raw_io_output -- mpstat: stopping
+remote-a.example.com INFO pbench-tool-meister.logger-stop log_raw_io_output -- mpstat: post-processing following stop
+remote-a.example.com INFO pbench-tool-meister.logger-start log_raw_io_output -- mpstat: running "/var/tmp/pbench-test-utils/opt/pbench-agent/tool-scripts/datalog/mpstat-datalog 42 --options=forty-two "
+remote-a.example.com INFO pbench-tool-meister.logger-stop log_raw_io_output -- mpstat: stopping
+remote-a.example.com INFO pbench-tool-meister.logger-stop log_raw_io_output -- mpstat: post-processing following stop
+remote-b.example.com INFO pbench-tool-meister.logger-start log_raw_io_output -- mpstat: running "/var/tmp/pbench-test-utils/opt/pbench-agent/tool-scripts/datalog/mpstat-datalog 42 --options=forty-two "
+remote-b.example.com INFO pbench-tool-meister.logger-stop log_raw_io_output -- mpstat: stopping
+remote-b.example.com INFO pbench-tool-meister.logger-stop log_raw_io_output -- mpstat: post-processing following stop
+remote-b.example.com INFO pbench-tool-meister.logger-start log_raw_io_output -- mpstat: running "/var/tmp/pbench-test-utils/opt/pbench-agent/tool-scripts/datalog/mpstat-datalog 42 --options=forty-two "
+remote-b.example.com INFO pbench-tool-meister.logger-stop log_raw_io_output -- mpstat: stopping
+remote-b.example.com INFO pbench-tool-meister.logger-stop log_raw_io_output -- mpstat: post-processing following stop
+testhost.example.com INFO pbench-tool-meister.logger-start log_raw_io_output -- mpstat: running "/var/tmp/pbench-test-utils/opt/pbench-agent/tool-scripts/datalog/mpstat-datalog 42 --options=forty-two "
+testhost.example.com INFO pbench-tool-meister.logger-stop log_raw_io_output -- mpstat: stopping
+testhost.example.com INFO pbench-tool-meister.logger-stop log_raw_io_output -- mpstat: post-processing following stop
+testhost.example.com INFO pbench-tool-meister.logger-start log_raw_io_output -- mpstat: running "/var/tmp/pbench-test-utils/opt/pbench-agent/tool-scripts/datalog/mpstat-datalog 42 --options=forty-two "
+testhost.example.com INFO pbench-tool-meister.logger-stop log_raw_io_output -- mpstat: stopping
+testhost.example.com INFO pbench-tool-meister.logger-stop log_raw_io_output -- mpstat: post-processing following stop
 --- mock-run/tm/tm.logs file contents
 +++ tools-lite/pcp/pmproxy-proc.log file contents
 --- tools-lite/pcp/pmproxy-proc.log file contents