Run Tox tests in the Github Action CI (#49)

This patch enables running the Tox tests in Github Actions CI.
It also adds some dependencies required to run the tests locally
with Tox.

An important change is that we move from LLVM's FileCheck
implementation to a near-equivalent Python version that can be
installed with pip. This will make the test suite more independent
and easier to run: requiring a full LLVM monorepo checkout and a
configured LLVM build just to run the LNT unit tests seems like a
very cumbersome requirement.

Also, this updates the documentation to reflect my current
understanding of how the test suite can be run.

Note that the tests are not passing at the moment, we will need to
update some dependencies and fix other issues. But this gets us
closer to running them, and at least we can see the progress being
made in the CI.

Author: ldionne

.github/workflows/flake8_docs.yaml renamed to .github/workflows/tox.yaml

@@ -1,4 +1,4 @@
-name: Python Flake8, Docs, mypy
+name: Run Tox tests
 
 on:
   - push
@@ -24,3 +24,5 @@ jobs:
       run: tox -e docs
     - name: Tox mypy
       run: tox -e mypy
+    - name: Tox py3
+      run: tox -e py3

docs/developer_guide.rst

@@ -24,20 +24,20 @@ all times, therefore you should run the regression tests as part of your
 development work-flow, just like you do when developing on other LLVM
 sub-projects.
 
-The LNT regression tests make use of lit and other tools like FileCheck. At
-the moment, probably the easiest way to get them installed is to compile LLVM
-and use the binaries that are generated there. Assuming you've build LLVM
-into $LLVMBUILD, and installed lnt in $LNTINSTALL you can run the regression
-tests using the following command::
+The LNT regression tests make use of lit and other tools like [filecheck](https://github.com/AntonLydike/filecheck).
+To run the tests, we recomment using ``tox`` in a virtual environment::
 
-     PATH=$LLVMBUILD/bin:$LNTINSTALL/bin:$PATH llvm-lit -sv ./tests
+    python3 -m venv .venv
+    source .venv/bin/activate
+    pip install tox
+    tox
 
-If you don't like temporary files being created in your LNT source directory,
-you can run the tests in a different directory too::
+You can also run individual unit tests with ``lit`` directly::
 
-     mkdir ../run_lnt_tests
-     cd ../run_lnt_tests
-     PATH=$LLVMBUILD/bin:$LNTINSTALL/bin:$PATH llvm-lit -sv ../lnt/tests
+    pip install lit
+    lit -sv ./tests
+
+However, that requires manually setting up the testing environment (``filecheck``, etc).
 
 For simple changes, adding a regression test and making sure all regression
 tests pass, is often a good enough testing approach. For some changes, the
@@ -73,4 +73,4 @@ default. You can enable them by passing additional flags to lit:
 
 Example::
 
-     PATH=$LLVMBUILD/bin:$LNTINSTALL/bin:$PATH llvm-lit -sv -Dpostgres=1 -Dmysql=1 -Dtidylib=1 ../lnt/tests
+    lit -sv -Dpostgres=1 -Dmysql=1 -Dtidylib=1 ./tests

tests/Formats/json.py

@@ -1,4 +1,4 @@
-# RUN: lnt convert --to=json %S/Inputs/test.json | FileCheck %s
-# RUN: lnt convert --to=json < %S/Inputs/test.json | FileCheck %s
+# RUN: lnt convert --to=json %S/Inputs/test.json | filecheck %s
+# RUN: lnt convert --to=json < %S/Inputs/test.json | filecheck %s
 
 # CHECK: {"a": 1}

tests/Formats/plist.py

@@ -1,4 +1,4 @@
-# RUN: lnt convert --to=json %S/Inputs/test.plist | FileCheck %s
-# RUN: lnt convert --to=json < %S/Inputs/test.plist | FileCheck %s
+# RUN: lnt convert --to=json %S/Inputs/test.plist | filecheck %s
+# RUN: lnt convert --to=json < %S/Inputs/test.plist | filecheck %s
 
 # CHECK: {"a": 1}

tests/lnttool/PostgresDB.shtest

@@ -22,7 +22,7 @@ lnt import "${TESTDIR}/instance" "${SHARED_INPUTS}/sample-b-small.plist" --show-
 #
 lnt updatedb "${TESTDIR}/instance" --testsuite nts --delete-run 1 \
 	--show-sql >& "${TESTDIR}/runrm.out"
-# RUN: FileCheck --check-prefix CHECK-RUNRM %s < "%t.install/runrm.out"
+# RUN: filecheck --check-prefix CHECK-RUNRM %s < "%t.install/runrm.out"
 
 # CHECK-RUNRM: DELETE FROM "NT_Sample" WHERE "NT_Sample"."ID" = %(ID)s
 # CHECK-RUNRM-NEXT: ({'ID': 1}, {'ID': 2})
@@ -38,7 +38,7 @@ lnt import "${TESTDIR}/instance" "${SHARED_INPUTS}/sample-a-small.plist" --show-
 lnt updatedb "${TESTDIR}/instance" --testsuite nts \
 	--delete-machine "LNT SAMPLE MACHINE" \
 	--show-sql >& "${TESTDIR}/machinerm.out"
-# RUN: FileCheck --check-prefix CHECK-MACHINERM %s < "%t.install/machinerm.out"
+# RUN: filecheck --check-prefix CHECK-MACHINERM %s < "%t.install/machinerm.out"
 
 # CHECK-MACHINERM: DELETE FROM "NT_Sample" WHERE "NT_Sample"."ID" = %(ID)s
 # CHECK-MACHINERM-NEXT: ({'ID': 3}, {'ID': 4}, {'ID': 5})

tests/lnttool/Profile.py

@@ -1,13 +1,13 @@
-# RUN: lnt profile getVersion %S/Inputs/test.lntprof | FileCheck --check-prefix=CHECK-GETVERSION %s
+# RUN: lnt profile getVersion %S/Inputs/test.lntprof | filecheck --check-prefix=CHECK-GETVERSION %s
 # CHECK-GETVERSION: 1
 
-# RUN: lnt profile getTopLevelCounters %S/Inputs/test.lntprof | FileCheck --check-prefix=CHECK-GETTLC %s
+# RUN: lnt profile getTopLevelCounters %S/Inputs/test.lntprof | filecheck --check-prefix=CHECK-GETTLC %s
 # CHECK-GETTLC: {"cycles": 12345.0, "branch-misses": 200.0}
 
-# RUN: lnt profile getFunctions --sortkeys %S/Inputs/test.lntprof | FileCheck --check-prefix=CHECK-GETFUNCTIONS %s
+# RUN: lnt profile getFunctions --sortkeys %S/Inputs/test.lntprof | filecheck --check-prefix=CHECK-GETFUNCTIONS %s
 # CHECK-GETFUNCTIONS: {"fn1": {"counters": {"branch-misses": 10.0, "cycles": 45.0}, "length": 2}}
 
-# RUN: lnt profile getCodeForFunction %S/Inputs/test.lntprof fn1 | FileCheck --check-prefix=CHECK-GETFN1 %s
+# RUN: lnt profile getCodeForFunction %S/Inputs/test.lntprof fn1 | filecheck --check-prefix=CHECK-GETFN1 %s
 # CHECK-GETFN1: [{}, 1048576, "add r0, r0, r0"], [{"cycles": 100.0}, 1048580, "sub r1, r0, r0"]]
 
 # RUN: mkdir -p %t

tests/lnttool/UpdateDB.py

@@ -9,7 +9,7 @@
 #
 # RUN: lnt updatedb %t.install --testsuite nts \
 # RUN:     --delete-run 1 --show-sql >& %t.out
-# RUN: FileCheck --check-prefix CHECK-RUNRM %s < %t.out
+# RUN: filecheck --check-prefix CHECK-RUNRM %s < %t.out
 
 # CHECK-RUNRM: DELETE FROM "NT_Sample" WHERE "NT_Sample"."ID" = ?
 # CHECK-RUNRM-NEXT: ((1,), (2,))
@@ -25,7 +25,7 @@
 # RUN:     --show-sample-count
 # RUN: lnt updatedb %t.install --testsuite nts \
 # RUN:     --delete-machine "LNT SAMPLE MACHINE" --show-sql >& %t.out
-# RUN: FileCheck --check-prefix CHECK-MACHINERM %s < %t.out
+# RUN: filecheck --check-prefix CHECK-MACHINERM %s < %t.out
 
 # CHECK-MACHINERM: DELETE FROM "NT_Sample" WHERE "NT_Sample"."ID" = ?
 # CHECK-MACHINERM-NEXT: ((1,), (2,))

tests/lnttool/admin.shtest

@@ -12,7 +12,7 @@ cd "$DIR"
 rm -rf lntadmin.yaml
 lnt admin create-config > create_config.stdout
 mv lntadmin.yaml create_config.txt
-# RUN: FileCheck %s --check-prefix=CREATE_CONFIG < %t.tmp/create_config.txt
+# RUN: filecheck %s --check-prefix=CREATE_CONFIG < %t.tmp/create_config.txt
 # CREATE_CONFIG: lnt_url: "http://localhost:8000"
 # CREATE_CONFIG-NEXT: database: default
 # CREATE_CONFIG-NEXT: testsuite: nts
@@ -28,11 +28,11 @@ auth_token: test_token
 __EOF__
 
 lnt admin post-run "${SHARED_INPUTS}/sample-a-small.plist" > post_run.stdout
-# RUN: FileCheck %s --check-prefix=POST_RN < %t.tmp/post_run.stdout
+# RUN: filecheck %s --check-prefix=POST_RN < %t.tmp/post_run.stdout
 # POST_RN: http://localhost:9092/api/db_default/v4/nts/runs/3
 
 lnt admin machine-info 1 > machine_info.stdout
-# RUN: FileCheck %s --check-prefix=MACHINE_INFO < %t.tmp/machine_info.stdout
+# RUN: filecheck %s --check-prefix=MACHINE_INFO < %t.tmp/machine_info.stdout
 # MACHINE_INFO: name: localhost__clang_DEV__x86_64
 # MACHINE_INFO: id: 1
 # MACHINE_INFO: hardware: x86_64
@@ -42,7 +42,7 @@ lnt admin machine-info 1 > machine_info.stdout
 
 rm -rf run_3.json
 lnt admin get-run 3 > get_run.stdout
-# RUN: FileCheck %s --check-prefix=GET_RN < %t.tmp/run_3.json
+# RUN: filecheck %s --check-prefix=GET_RN < %t.tmp/run_3.json
 # GET_RN: {
 # GET_RN:   "generated_by":
 # GET_RN:   "machine": {
@@ -60,18 +60,18 @@ lnt admin get-run 3 > get_run.stdout
 # GET_RN: }
 
 lnt admin list-machines > list_machines.stdout
-# RUN: FileCheck %s --check-prefix=LIST_MACHINES < %t.tmp/list_machines.stdout
+# RUN: filecheck %s --check-prefix=LIST_MACHINES < %t.tmp/list_machines.stdout
 # LIST_MACHINES: localhost__clang_DEV__x86_64:1
 # LIST_MACHINES-NEXT: LNT SAMPLE MACHINE:2
 
 lnt admin list-runs 1 > list_runs.stdout
-# RUN: FileCheck %s --check-prefix=LIST_RUNS < %t.tmp/list_runs.stdout
+# RUN: filecheck %s --check-prefix=LIST_RUNS < %t.tmp/list_runs.stdout
 # LIST_RUNS: llvm_project_revision=154331 1
 # LIST_RUNS: llvm_project_revision=152289 2
 
 rm -rf machine_1.json
 lnt admin get-machine 1 > get_machine.stdout
-# RUN: FileCheck %s --check-prefix=GET_MACHINE_JSON < %t.tmp/machine_1.json
+# RUN: filecheck %s --check-prefix=GET_MACHINE_JSON < %t.tmp/machine_1.json
 # GET_MACHINE_JSON: {
 # GET_MACHINE_JSON:   "generated_by":
 # GET_MACHINE_JSON:   "machine": {
@@ -88,20 +88,20 @@ lnt admin get-machine 1 > get_machine.stdout
 # GET_MACHINE_JSON: }
 
 lnt admin rm-machine 1 > rm_machine.stdout
-# RUN: FileCheck %s --check-prefix=RM_MACHINE < %t.tmp/rm_machine.stdout
+# RUN: filecheck %s --check-prefix=RM_MACHINE < %t.tmp/rm_machine.stdout
 # RM_MACHINE: Deleting runs 1 2 (2/2)
 # RM_MACHINE: Deleted machine localhost__clang_DEV__x86_64:1
 
 lnt admin list-machines > list_machines2.stdout
-# RUN: FileCheck %s --check-prefix=LIST_MACHINES2 < %t.tmp/list_machines2.stdout
+# RUN: filecheck %s --check-prefix=LIST_MACHINES2 < %t.tmp/list_machines2.stdout
 # LIST_MACHINES2-NOT: localhost__clang_DEV__x86_64:1
 # LIST_MACHINES2: LNT SAMPLE MACHINE:2
 
 lnt admin rename-machine 2 hal9000
 # No output
 
 lnt admin list-machines > list_machines3.stdout
-# RUN: FileCheck %s --check-prefix=LIST_MACHINES3 < %t.tmp/list_machines3.stdout
+# RUN: filecheck %s --check-prefix=LIST_MACHINES3 < %t.tmp/list_machines3.stdout
 # LIST_MACHINES3: hal9000:2
 
 # Just post again so we have a machine to merge

tests/lnttool/checkformat.shtest

@@ -1,5 +1,5 @@
 # Check an old plist format
-# RUN: lnt checkformat %{shared_inputs}/sample-a-small.plist 2>&1 | FileCheck %s --check-prefix=CHECK0
+# RUN: lnt checkformat %{shared_inputs}/sample-a-small.plist 2>&1 | filecheck %s --check-prefix=CHECK0
 #
 # CHECK0: Import succeeded.
 # CHECK0: Imported Data
@@ -14,7 +14,7 @@
 #
 #
 # Check an old json format
-# RUN: lnt checkformat %{shared_inputs}/sample-report.json 2>&1 | FileCheck %s --check-prefix=CHECK1
+# RUN: lnt checkformat %{shared_inputs}/sample-report.json 2>&1 | filecheck %s --check-prefix=CHECK1
 #
 # CHECK1: Import succeeded.
 # CHECK1: Imported Data
@@ -29,7 +29,7 @@
 #
 #
 # Check new-style json format
-# RUN: lnt checkformat %{src_root}/docs/report-example.json 2>&1 | FileCheck %s --check-prefix=CHECK2
+# RUN: lnt checkformat %{src_root}/docs/report-example.json 2>&1 | filecheck %s --check-prefix=CHECK2
 #
 # CHECK2: Import succeeded.
 # CHECK2: Imported Data
@@ -43,14 +43,14 @@
 # CHECK2: PASS : 10
 #
 #
-# RUN: lnt checkformat %S/Inputs/minimal.json 2>&1 | FileCheck %s --check-prefix=MINIMAL
+# RUN: lnt checkformat %S/Inputs/minimal.json 2>&1 | filecheck %s --check-prefix=MINIMAL
 # MINIMAL: Import succeeded.
 # MINIMAL: Added Machines: 1
 # MINIMAL: Added Runs    : 1
 #
 #
 # Check invalid format
-# RUN: lnt checkformat %S/Inputs/invalid_submission0.json 2>&1 | FileCheck %s --check-prefix=CHECKFAIL0
+# RUN: lnt checkformat %S/Inputs/invalid_submission0.json 2>&1 | filecheck %s --check-prefix=CHECKFAIL0
 #
 # CHECKFAIL0: Import Failed:
 # CHECKFAIL0: could not parse input format

tests/lnttool/email_tools.py

@@ -9,7 +9,7 @@
 
 # RUN: lnt send-run-comparison --dry-run --to [email protected] \
 # RUN: --from [email protected]  \
-# RUN: --host localhost %t.instance 1 2 | FileCheck %s --check-prefix CHECK0
+# RUN: --host localhost %t.instance 1 2 | filecheck %s --check-prefix CHECK0
 #
 # CHECK0: From: [email protected]
 # CHECK0: To: [email protected]
@@ -37,7 +37,7 @@
 
 # RUN: lnt send-daily-report --dry-run --from [email protected] \
 # RUN: --host localhost --testsuite nts --filter-machine-regex=machine.? \
-# RUN: %t.instance [email protected] | FileCheck %s --check-prefix CHECK1
+# RUN: %t.instance [email protected] | filecheck %s --check-prefix CHECK1
 #
 # CHECK1: From: [email protected]
 # CHECK1: To: [email protected]