[Testing:Plagiarism] basic tokenizer tests (#40)

* Initial rewrite of process_all.sh

* Update process_all.sh

* Make modifications to file paths and add timers

* Overhaul concatenate_all.py

* Fix python errors

* Progress: everything through tokenization finished

* Everything works

* Add timers

* remove unnecessary code

* little python changes

* William made an oopsie (forgot to deal with provided code)

* Fix minor bugs

Fix process_all.sh script plus fix spelling issue and prevent hash_all.py from breaking when empty tokenized files are written

* Fix permissions issue with provided code editing

* Add initial script

* Update lichen_run.yml

* Update lichen_run.yml

* Update lichen_run.yml

* Update lichen_run.yml

* Update lichen_run.yml

* add boost

* add testing file

* forgot that paths are important

* Make separate setup.sh script

* Update lichen_run.yml

* Adjust file structure, add setup script

* need sudo for test

* Update tests.py

* fix path

* fix path

* Update tests.py

* add assertion to implement test

* fix more paths

* fix another path issue

* Add second test

* Update tests.py

* it's important to run the right command to get the right results...

* Add third test

* Add remaining plaintext tokenizer tests

* Add C tokenizer tests

* Update lichen_run.yml

* Update lichen_run.yml

* Update lichen_run.yml

* Add MIPS tokenizer

* Update tests.py

* Update tests.py

* Fix paths in tests.py such that it can be run in vagrant

* Fix github actions

* Remove old code

Co-authored-by: sbelsk <[email protected]>

Author: williamjallen

.flake8

@@ -5,3 +5,4 @@ exclude=
 
 per-file-ignores =
     tokenizer/mips/mips_tokenizer.py:W605
+    tests/tests.py:E501

.github/workflows/lichen_run.yml

@@ -0,0 +1,22 @@
+name: Test Lichen
+
+on: [push]
+
+jobs:
+  test-lichen:
+    runs-on: ubuntu-20.04
+    steps:
+      - uses: actions/checkout@v2
+      - uses: actions/setup-python@v2
+        with:
+          python-version: '3.6'
+      - name: Install Dependencies
+        run: |
+          sudo apt install libboost-all-dev
+      - name: Create Directory Structure
+        run: |
+          sudo bash ./tests/setup.sh
+      - name: Run Tests
+        run: |
+          cd /usr/local/submitty/GIT_CHECKOUT/Lichen/tests
+          sudo python3 -m unittest discover

.gitignore

@@ -1,2 +1,3 @@
 *~
 tools/assignments/*
+tests/__pycache__

tokenizer/c/expected_output/output.json renamed to tests/data/tokenizer/c/expected_output/output.json

@@ -51,7 +51,7 @@
         "char": 20,
         "line": 3,
         "type": "number",
-        "value": "1"
+        "value": 1
     },
     {
         "char": 22,
@@ -159,7 +159,7 @@
         "char": 26,
         "line": 4,
         "type": "number",
-        "value": "1"
+        "value": 1
     },
     {
         "char": 27,
@@ -171,7 +171,7 @@
         "char": 28,
         "line": 4,
         "type": "number",
-        "value": "2"
+        "value": 2
     },
     {
         "char": 29,

tokenizer/c/input.cpp renamed to tests/data/tokenizer/c/input.cpp

@@ -39,7 +39,7 @@
         "char": 20,
         "line": 3,
         "type": "number",
-        "value": "1"
+        "value": 1
     },
     {
         "char": 22,
@@ -141,7 +141,7 @@
         "char": 26,
         "line": 4,
         "type": "number",
-        "value": "1"
+        "value": 1
     },
     {
         "char": 27,
@@ -153,7 +153,7 @@
         "char": 28,
         "line": 4,
         "type": "number",
-        "value": "2"
+        "value": 2
     },
     {
         "char": 29,