tools: Install tools as part of this python package

This makes installing easier as we don't have to checkout the repository
and rely on running pip install -r requirements_tools.

These scripts will be used by the datastore after running the
datagetter and take over from the datatester.

Author: michaelwood

README.md

@@ -25,23 +25,6 @@ $ ./cove/manage.py runserver
 
 CoVE is based on Django and be deployed using the deployment mechanisms as defined in the [Django docs](https://docs.djangoproject.com/en/3.1/howto/deployment/)
 
-## Tools
-
-The tools directory contains a number of command line tools that can be run on TheeSixtyGiving standard data to generate various reports.
-
-### Installing and running tools
-
-Install the python dependences:
-```
-$ virtualenv .ve --python python3
-$ source .ve/bin/activate
-$ pip install -r requirements_tools.txt
-```
-
-Running various tools example:
-```
-python ./tools/cove_checks.py ../a001p00000tuoBKAAY.ods
-```
 
 ## lib360dataquality
 
@@ -59,6 +42,15 @@ $ source .ve/bin/activate
 $ pip install -e .
 ```
 
+### Using the tools
+
+The tools directory contains a number of command line tools that can be run on TheeSixtyGiving standard data to generate various reports.
+
+Running various tools example:
+```
+cove_checks.py ../a001p00000tuoBKAAY.ods
+```
+
 ### Using the python library
 
 To use the python library import it into your python program, for example to use the additional checks:
@@ -75,4 +67,4 @@ data = '' # Your JSON data
 common_checks_360(context, working_dir, data, Schema360())
 
 print(context)
-```
+```

requirements_tools.in

@@ -14,6 +14,7 @@
         'libcove>=0.18.0',
         'python-dateutil',
         'rangedict',
+        'ijson',
     ],
     extras_require={
         'perf': [
@@ -26,6 +27,15 @@
             'isort',
         ],
     },
+    scripts=[
+        "tools/aggregates.py",
+        "tools/check_grantnav_assumptions.py",
+        "tools/cove_checks.py",
+        "tools/coverage.py",
+        "tools/report.py",
+        "tools/generate-reports.sh",
+        "tools/deploy.sh",
+    ],
     classifiers=[
         'License :: OSI Approved :: GNU Affero General Public License v3 or later (AGPLv3+)',
         'Programming Language :: Python :: 3.6',

requirements_tools.txt

@@ -1,3 +1,4 @@
+#!/usr/bin/env python3
 import json
 from lib360dataquality.cove.threesixtygiving import get_grants_aggregates
 

setup.py

@@ -1,3 +1,4 @@
+#!/usr/bin/env python3
 import ijson
 import json
 import os

tools/aggregates.py

@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+
 import json
 from lib360dataquality.cove.schema import Schema360
 from lib360dataquality.coverage import get_unique_fields_present

tools/check_grantnav_assumptions.py

@@ -1,3 +1,5 @@
+#!/usr/bin/env python3
+
 import csv
 import json
 from collections import defaultdict