Insert code into db

Author: colinmcneil

prompts/pylint/3-4-json-violations->sql-violations.md renamed to prompts/pylint/3-generate-violation-insert.md

@@ -8,9 +8,6 @@ tools:
         database:
           type: string
           description: the path to the database
-        sql:
-          type: string
-          description: the sql statement to run
     container:
       image: vonwig/sqlite:latest
       command:
@@ -30,40 +27,40 @@ docker run -it --rm -v thread:/thread \
 ```
 docker run -it --rm -v thread:/thread \
            vonwig/sqlite:latest \
-           /thread/db.dqlite \
+           /thread/db.sqlite \
            ".schema RANGES"
 ```
 
 ```
 docker run -it --rm -v thread:/thread \
            vonwig/sqlite:latest \
-           /thread/db.dqlite \
+           /thread/db.sqlite \
            ".schema VIOLATIONS"
 ```
 
 ```
 docker run -it --rm -v thread:/thread \
            vonwig/sqlite:latest \
-           /thread/db.dqlite \
+           /thread/db.sqlite \
            ".open /thread/insert.sql"
 ```
 
 ```
 docker run -it --rm -v thread:/thread \
            vonwig/sqlite:latest \
-           /thread/db.dqlite \
+           /thread/db.sqlite \
            "SELECT * FROM VIOLATIONS INNER JOIN RANGES ON VIOLATIONS.RANGE = RANGES.ID WHERE RANGES.PATH = 'src/app.py'"
 ```
 
 ```
 docker run -it --rm -v thread:/thread \
            vonwig/sqlite:latest \
-           /thread/db.dqlite \
+           /thread/db.sqlite \
            "SELECT * FROM RANGES WHERE RANGES.PATH = 'src/app.py'"
 ```
 
 
 # prompt user
 
-Run the sqlite command with the path to the database set to `/thread/db.dqlite`.
+Run the sqlite command with the path to the database set to `/thread/db.sqlite`.
 

prompts/pylint/3-5-violations-sql-run.md renamed to prompts/pylint/4-run-violation-insert.md

@@ -118,16 +118,16 @@ An example parsed file:
   ``` 
   to get the ranges of each top level node into capturing groups.
   Write the ranges to `/thread/ranges.sql` in the form of sql 
-  `INSERT INTO RANGES (LINE, END_LINE, COLUMN, END_COLUMN, PATH) values (...)`
+  `INSERT INTO RANGES (START_LINE, END_LINE, START_COLUMN, END_COLUMN, PATH) values (...)`
 
 Example output of script would be the creation of file `/thread/ranges.sql` with contents:
 
 ```sql
-INSERT INTO RANGES (LINE, COLUMN, END_LINE, END_COLUMN, NODE_TYPE, PATH) values (0,0,0,21,'comment', 'main.py')
-INSERT INTO RANGES (LINE, COLUMN, END_LINE, END_COLUMN, NODE_TYPE, PATH) values (2,0,2,37,'import_from_statement', 'main.py')
-INSERT INTO RANGES (LINE, COLUMN, END_LINE, END_COLUMN, NODE_TYPE, PATH) values (4,0,5,17,'if_statement', 'main.py')
+INSERT INTO RANGES (START_LINE, START_COLUMN, END_LINE, END_COLUMN, PATH) values (0,0,0,21, 'main.py')
+INSERT INTO RANGES (START_LINE, START_COLUMN, END_LINE, END_COLUMN, PATH) values (2,0,2,37, 'main.py')
+INSERT INTO RANGES (START_LINE, START_COLUMN, END_LINE, END_COLUMN, PATH) values (4,0,5,17, 'main.py')
 ```
 
 5. Execute the code in the javascript sandbox.
 
-6. Finally, run `.read /thread/ranges.sql` in db.sqlite
+6. Finally, run `.read /thread/ranges.sql` in `/thread/db.sqlite`

prompts/pylint/5-6-top-level-ranges.md renamed to prompts/pylint/5-1-insert-top-level-ranges.md

@@ -0,0 +1,80 @@
+---
+tools:
+  - name: sqlite
+    description: run the sqlite command
+    parameters:
+      type: object
+      properties:
+        database:
+          type: string
+          description: the path to the database
+        sql:
+          type: string
+          description: the sql statement to run
+    container:
+      image: vonwig/sqlite:latest
+      command:
+        - "{{database}}"
+        - "{{sql|safe}}"
+  - name: run-javascript-sandbox
+    description: execute javascript code in node
+    parameters:
+      type: object
+      properties:
+        javascript:
+          type: string
+          description: the javascript code to run
+    container:
+      image: vonwig/javascript-runner
+      command:
+        - "{{javascript|safe}}"
+---
+
+# prompt user
+
+We are using database `/thread/db.sqlite`.
+
+0. Use JS sandbox to delete folder `/thread/code_insert` and create it again
+
+1. Drop and re-create the `CODE` table with the following schema:
+```
+(ID,RANGE_ID,CODE_AT_RANGE)
+```
+
+where RANGE_ID is a foreign key to the `RANGES` table.
+ID should just be a regular autoincrementing primary key
+CODE_AT_RANGE is text.
+
+2. Prepare a script similar to the following:
+```
+.mode csv
+.headers on
+.once /thread/code_insert/all_ranges.csv
+<Query>
+``` 
+where query selects everything from the RANGES table where END_LINE is not null. Use the javascript sandbox with fs to write this script to `/thread/code_insert/query_ranges.sql`.
+
+3. Execute the script against the DB: `.read /thread/code_insert/query_ranges.sql`
+
+4. Now that we have `/thread/code_insert/all_ranges.csv`,
+
+we get something like
+```csv
+ID,PATH,START_LINE,END_LINE,START_COLUMN,END_COLUMN
+125,wsgi_skylines.py,11,104,0,61
+```
+
+So, write a JS script to 
+- Read the CSV file manually (no modules) and iterate over the rows
+- Read the contents at PATH with fs and slice out the content between `START_LINE` and `END_LINE`
+- Write a line of SQL to populate the row in the `CODE` table. This INSERT statement should then be appended to `/thread/code_insert/insert_code.sql`.
+
+The RANGE_ID should be the first value in the CSV row
+The CODE_AT_RANGE is the sliced content at PATH
+
+Example `/thread/code_insert/insert_code.sql` file:
+
+```sql
+INSERT INTO CODE (RANGE_ID, CODE_AT_RANGE) (125, "<Contents of wsgi_skylines from line 11 to 104>")
+```
+

prompts/pylint/5-2-insert-code.md

@@ -20,8 +20,4 @@ tools:
 
 # prompt user
 
-1. Run `.read /thread/insert.sql` against the database `/thread/db.sqlite`.
-
-2. After executing the SQL, count the number of rows in the `VIOLATIONS` and `RANGES` tables:
-   - `SELECT count(*) FROM VIOLATIONS;`
-   - `SELECT count(*) FROM RANGES;`
+How many rows are in the ranges and violations tables? Use database `/thread/db.sqlite`