22
33This document describes the manual test plan for the QL extension for Visual Studio Code.
44
5- The plan will be executed manually to start with but the goal is to eventually automate parts of the process (based on
5+ The plan will be executed manually to start with but the goal is to eventually automate parts of the process (based on
66effort vs value basis).
77
8- #### What this doesn't cover
8+ ## What this doesn't cover
9+
910We don't need to test features (and permutations of features) that are covered by automated tests.
1011
11- ### Before releasing the VS Code extension
12+ ## Before releasing the VS Code extension
13+
1214- Go through the required test cases listed below
13- - Check major PRs since the previous release for specific one-off things to test. Based on that, you might want to
15+ - Check major PRs since the previous release for specific one-off things to test. Based on that, you might want to
1416choose to go through some of the Optional Test Cases.
1517- Run a query using the existing version of the extension (to generate an "old" query history item)
1618
@@ -24,23 +26,25 @@ choose to go through some of the Optional Test Cases.
2426
25271 . Open the [ UnsafeJQueryPlugin query] ( https://github.com/github/codeql/blob/main/javascript/ql/src/Security/CWE-079/UnsafeJQueryPlugin.ql ) .
26282 . Run a MRVA against the following repo list:
27- ```
28- {
29- "name": "test-repo-list",
30- "repositories": [
31- "angular-cn/ng-nice",
32- "apache/hadoop",
33- "apache/hive"
34- ]
35- }
36- ```
29+
30+ ``` json
31+ {
32+ "name" : " test-repo-list" ,
33+ "repositories" : [
34+ " angular-cn/ng-nice" ,
35+ " apache/hadoop" ,
36+ " apache/hive"
37+ ]
38+ }
39+ ```
40+
37413 . Check that a notification message pops up and the results view is opened.
38424 . Check the query history. It should:
3943 - Show that an item has been added to the query history
4044 - The item should be marked as "in progress".
41455 . Once the query starts:
42- - Check the results view
43- - Check the code paths view, including the code paths drop down menu.
46+ - Check the results view
47+ - Check the code paths view, including the code paths drop down menu.
4448 - Check that the repository filter box works
4549 - Click links to files/locations on GitHub
4650 - Check that the query history item is updated to show the number of results
@@ -74,7 +78,7 @@ choose to go through some of the Optional Test Cases.
74781 . Click a history item (for MRVA):
7579 - Check that exporting results works
7680 - Check that sorting results works
77- - Check that copying repo lists works
81+ - Check that copying repo lists works
78822 . Open the query results directory:
7983 - Check that the correct directory is opened and there are results in it
80843 . View logs
@@ -84,12 +88,12 @@ choose to go through some of the Optional Test Cases.
8488
8589Run one of the above MRVAs, but cancel it from within VS Code:
8690- Check that the query is canceled and the query history item is updated.
87- - Check that the workflow run is also canceled.
91+ - Check that the workflow run is also canceled.
8892- Check that any available results are visible in VS Code.
8993
90- ### Test Case 6: MRVA - Change to a different colour theme
94+ ### Test Case 6: MRVA - Change to a different colour theme
9195
92- Open one of the above MRVAs, try changing to a different colour theme and check that everything looks sensible.
96+ Open one of the above MRVAs, try changing to a different colour theme and check that everything looks sensible.
9397Are there any components that are not showing up?
9498
9599## Optional Test Cases
@@ -99,24 +103,27 @@ These are mostly aimed at MRVA, but some of them are also applicable to non-MRVA
99103### Selecting repositories to run on
100104
101105#### Test case 1: Running a query on a single repository
102- 1 . When the repository exists and is public
103- 1 . Has a CodeQL database for the correct language
104- 2 . Has a CodeQL database for another language
106+
107+ 1 . When the repository exists and is public
108+ 1 . Has a CodeQL database for the correct language
109+ 2 . Has a CodeQL database for another language
105110 3 . Does not have any CodeQL databases
1061112 . When the repository exists and is private
107112 1 . Is accessible and has a CodeQL database
108113 2 . Is not accessible
1091143 . When the repository does not exist
110115
111116#### Test case 2: Running a query on a custom repository list
117+
1121181 . The repository list is non-empty
113- 1 . All repositories in the list have a CodeQL database
119+ 1 . All repositories in the list have a CodeQL database
114120 2 . Some but not all repositories in the list have a CodeQL database
115121 3 . No repositories in the list have a CodeQL database
1161222 . The repository list is empty
117123
118124#### Test case 3: Running a query on all repositories in an organization
119- 1 . The org exists
125+
126+ 1 . The org exists
120127 1 . The org contains repositories that have CodeQL databases
121128 2 . The org contains repositories of the right language but without CodeQL databases
122129 3 . The org contains repositories not of the right language
@@ -126,20 +133,25 @@ These are mostly aimed at MRVA, but some of them are also applicable to non-MRVA
126133### Using different types of controller repos
127134
128135#### Test case 1: Running a query when the controller repository is public
136+
1291371 . Can run queries on public repositories
1301382 . Can not run queries on private repositories
131139
132140#### Test case 2: Running a query when the controller repository is private
141+
1331421 . Can run queries on public repositories
1341432 . Can run queries on private repositories
135144
136145#### Test case 3: Running a query when the controller repo exists but you do not have write access
146+
1371471 . Cannot run queries
138148
139149#### Test case 4: Running a query when the controller repo doesn’t exist
150+
1401511 . Cannot run queries
141152
142153#### Test case 5: Running a query when the "config field" for the controller repo is not set
154+
1431551 . Cannot run queries
144156
145157### Query History
@@ -150,6 +162,7 @@ The first test case specifies actions that you can do when the query is first ru
150162with this since it has quite a limited number of actions you can do.
151163
152164#### Test case 1: When variant analysis state is "pending"
165+
1531661 . Starts monitoring variant analysis
1541672 . Cannot open query history item
1551683 . Can delete a query history item
@@ -160,8 +173,8 @@ with this since it has quite a limited number of actions you can do.
160173 2 . By query date
161174 3 . By result count
1621755 . Cannot open query directory
163- 6 . Can open query that produced these results
164- 1 . When the file still exists and has not moved
176+ 6 . Can open query that produced these results
177+ 1 . When the file still exists and has not moved
165178 2 . When the file does not exist
1661797 . Cannot view logs
1671808 . Cannot copy repository list
@@ -171,6 +184,7 @@ with this since it has quite a limited number of actions you can do.
17118412 . Cannot cancel analysis
172185
173186#### Test case 2: When the variant analysis state is not "pending"
187+
1741881 . Query history is loaded when VSCode starts
1751892 . Handles when action workflow was canceled while VSCode was closed
1761903 . Can open query history item
@@ -204,12 +218,14 @@ with this since it has quite a limited number of actions you can do.
204218 4 . A popup allows you to open the directory
205219
206220#### Test case 3: When variant analysis state is "in_progress"
221+
2072221 . Starts monitoring variant analysis
208- 1 . Ready results are downloaded
209- 2 . Can cancel analysis
223+ 1 . Ready results are downloaded
224+ 2 . Can cancel analysis
210225 1 . Causes the actions run to be canceled
211226
212227#### Test case 4: When variant analysis state is in final state ("succeeded"/"failed"/"canceled")
228+
2132291 . Stops monitoring variant analysis
214230 1 . All results are downloaded if state is succeeded
215231 2 . Otherwise, ready results are downloaded, if any are available
@@ -220,16 +236,18 @@ with this since it has quite a limited number of actions you can do.
220236This requires running a MRVA query and seeing the results view.
221237
222238#### Test case 1: When variant analysis state is "pending"
239+
2232401 . Can open a results view
2242412 . Results view opens automatically
225242 - When starting variant analysis run
226243 - When VSCode opens (if view was open when VSCode was closed)
2272443 . Results view is empty
228245
229246#### Test case 2: When variant analysis state is not "pending"
247+
2302481 . Can open a results view
2312492 . Results view opens automatically
232- 1 . When starting variant analysis run
250+ 1 . When starting variant analysis run
233251 2 . When VSCode opens (if view was open when VSCode was closed)
2342523 . Can copy repository list
235253 1 . Text is copied to clipboard
@@ -240,43 +258,45 @@ This requires running a MRVA query and seeing the results view.
2402586 . Can open query file
241259 1 . When the file still exists and has not moved
242260 2 . When the file does not exist
243- 7 . Can open query text
244- 8 . Can sort repos
245- 1 . By name
246- 2 . By results
247- 3 . By stars
261+ 7 . Can open query text
262+ 8 . Can sort repos
263+ 1 . By name
264+ 2 . By results
265+ 3 . By stars
248266 4 . By last updated
2492679 . Can filter repos
250- 10 . Shows correct statistics
251- 1 . Total number of results
252- 2 . Total number of repositories
268+ 10 . Shows correct statistics
269+ 1 . Total number of results
270+ 2 . Total number of repositories
253271 3 . Duration
254- 11 . Can see live results
272+ 11 . Can see live results
255273 1 . Results appear in extension as soon as each query is completed
25627412 . Can view interpreted results (i.e. for a "problem" query)
257- 1 . Can view non-path results
275+ 1 . Can view non-path results
258276 2 . Can view code paths for "path-problem" queries
25927713 . Can view raw results (i.e. for a non "problem" query)
260278 1 . Renders a table
261- 14 . Can see skipped repositories
262- 1 . Can see repos with no db in a tab
263- 1 . Shown warning that explains the tab
279+ 14 . Can see skipped repositories
280+ 1 . Can see repos with no db in a tab
281+ 1 . Shown warning that explains the tab
264282 2 . Can see repos with no access in a tab
265- 1 . Shown warning that explains the tab
283+ 1 . Shown warning that explains the tab
266284 3 . Only shows tab when there are skipped repos
267- 15 . Result downloads
268- 1 . All results are downloaded automatically
285+ 15 . Result downloads
286+ 1 . All results are downloaded automatically
269287 2 . Download status is indicated by a spinner (Not currently any indication of progress beyond "downloading" and "not downloading")
270- 3 . Only 3 items are downloaded at a time
271- 4 . Results for completed queries are still downloaded when
272- 1 . Some but not all queries failed
288+ 3 . Only 3 items are downloaded at a time
289+ 4 . Results for completed queries are still downloaded when
290+ 1 . Some but not all queries failed
273291 2 . The variant analysis was canceled after some queries completed
274292
275293#### Test case 3: When variant analysis state is in "succeeded" state
294+
2762951 . Can view logs
277- 2 . All results are downloaded
296+ 2 . All results are downloaded
278297
279298#### Test case 4: When variant analysis is in "failed" or "canceled" state
299+
2803001 . Can view logs
2813011 . Results for finished queries are still downloaded.
282302
@@ -305,14 +325,17 @@ This requires running a MRVA query and seeing the results view.
3053251 . Collapse/expand tree nodes
306326
307327Error cases that trigger an error notification:
308- 1 . Try to add a list with a name that already exists
328+
329+ 1 . Try to add a list with a name that already exists
3093301 . Try to add a top-level database that already exists
3103311 . Try to add a database in a list that already exists in the list
311332
312333Error cases that show an error in the panel (and only the edit button should be visible):
334+
3133351 . Edit the db config file directly and save invalid JSON
3143361 . Edit the db config file directly and save valid JSON but invalid config (e.g. add an unknown property)
315- 1 . Edit the db config file directly and save two lists with the same name
337+ 1 . Edit the db config file directly and save two lists with the same name
316338
317339Cases where there the welcome view is shown:
318- 1 . No controller repo is set in the user's settings JSON.
340+
341+ 1 . No controller repo is set in the user's settings JSON.
0 commit comments