Merge branch 'master' into remove-field-conflation-from-ir-fieldflow

Author: MathiasVP

README.md

@@ -1,6 +1,6 @@
 # CodeQL
 
-This open source repository contains the standard CodeQL libraries and queries that power [LGTM](https://lgtm.com) and the other CodeQL products that [GitHub](https://github.com) makes available to its customers worldwide.
+This open source repository contains the standard CodeQL libraries and queries that power [LGTM](https://lgtm.com) and the other CodeQL products that [GitHub](https://github.com) makes available to its customers worldwide. For the queries, libraries, and extractor that power Go analysis, visit the [CodeQL for Go repository](https://github.com/github/codeql-go).
 
 ## How do I learn CodeQL and run queries?
 

change-notes/1.25/analysis-cpp.md

@@ -16,6 +16,7 @@ The following changes in version 1.25 affect C/C++ analysis in all applications.
 
 ## Changes to libraries
 
+* The library `VCS.qll` and all queries that imported it have been removed.
 * The data-flow library has been improved, which affects most security queries by potentially
   adding more results. Flow through functions now takes nested field reads/writes into account.
   For example, the library is able to track flow from `taint()` to `sink()` via the method
@@ -39,3 +40,5 @@ The following changes in version 1.25 affect C/C++ analysis in all applications.
       }
   };
   ```
+* The security pack taint tracking library (`semmle.code.cpp.security.TaintTracking`) now considers that equality checks may block the flow of taint.  This results in fewer false positive results from queries that use this library.
+

change-notes/1.25/analysis-javascript.md

@@ -3,7 +3,10 @@
 ## General improvements
 
 * Support for the following frameworks and libraries has been improved:
+  - [Promise](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Promise)
+  - [bluebird](http://bluebirdjs.com/)
   - [express](https://www.npmjs.com/package/express)
+  - [fastify](https://www.npmjs.com/package/fastify)
   - [fstream](https://www.npmjs.com/package/fstream)
   - [jGrowl](https://github.com/stanlemon/jGrowl)
   - [jQuery](https://jquery.com/)
@@ -15,8 +18,10 @@
   - [sequelize](https://www.npmjs.com/package/sequelize)
   - [spanner](https://www.npmjs.com/package/spanner)
   - [sqlite](https://www.npmjs.com/package/sqlite)
-  - [ssh2](https://www.npmjs.com/package/ssh2)
   - [ssh2-streams](https://www.npmjs.com/package/ssh2-streams)
+  - [ssh2](https://www.npmjs.com/package/ssh2)
+
+* TypeScript 3.9 is now supported.
 
 ## New queries
 
@@ -25,48 +30,56 @@
 | Cross-site scripting through DOM (`js/xss-through-dom`) | security, external/cwe/cwe-079, external/cwe/cwe-116 | Highlights potential XSS vulnerabilities where existing text from the DOM is used as HTML. Results are not shown on LGTM by default. |
 | Incomplete HTML attribute sanitization (`js/incomplete-html-attribute-sanitization`) | security, external/cwe/cwe-20, external/cwe/cwe-079, external/cwe/cwe-116 | Highlights potential XSS vulnerabilities due to incomplete sanitization of HTML meta-characters. Results are shown on LGTM by default. |
 | Unsafe expansion of self-closing HTML tag (`js/unsafe-html-expansion`) | security, external/cwe/cwe-079, external/cwe/cwe-116 | Highlights potential XSS vulnerabilities caused by unsafe expansion of self-closing HTML tags. |
+| Unsafe shell command constructed from library input (`js/shell-command-constructed-from-input`) | correctness, security, external/cwe/cwe-078, external/cwe/cwe-088 | Highlights potential command injections due to a shell command being constructed from library inputs. Results are shown on LGTM by default. |
 
 ## Changes to existing queries
 
 | **Query**                      | **Expected impact**          | **Change**                                                                |
 |--------------------------------|------------------------------|---------------------------------------------------------------------------|
-| Misspelled variable name (`js/misspelled-variable-name`) | Message changed | The message for this query now correctly identifies the misspelled variable in additional cases. |
-| Uncontrolled data used in path expression (`js/path-injection`) | More results | This query now recognizes additional file system calls. |
-| Uncontrolled command line (`js/command-line-injection`) | More results | This query now recognizes additional command execution calls. |
-| Client-side URL redirect (`js/client-side-unvalidated-url-redirection`) | Less results | This query now recognizes additional safe patterns of doing URL redirects. |
-| Client-side cross-site scripting (`js/xss`) | Less results | This query now recognizes additional safe strings based on URLs. |
+| Client-side cross-site scripting (`js/xss`) | Fewer results | This query no longer flags optionally sanitized values. |
+| Client-side URL redirect (`js/client-side-unvalidated-url-redirection`) | Fewer results | This query now recognizes additional safe patterns of doing URL redirects. |
+| Client-side cross-site scripting (`js/xss`) | Fewer results | This query now recognizes additional safe patterns of constructing HTML. |
+| Code injection (`js/code-injection`) | More results | More potential vulnerabilities involving NoSQL code operators are now recognized. |
+| Expression has no effect (`js/useless-expression`) | Fewer results | This query no longer flags an expression when that expression is the only content of the containing file. |
 | Incomplete URL scheme check (`js/incomplete-url-scheme-check`) | More results | This query now recognizes additional url scheme checks. |
+| Misspelled variable name (`js/misspelled-variable-name`) | Message changed | The message for this query now correctly identifies the misspelled variable in additional cases. |
 | Prototype pollution in utility function (`js/prototype-pollution-utility`) | More results | This query now recognizes additional utility functions as vulnerable to prototype polution. |
-| Expression has no effect (`js/useless-expression`) | Less results | This query no longer flags an expression when that expression is the only content of the containing file. |
-| Unknown directive (`js/unknown-directive`) | Less results | This query no longer flags directives generated by the Babel compiler. |
-| Code injection (`js/code-injection`) | More results | More potential vulnerabilities involving NoSQL code operators are now recognized. |
+| Prototype pollution in utility function (`js/prototype-pollution-utility`) | More results | This query now recognizes more coding patterns that are vulnerable to prototype pollution. |
+| Uncontrolled command line (`js/command-line-injection`) | More results | This query now recognizes additional command execution calls. |
+| Uncontrolled data used in path expression (`js/path-injection`) | More results | This query now recognizes additional file system calls. |
+| Unknown directive (`js/unknown-directive`) | Fewer results | This query no longer flags directives generated by the Babel compiler. |
+| Unused property (`js/unused-property`) | Fewer results | This query no longer flags properties of objects that are operands of `yield` expressions. |
 | Zip Slip (`js/zipslip`) | More results | This query now recognizes additional vulnerabilities. |
-| Unused property (`js/unused-property`) | Less results | This query no longer flags properties of objects that are operands of `yield` expressions. |
 
 The following low-precision queries are no longer run by default on LGTM (their results already were not displayed):
 
   - `js/angular/dead-event-listener`
   - `js/angular/unused-dependency`
-  - `js/conflicting-html-attribute`
-  - `js/useless-assignment-to-global`
-  - `js/too-many-parameters`
-  - `js/unused-property`
   - `js/bitwise-sign-check`
   - `js/comparison-of-identical-expressions`
-  - `js/misspelled-identifier`
+  - `js/conflicting-html-attribute`
+  - `js/ignored-setter-parameter`
   - `js/jsdoc/malformed-param-tag`
-  - `js/jsdoc/unknown-parameter`
   - `js/jsdoc/missing-parameter`
-  - `js/omitted-array-element`
-  - `js/ignored-setter-parameter`
+  - `js/jsdoc/unknown-parameter`
   - `js/json-in-javascript-file`
+  - `js/misspelled-identifier`
+  - `js/nested-loops-with-same-variable`
   - `js/node/cyclic-import`
   - `js/node/unused-npm-dependency`
-  - `js/single-run-loop`
-  - `js/nested-loops-with-same-variable`
+  - `js/omitted-array-element`
   - `js/return-outside-function`
+  - `js/single-run-loop`
+  - `js/too-many-parameters`
+  - `js/unused-property`
+  - `js/useless-assignment-to-global`
 
 ## Changes to libraries
 
 * A library `semmle.javascript.explore.CallGraph` has been added to help write queries for exploring the call graph.
 * Added data flow for `Map` and `Set`, and added matching type-tracking steps that can accessed using the `CollectionsTypeTracking` module.
+* The data-flow node representing a parameter or destructuring pattern is now always the `ValueNode` corresponding to that AST node. This has a few consequences:
+  - `Parameter.flow()` now gets the correct data flow node for a parameter. Previously this had a result, but the node was disconnected from the data flow graph.
+  - `ParameterNode.asExpr()` and `.getAstNode()` now gets the parameter's AST node, whereas previously it had no result.
+  - `Expr.flow()` now has a more meaningful result for destructuring patterns. Previously this node was disconnected from the data flow graph. Now it represents the values being destructured by the pattern.
+* The global data-flow and taint-tracking libraries now model indirect parameter accesses through the `arguments` object in some cases, which may lead to additional results from some of the security queries, particularly "Prototype pollution in utility function".

cpp/ql/src/Likely Bugs/Underspecified Functions/TooFewArguments.qll

@@ -6,29 +6,50 @@
 
 import cpp
 
-// True if function was ()-declared, but not (void)-declared or K&R-defined
+/**
+ * Holds if `fde` has a parameter declaration that's clear on the minimum
+ * number of parameters. This is essentially true for everything except
+ * `()`-declarations.
+ */
+private predicate hasDefiniteNumberOfParameters(FunctionDeclarationEntry fde) {
+  fde.hasVoidParamList()
+  or
+  fde.getNumberOfParameters() > 0
+  or
+  fde.isDefinition()
+}
+
+/* Holds if function was ()-declared, but not (void)-declared or K&R-defined. */
 private predicate hasZeroParamDecl(Function f) {
   exists(FunctionDeclarationEntry fde | fde = f.getADeclarationEntry() |
-    not fde.hasVoidParamList() and fde.getNumberOfParameters() = 0 and not fde.isDefinition()
+    not hasDefiniteNumberOfParameters(fde)
   )
 }
 
-// True if this file (or header) was compiled as a C file
+/* Holds if this file (or header) was compiled as a C file. */
 private predicate isCompiledAsC(File f) {
   f.compiledAsC()
   or
   exists(File src | isCompiledAsC(src) | src.getAnIncludedFile() = f)
 }
 
+/** Holds if `fc` is a call to `f` with too few arguments. */
 predicate tooFewArguments(FunctionCall fc, Function f) {
   f = fc.getTarget() and
   not f.isVarargs() and
   not f instanceof BuiltInFunction and
+  // This query should only have results on C (not C++) functions that have a
+  // `()` parameter list somewhere. If it has results on other functions, then
+  // it's probably because the extractor only saw a partial compilation.
   hasZeroParamDecl(f) and
   isCompiledAsC(f.getFile()) and
-  // There is an explicit declaration of the function whose parameter count is larger
-  // than the number of call arguments
-  exists(FunctionDeclarationEntry fde | fde = f.getADeclarationEntry() |
+  // Produce an alert when all declarations that are authoritative on the
+  // parameter count specify a parameter count larger than the number of call
+  // arguments.
+  forex(FunctionDeclarationEntry fde |
+    fde = f.getADeclarationEntry() and
+    hasDefiniteNumberOfParameters(fde)
+  |
     fde.getNumberOfParameters() > fc.getNumberOfArguments()
   )
 }