|
| 1 | +/** |
| 2 | + * Provides a library for writing QL tests whose success or failure is based on expected results |
| 3 | + * embedded in the test source code as comments, rather than a `.expected` file. |
| 4 | + * |
| 5 | + * To add this framework to a new language: |
| 6 | + * - Add a file `InlineExpectationsTestPrivate.qll` that defines a `ExpectationComment` class. This class |
| 7 | + * must support a `getContents` method that returns the contents of the given comment, _excluding_ |
| 8 | + * the comment indicator itself. It should also define `toString` and `getLocation` as usual. |
| 9 | + * |
| 10 | + * To create a new inline expectations test: |
| 11 | + * - Declare a class that extends `InlineExpectationsTest`. In the characteristic predicate of the |
| 12 | + * new class, bind `this` to a unique string (usually the name of the test). |
| 13 | + * - Override the `hasActualResult()` predicate to produce the actual results of the query. For each |
| 14 | + * result, specify a `Location`, a text description of the element for which the result was |
| 15 | + * reported, a short string to serve as the tag to identify expected results for this test, and the |
| 16 | + * expected value of the result. |
| 17 | + * - Override `getARelevantTag()` to return the set of tags that can be produced by |
| 18 | + * `hasActualResult()`. Often this is just a single tag. |
| 19 | + * |
| 20 | + * Example: |
| 21 | + * ```ql |
| 22 | + * class ConstantValueTest extends InlineExpectationsTest { |
| 23 | + * ConstantValueTest() { this = "ConstantValueTest" } |
| 24 | + * |
| 25 | + * override string getARelevantTag() { |
| 26 | + * // We only use one tag for this test. |
| 27 | + * result = "const" |
| 28 | + * } |
| 29 | + * |
| 30 | + * override predicate hasActualResult( |
| 31 | + * Location location, string element, string tag, string value |
| 32 | + * ) { |
| 33 | + * exists(Expr e | |
| 34 | + * tag = "const" and // The tag for this test. |
| 35 | + * value = e.getValue() and // The expected value. Will only hold for constant expressions. |
| 36 | + * location = e.getLocation() and // The location of the result to be reported. |
| 37 | + * element = e.toString() // The display text for the result. |
| 38 | + * ) |
| 39 | + * } |
| 40 | + * } |
| 41 | + * ``` |
| 42 | + * |
| 43 | + * There is no need to write a `select` clause or query predicate. All of the differences between |
| 44 | + * expected results and actual results will be reported in the `failures()` query predicate. |
| 45 | + * |
| 46 | + * To annotate the test source code with an expected result, place a comment on the |
| 47 | + * same line as the expected result, with text of the following format as the body of the comment: |
| 48 | + * |
| 49 | + * `$tag=expected-value` |
| 50 | + * |
| 51 | + * Where `tag` is the value of the `tag` parameter from `hasActualResult()`, and `expected-value` is |
| 52 | + * the value of the `value` parameter from `hasActualResult()`. The `=expected-value` portion may be |
| 53 | + * omitted, in which case `expected-value` is treated as the empty string. Multiple expectations may |
| 54 | + * be placed in the same comment, as long as each is prefixed by a `$`. Any actual result that |
| 55 | + * appears on a line that does not contain a matching expected result comment will be reported with |
| 56 | + * a message of the form "Unexpected result: tag=value". Any expected result comment for which there |
| 57 | + * is no matching actual result will be reported with a message of the form |
| 58 | + * "Missing result: tag=expected-value". |
| 59 | + * |
| 60 | + * Example: |
| 61 | + * ```cpp |
| 62 | + * int i = x + 5; // $const=5 |
| 63 | + * int j = y + (7 - 3) // $const=7 $const=3 $const=4 // The result of the subtraction is a constant. |
| 64 | + * ``` |
| 65 | + * |
| 66 | + * For tests that contain known false positives and false negatives, it is possible to further |
| 67 | + * annotate that a particular expected result is known to be a false positive, or that a particular |
| 68 | + * missing result is known to be a false negative: |
| 69 | + * |
| 70 | + * `$f+:tag=expected-value` // False positive |
| 71 | + * `$f-:tag=expected-value` // False negative |
| 72 | + * |
| 73 | + * A false positive expectation is treated as any other expected result, except that if there is no |
| 74 | + * matching actual result, the message will be of the form "Fixed false positive: tag=value". A |
| 75 | + * false negative expectation is treated as if there were no expected result, except that if a |
| 76 | + * matching expected result is found, the message will be of the form |
| 77 | + * "Fixed false negative: tag=value". |
| 78 | + * |
| 79 | + * If the same result value is expected for two or more tags on the same line, there is a shorthand |
| 80 | + * notation available: |
| 81 | + * |
| 82 | + * `$tag1,tag2=expected-value` |
| 83 | + * |
| 84 | + * is equivalent to: |
| 85 | + * |
| 86 | + * `$tag1=expected-value $tag2=expected-value` |
| 87 | + */ |
| 88 | + |
| 89 | +private import InlineExpectationsTestPrivate |
| 90 | + |
| 91 | +/** |
| 92 | + * Base class for tests with inline expectations. The test extends this class to provide the actual |
| 93 | + * results of the query, which are then compared with the expected results in comments to produce a |
| 94 | + * list of failure messages that point out where the actual results differ from the expected |
| 95 | + * results. |
| 96 | + */ |
| 97 | +abstract class InlineExpectationsTest extends string { |
| 98 | + bindingset[this] |
| 99 | + InlineExpectationsTest() { any() } |
| 100 | + |
| 101 | + /** |
| 102 | + * Returns all tags that can be generated by this test. Most tests will only ever produce a single |
| 103 | + * tag. Any expected result comments for a tag that is not returned by the `getARelevantTag()` |
| 104 | + * predicate for an active test will be ignored. This makes it possible to write multiple tests in |
| 105 | + * different `.ql` files that all query the same source code. |
| 106 | + */ |
| 107 | + abstract string getARelevantTag(); |
| 108 | + |
| 109 | + /** |
| 110 | + * Returns the actual results of the query that is being tested. Each result consist of the |
| 111 | + * following values: |
| 112 | + * - `location` - The source code location of the result. Any expected result comment must appear |
| 113 | + * on the start line of this location. |
| 114 | + * - `element` - Display text for the element on which the result is reported. |
| 115 | + * - `tag` - The tag that marks this result as coming from this test. This must be one of the tags |
| 116 | + * returned by `getARelevantTag()`. |
| 117 | + * - `value` - The value of the result, which will be matched against the value associated with |
| 118 | + * `tag` in any expected result comment on that line. |
| 119 | + */ |
| 120 | + abstract predicate hasActualResult(string file, int line, string element, string tag, string value); |
| 121 | + |
| 122 | + final predicate hasFailureMessage(FailureLocatable element, string message) { |
| 123 | + exists(ActualResult actualResult | |
| 124 | + actualResult.getTest() = this and |
| 125 | + element = actualResult and |
| 126 | + ( |
| 127 | + exists(FalseNegativeExpectation falseNegative | |
| 128 | + falseNegative.matchesActualResult(actualResult) and |
| 129 | + message = "Fixed false negative:" + falseNegative.getExpectationText() |
| 130 | + ) |
| 131 | + or |
| 132 | + not exists(ValidExpectation expectation | expectation.matchesActualResult(actualResult)) and |
| 133 | + message = "Unexpected result: " + actualResult.getExpectationText() |
| 134 | + ) |
| 135 | + ) |
| 136 | + or |
| 137 | + exists(ValidExpectation expectation | |
| 138 | + not exists(ActualResult actualResult | expectation.matchesActualResult(actualResult)) and |
| 139 | + expectation.getTag() = getARelevantTag() and |
| 140 | + element = expectation and |
| 141 | + ( |
| 142 | + expectation instanceof GoodExpectation and |
| 143 | + message = "Missing result:" + expectation.getExpectationText() |
| 144 | + or |
| 145 | + expectation instanceof FalsePositiveExpectation and |
| 146 | + message = "Fixed false positive:" + expectation.getExpectationText() |
| 147 | + ) |
| 148 | + ) |
| 149 | + or |
| 150 | + exists(InvalidExpectation expectation | |
| 151 | + element = expectation and |
| 152 | + message = "Invalid expectation syntax: " + expectation.getExpectation() |
| 153 | + ) |
| 154 | + } |
| 155 | +} |
| 156 | + |
| 157 | +/** |
| 158 | + * RegEx pattern to match a comment containing one or more expected results. The comment must have |
| 159 | + * `$` as its first non-whitespace character. Any subsequent character |
| 160 | + * is treated as part of the expected results, except that the comment may contain a `//` sequence |
| 161 | + * to treat the remainder of the line as a regular (non-interpreted) comment. |
| 162 | + */ |
| 163 | +private string expectationCommentPattern() { result = "\\s*(\\$(?:[^/]|/[^/])*)(?://.*)?" } |
| 164 | + |
| 165 | +/** |
| 166 | + * RegEx pattern to match a single expected result, not including the leading `$`. It starts with an |
| 167 | + * optional `f+:` or `f-:`, followed by one or more comma-separated tags containing only letters, |
| 168 | + * `-`, and `_`, optionally followed by `=` and the expected value. |
| 169 | + */ |
| 170 | +private string expectationPattern() { |
| 171 | + result = "(?:(f(?:\\+|-)):)?((?:[A-Za-z-_]+)(?:\\s*,\\s*[A-Za-z-_]+)*)(?:=(.*))?" |
| 172 | +} |
| 173 | + |
| 174 | +private string getAnExpectation(ExpectationComment comment) { |
| 175 | + result = comment.getContents().regexpCapture(expectationCommentPattern(), 1).splitAt("$").trim() and |
| 176 | + result != "" |
| 177 | +} |
| 178 | + |
| 179 | +private newtype TFailureLocatable = |
| 180 | + TActualResult( |
| 181 | + InlineExpectationsTest test, string file, int line, string element, string tag, string value |
| 182 | + ) { |
| 183 | + test.hasActualResult(file, line, element, tag, value) |
| 184 | + } or |
| 185 | + TValidExpectation(ExpectationComment comment, string tag, string value, string knownFailure) { |
| 186 | + exists(string expectation | |
| 187 | + expectation = getAnExpectation(comment) and |
| 188 | + expectation.regexpMatch(expectationPattern()) and |
| 189 | + tag = expectation.regexpCapture(expectationPattern(), 2).splitAt(",").trim() and |
| 190 | + ( |
| 191 | + if exists(expectation.regexpCapture(expectationPattern(), 3)) |
| 192 | + then value = expectation.regexpCapture(expectationPattern(), 3) |
| 193 | + else value = "" |
| 194 | + ) and |
| 195 | + ( |
| 196 | + if exists(expectation.regexpCapture(expectationPattern(), 1)) |
| 197 | + then knownFailure = expectation.regexpCapture(expectationPattern(), 1) |
| 198 | + else knownFailure = "" |
| 199 | + ) |
| 200 | + ) |
| 201 | + } or |
| 202 | + TInvalidExpectation(ExpectationComment comment, string expectation) { |
| 203 | + expectation = getAnExpectation(comment) and |
| 204 | + not expectation.regexpMatch(expectationPattern()) |
| 205 | + } |
| 206 | + |
| 207 | +class FailureLocatable extends TFailureLocatable { |
| 208 | + string toString() { none() } |
| 209 | + |
| 210 | + predicate hasLocation(string file, int line) { none() } |
| 211 | + |
| 212 | + final string getExpectationText() { result = getTag() + "=" + getValue() } |
| 213 | + |
| 214 | + string getTag() { none() } |
| 215 | + |
| 216 | + string getValue() { none() } |
| 217 | +} |
| 218 | + |
| 219 | +class ActualResult extends FailureLocatable, TActualResult { |
| 220 | + InlineExpectationsTest test; |
| 221 | + string file; |
| 222 | + int line; |
| 223 | + string element; |
| 224 | + string tag; |
| 225 | + string value; |
| 226 | + |
| 227 | + ActualResult() { this = TActualResult(test, file, line, element, tag, value) } |
| 228 | + |
| 229 | + override string toString() { result = element } |
| 230 | + |
| 231 | + override predicate hasLocation(string f, int l) { f = file and l = line } |
| 232 | + |
| 233 | + InlineExpectationsTest getTest() { result = test } |
| 234 | + |
| 235 | + override string getTag() { result = tag } |
| 236 | + |
| 237 | + override string getValue() { result = value } |
| 238 | +} |
| 239 | + |
| 240 | +abstract private class Expectation extends FailureLocatable { |
| 241 | + ExpectationComment comment; |
| 242 | + |
| 243 | + override string toString() { result = comment.toString() } |
| 244 | + |
| 245 | + override predicate hasLocation(string file, int line) { comment.hasLocationInfo(file, line, _, _, _) } |
| 246 | +} |
| 247 | + |
| 248 | +private class ValidExpectation extends Expectation, TValidExpectation { |
| 249 | + string tag; |
| 250 | + string value; |
| 251 | + string knownFailure; |
| 252 | + |
| 253 | + ValidExpectation() { this = TValidExpectation(comment, tag, value, knownFailure) } |
| 254 | + |
| 255 | + override string getTag() { result = tag } |
| 256 | + |
| 257 | + override string getValue() { result = value } |
| 258 | + |
| 259 | + string getKnownFailure() { result = knownFailure } |
| 260 | + |
| 261 | + predicate matchesActualResult(ActualResult actualResult) { |
| 262 | + exists(string file, int line | actualResult.hasLocation(file, line) | |
| 263 | + this.hasLocation(file, line) |
| 264 | + ) and |
| 265 | + getTag() = actualResult.getTag() and |
| 266 | + getValue() = actualResult.getValue() |
| 267 | + } |
| 268 | +} |
| 269 | + |
| 270 | +class GoodExpectation extends ValidExpectation { |
| 271 | + GoodExpectation() { getKnownFailure() = "" } |
| 272 | +} |
| 273 | + |
| 274 | +class FalsePositiveExpectation extends ValidExpectation { |
| 275 | + FalsePositiveExpectation() { getKnownFailure() = "f+" } |
| 276 | +} |
| 277 | + |
| 278 | +class FalseNegativeExpectation extends ValidExpectation { |
| 279 | + FalseNegativeExpectation() { getKnownFailure() = "f-" } |
| 280 | +} |
| 281 | + |
| 282 | +class InvalidExpectation extends Expectation, TInvalidExpectation { |
| 283 | + string expectation; |
| 284 | + |
| 285 | + InvalidExpectation() { this = TInvalidExpectation(comment, expectation) } |
| 286 | + |
| 287 | + string getExpectation() { result = expectation } |
| 288 | +} |
| 289 | + |
| 290 | +query predicate failures(string file, int line, FailureLocatable element, string message) { |
| 291 | + exists(InlineExpectationsTest test | test.hasFailureMessage(element, message) | |
| 292 | + element.hasLocation(file, line) |
| 293 | + ) |
| 294 | +} |
0 commit comments