Skip to content

Commit df3eb9f

Browse files
tausbntausbn
authored
Merge pull request github#3790 from RasmusWL/python-add-annotated-callgraph-tests
Python: Add annotated call-graph tests
2 parents 2941f41 + d00e739 commit df3eb9f

21 files changed

+516
-0
lines changed

‎python/ql/test/experimental/library-tests/CallGraph-xfail/CallGraphTest.qll

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../CallGraph/CallGraphTest.qll

‎python/ql/test/experimental/library-tests/CallGraph-xfail/PointsTo.expected

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
debug_missingAnnotationForCallable
2+
| annotation_xfail.py:10:1:10:24 | callable_not_annotated() | This call is annotated with 'callable_not_annotated', but no callable with that annotation was extracted. Please fix. |
3+
debug_nonUniqueAnnotationForCallable
4+
| annotation_xfail.py:13:1:13:17 | Function non_unique | Multiple callables are annotated with 'non_unique'. Please fix. |
5+
| annotation_xfail.py:17:1:17:26 | Function too_much_copy_paste | Multiple callables are annotated with 'non_unique'. Please fix. |
6+
debug_missingAnnotationForCall
7+
| annotation_xfail.py:2:1:2:24 | Function no_annotated_call | This callable is annotated with 'no_annotated_call', but no call with that annotation was extracted. Please fix. |
8+
expectedCallEdgeNotFound
9+
| call_edge_xfail.py:36:1:36:11 | xfail_foo() | call_edge_xfail.py:8:1:8:16 | Function xfail_bar |
10+
| call_edge_xfail.py:39:1:39:11 | xfail_baz() | call_edge_xfail.py:8:1:8:16 | Function xfail_bar |
11+
unexpectedCallEdgeFound
12+
| call_edge_xfail.py:29:1:29:6 | func() | call_edge_xfail.py:4:1:4:16 | Function xfail_foo | Call resolved to the callable named 'xfail_foo' but was not annotated as such |
13+
| call_edge_xfail.py:29:1:29:6 | func() | call_edge_xfail.py:8:1:8:16 | Function xfail_bar | Call resolved to the callable named 'xfail_bar' but was not annotated as such |
14+
| call_edge_xfail.py:30:1:30:11 | xfail_foo() | call_edge_xfail.py:4:1:4:16 | Function xfail_foo | Call resolved to the callable named 'xfail_foo' but was not annotated as such |
15+
| call_edge_xfail.py:31:1:31:14 | xfail_lambda() | call_edge_xfail.py:15:16:15:44 | Function lambda | Call resolved to the callable named 'xfail_lambda' but was not annotated as such |
16+
| call_edge_xfail.py:36:1:36:11 | xfail_foo() | call_edge_xfail.py:4:1:4:16 | Function xfail_foo | Call resolved to the callable named 'xfail_foo' but was not annotated as such |
17+
| call_edge_xfail.py:39:1:39:11 | xfail_baz() | call_edge_xfail.py:11:1:11:16 | Function xfail_baz | Annotated call resolved to unannotated callable |
18+
| call_edge_xfail.py:43:1:43:6 | func() | call_edge_xfail.py:8:1:8:16 | Function xfail_bar | Call resolved to the callable named 'xfail_bar' but was not annotated as such |

‎python/ql/test/experimental/library-tests/CallGraph-xfail/PointsTo.ql

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
../CallGraph/PointsTo.ql

‎python/ql/test/experimental/library-tests/CallGraph-xfail/README.md

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
Test that show our failure handling in [CallGraph](../CallGraph/) works as expected.

‎python/ql/test/experimental/library-tests/CallGraph-xfail/annotation_xfail.py

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# name:no_annotated_call
2+
def no_annotated_call():
3+
pass
4+
5+
def callable_not_annotated():
6+
pass
7+
8+
no_annotated_call()
9+
# calls:callable_not_annotated
10+
callable_not_annotated()
11+
12+
# name:non_unique
13+
def non_unique():
14+
pass
15+
16+
# name:non_unique
17+
def too_much_copy_paste():
18+
pass
19+
20+
# calls:non_unique
21+
non_unique()

‎python/ql/test/experimental/library-tests/CallGraph-xfail/call_edge_xfail.py

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
import sys
2+
3+
# name:xfail_foo
4+
def xfail_foo():
5+
print('xfail_foo')
6+
7+
# name:xfail_bar
8+
def xfail_bar():
9+
print('xfail_bar')
10+
11+
def xfail_baz():
12+
print('xfail_baz')
13+
14+
# name:xfail_lambda
15+
xfail_lambda = lambda: print('xfail_lambda')
16+
17+
if len(sys.argv) >= 2 and not sys.argv[1] in ['0', 'False', 'false']:
18+
func = xfail_foo
19+
else:
20+
func = xfail_bar
21+
22+
# Correct usage to suppress bad annotation errors
23+
# calls:xfail_foo calls:xfail_bar
24+
func()
25+
# calls:xfail_lambda
26+
xfail_lambda()
27+
28+
# These are not annotated, and will give rise to unexpectedCallEdgeFound
29+
func()
30+
xfail_foo()
31+
xfail_lambda()
32+
33+
# These are annotated wrongly, and will give rise to unexpectedCallEdgeFound
34+
35+
# calls:xfail_bar
36+
xfail_foo()
37+
38+
# calls:xfail_bar
39+
xfail_baz()
40+
41+
# The annotation is incomplete (does not include the call to xfail_bar)
42+
# calls:xfail_foo
43+
func()

‎python/ql/test/experimental/library-tests/CallGraph/CallGraphTest.qll

Lines changed: 147 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,147 @@
1+
import python
2+
3+
/** Gets the comment on the line above `ast` */
4+
Comment commentFor(AstNode ast) {
5+
exists(int line | line = ast.getLocation().getStartLine() - 1 |
6+
result
7+
.getLocation()
8+
.hasLocationInfo(ast.getLocation().getFile().getAbsolutePath(), line, _, line, _)
9+
)
10+
}
11+
12+
/** Gets the value from `tag:value` in the comment for `ast` */
13+
string getAnnotation(AstNode ast, string tag) {
14+
exists(Comment comment, string match, string theRegex |
15+
theRegex = "([\\w]+):([\\w.]+)" and
16+
comment = commentFor(ast) and
17+
match = comment.getText().regexpFind(theRegex, _, _) and
18+
tag = match.regexpCapture(theRegex, 1) and
19+
result = match.regexpCapture(theRegex, 2)
20+
)
21+
}
22+
23+
/** Gets a callable annotated with `name:name` */
24+
Function annotatedCallable(string name) { name = getAnnotation(result, "name") }
25+
26+
/** Gets a call annotated with `calls:name` */
27+
Call annotatedCall(string name) { name = getAnnotation(result, "calls") }
28+
29+
predicate missingAnnotationForCallable(string name, Call call) {
30+
call = annotatedCall(name) and
31+
not exists(annotatedCallable(name))
32+
}
33+
34+
predicate nonUniqueAnnotationForCallable(string name, Function callable) {
35+
strictcount(annotatedCallable(name)) > 1 and
36+
callable = annotatedCallable(name)
37+
}
38+
39+
predicate missingAnnotationForCall(string name, Function callable) {
40+
not exists(annotatedCall(name)) and
41+
callable = annotatedCallable(name)
42+
}
43+
44+
/** There is an obvious problem with the annotation `name` */
45+
predicate nameInErrorState(string name) {
46+
missingAnnotationForCallable(name, _)
47+
or
48+
nonUniqueAnnotationForCallable(name, _)
49+
or
50+
missingAnnotationForCall(name, _)
51+
}
52+
53+
/** Source code has annotation with `name` showing that `call` will call `callable` */
54+
predicate annotatedCallEdge(string name, Call call, Function callable) {
55+
not nameInErrorState(name) and
56+
call = annotatedCall(name) and
57+
callable = annotatedCallable(name)
58+
}
59+
60+
// ------------------------- Annotation debug query predicates -------------------------
61+
query predicate debug_missingAnnotationForCallable(Call call, string message) {
62+
exists(string name |
63+
message =
64+
"This call is annotated with '" + name +
65+
"', but no callable with that annotation was extracted. Please fix." and
66+
missingAnnotationForCallable(name, call)
67+
)
68+
}
69+
70+
query predicate debug_nonUniqueAnnotationForCallable(Function callable, string message) {
71+
exists(string name |
72+
message = "Multiple callables are annotated with '" + name + "'. Please fix." and
73+
nonUniqueAnnotationForCallable(name, callable)
74+
)
75+
}
76+
77+
query predicate debug_missingAnnotationForCall(Function callable, string message) {
78+
exists(string name |
79+
message =
80+
"This callable is annotated with '" + name +
81+
"', but no call with that annotation was extracted. Please fix." and
82+
missingAnnotationForCall(name, callable)
83+
)
84+
}
85+
86+
// ------------------------- Call Graph resolution -------------------------
87+
private newtype TCallGraphResolver =
88+
TPointsToResolver() or
89+
TTypeTrackerResolver()
90+
91+
/** Describes a method of call graph resolution */
92+
abstract class CallGraphResolver extends TCallGraphResolver {
93+
abstract predicate callEdge(Call call, Function callable);
94+
95+
/**
96+
* Holds if annotations show that `call` will call `callable`,
97+
* but our call graph resolver was not able to figure that out
98+
*/
99+
predicate expectedCallEdgeNotFound(Call call, Function callable) {
100+
annotatedCallEdge(_, call, callable) and
101+
not this.callEdge(call, callable)
102+
}
103+
104+
/**
105+
* Holds if there are no annotations that show that `call` will call `callable` (where at least one of these are annotated),
106+
* but the call graph resolver claims that `call` will call `callable`
107+
*/
108+
predicate unexpectedCallEdgeFound(Call call, Function callable, string message) {
109+
this.callEdge(call, callable) and
110+
not annotatedCallEdge(_, call, callable) and
111+
(
112+
exists(string name |
113+
message = "Call resolved to the callable named '" + name + "' but was not annotated as such" and
114+
callable = annotatedCallable(name) and
115+
not nameInErrorState(name)
116+
)
117+
or
118+
exists(string name |
119+
message = "Annotated call resolved to unannotated callable" and
120+
call = annotatedCall(name) and
121+
not nameInErrorState(name) and
122+
not exists( | callable = annotatedCallable(_))
123+
)
124+
)
125+
}
126+
127+
string toString() { result = "CallGraphResolver" }
128+
}
129+
130+
/** A call graph resolver based on the existing points-to analysis */
131+
class PointsToResolver extends CallGraphResolver, TPointsToResolver {
132+
override predicate callEdge(Call call, Function callable) {
133+
exists(PythonFunctionValue funcValue |
134+
funcValue.getScope() = callable and
135+
call = funcValue.getACall().getNode()
136+
)
137+
}
138+
139+
override string toString() { result = "PointsToResolver" }
140+
}
141+
142+
/** A call graph resolved based on Type Trackers */
143+
class TypeTrackerResolver extends CallGraphResolver, TTypeTrackerResolver {
144+
override predicate callEdge(Call call, Function callable) { none() }
145+
146+
override string toString() { result = "TypeTrackerResolver" }
147+
}

‎python/ql/test/experimental/library-tests/CallGraph/PointsTo.expected

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
debug_missingAnnotationForCallable
2+
debug_nonUniqueAnnotationForCallable
3+
debug_missingAnnotationForCall
4+
expectedCallEdgeNotFound
5+
| code/underscore_prefix_func_name.py:16:5:16:19 | some_function() | code/underscore_prefix_func_name.py:10:1:10:20 | Function some_function |
6+
unexpectedCallEdgeFound

‎python/ql/test/experimental/library-tests/CallGraph/PointsTo.ql

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
import python
2+
import CallGraphTest
3+
4+
query predicate expectedCallEdgeNotFound(Call call, Function callable) {
5+
any(PointsToResolver r).expectedCallEdgeNotFound(call, callable)
6+
}
7+
8+
query predicate unexpectedCallEdgeFound(Call call, Function callable, string message) {
9+
any(PointsToResolver r).unexpectedCallEdgeFound(call, callable, message)
10+
}

‎python/ql/test/experimental/library-tests/CallGraph/README.md

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
# Call Graph Tests
2+
3+
A small testing framework for our call graph resolution. It relies on manual annotation of calls and callables, **and will only include output if something is wrong**. For example, if we are not able to resolve that the `foo()` call will call the `foo` function, that should give an alert.
4+
5+
```py
6+
# name:foo
7+
def foo():
8+
pass
9+
# calls:foo
10+
foo()
11+
```
12+
13+
This is greatly inspired by [`CallGraphs/AnnotatedTest`](https://github.com/github/codeql/blob/696d19cb1440b6f6a75c6a2c1319e18860ceb436/javascript/ql/test/library-tests/CallGraphs/AnnotatedTest/Test.ql) from JavaScript.
14+
15+
IMPORTANT: Names used in annotations are not scoped, so must be unique globally. (this is a bit annoying, but makes things simple). If multiple identical annotations are used, an error message will be output.
16+
17+
Important files:
18+
19+
- `CallGraphTest.qll`: main code to find annotated calls/callables and setting everything up.
20+
- `PointsTo.ql`: results when using points-to for call graph resolution.
21+
- `TypeTracker.ql`: results when using TypeTracking for call graph resolution.
22+
- `Relative.ql`: differences between using points-to and TypeTracking.
23+
- `code/` contains the actual Python code we test against (included by `test.py`).
24+
25+
All queries will also execute some `debug_*` predicates. These highlight any obvious problems with the annotation setup, and so there should never be any results committed. To show that this works as expected, see the [CallGraph-xfail](../CallGraph-xfail/) which uses symlinked versions of the files in this directory (can't include as subdir, so has to be a sibling).
26+
27+
## `options` file
28+
29+
If the value for `--max-import-depth` is set so that `import random` will extract `random.py` from the standard library, BUT NO transitive imports are extracted, then points-to analysis will fail to handle the following snippet.
30+
31+
```py
32+
import random
33+
if random.random() < 0.5:
34+
func = foo
35+
else:
36+
func = bar
37+
func()
38+
```

0 commit comments

Comments
 (0)