Skip to content

Commit f7289d6

Browse files
author
Alvaro Muñoz
committed
Add Hotspots query generator
1 parent 01d7bee commit f7289d6

File tree

13 files changed

+1373
-0
lines changed

13 files changed

+1373
-0
lines changed

‎.gitignore

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,4 +19,10 @@ test-*.ql
1919
**/*.testproj/*
2020
*/test-output.txt
2121

22+
# Temporary files
23+
ql/hotspots/output/**
24+
25+
# Cache
26+
**/__pycache__/
27+
2228
.DS_Store

‎ql/hotspots/Hotspots.ql

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
/**
2+
* @name
3+
* @id githubsecuritylab/hotspots-query-generator
4+
* @description Finds all security-related TaintTracking sinks
5+
* @kind problem
6+
* @precision low
7+
*/
8+
9+
import ql
10+
import utils.hotspots
11+
12+
predicate debug_counts(int a, int b) {
13+
a = count(SecurityQuery q) and
14+
b = count(TaintTrackingSecurityQuery q)
15+
}
16+
17+
predicate debug_missing(SecurityQuery q, string lang) {
18+
not exists(TaintTrackingSecurityQuery tq | tq = q) and q.getLanguage() = lang
19+
}
20+
21+
predicate supportedLanguage(string lang) {
22+
lang = ["javascript", "java", "ruby", "csharp", "go", "python", "cpp"]
23+
}
24+
25+
bindingset[severity]
26+
predicate supportedSeverity(float severity) { severity > 7.0 or severity = -1.0 }
27+
28+
from TaintTrackingSecurityQuery q, TaintTrackingConfiguration c
29+
where
30+
supportedLanguage(q.getLanguage()) and
31+
supportedSeverity(q.getSeverity()) and
32+
c = q.getTaintTrackingConfiguration()
33+
// 1. language, 2. query id, 3. config path, 4. config name, 5. query import stmt, 6. query pack, 7. query severity, config kind, config isStateConfig
34+
select q.getLanguage(), q.getId(), c.getPath(), c.getQualifiedName(), c.getImportStringFrom(q),
35+
c.getQLPack(), q.getSeverity().toString(), c.getKind(), c.isStateConfig()

‎ql/hotspots/README.md

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
# Hotspot query generator
2+
3+
This script uses QL-4-QL to find all security related path-problem queries and extract their TaintTracking configuration and the import statement needed to run them.
4+
5+
## Arguments
6+
7+
| Option | Description |
8+
| ----------------- | ----------------------------------------------------------------- |
9+
| `--ql-extractor` | Path to the CodeQL extractor (required) |
10+
| `--ql-path` | Path to the CodeQL repository to extract hotspots from (required) |
11+
| `--ql-executable` | Path to the CodeQL binary (default: "codeql") |
12+
13+
## Configuration
14+
15+
Configuration is located in `config/hotspots-config.yml` file (or where specified) and contains a configuration for each language.
16+
17+
E.g:
18+
19+
```yaml
20+
java:
21+
disallowed_patterns:
22+
- ".*-local"
23+
- ".*-experimental"
24+
disallowed_queries:
25+
- java/untrusted-data-to-external-api
26+
- java/log-injection
27+
- java/android/intent-redirection
28+
- java/improper-validation-of-array-construction
29+
ruby:
30+
allowed_queries:
31+
- rb/code-injection
32+
- rb/sql-injection
33+
```
34+
35+
- `allowed_queries`: List of query IDs to use to extract Hotspots from
36+
- `disallowed_queries`: List of queries to skip when processing TaintTracking queries to extract Hotspots from
37+
- `disallowed_patterns`: List of regexp patterns of queries to skip when processing TaintTracking queries to extract Hotspots from
38+
39+
## Usage
40+
41+
E.g:
42+
43+
- If you havent build the extractor for QL yet, cd into the `ql` folder of your CodeQL distribution (eg: `~/src/codeql/ql`) and run `./scripts/create-extractor-pack.sh`. This will generate `~/src/codeql/ql/extractor-pack`.
44+
45+
- Extract the hotspots info, dump it into `hotspots.csv` and create the `Hotspots.ql` queries for each language
46+
47+
```bash
48+
python scripts/generate-hotspots-queries.py --ql-extractor ~/src/codeql/ql/extractor-pack --ql-path ~/src/github/codeql
49+
```
50+
51+
- Create a patched version of CodeQL distro (remove private modifiers and rename files/directories to remove whitespaces and dashes)
52+
53+
```bash
54+
python scripts/patch_codeql.py --hotspots hotspots.csv --ql ~/src/codeql --dest /tmp/hotspots-distro --qlpack-version 0.0.1
55+
```
56+
57+
- Run Hotspots query (eg: `/tmp/hotspots-distro/java/ql/src/Hotspots.ql`)

‎ql/hotspots/config/hotspots-config.yml

Lines changed: 128 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,128 @@
1+
java:
2+
disallowed_patterns:
3+
- '.*-local'
4+
- '.*-experimental'
5+
disallowed_queries:
6+
- java/untrusted-data-to-external-api
7+
- java/log-injection
8+
- java/android/intent-redirection
9+
- java/improper-validation-of-array-construction
10+
- java/improper-validation-of-array-construction-code-specified
11+
- java/improper-validation-of-array-index
12+
- java/improper-validation-of-array-index-code-specified
13+
- java/tainted-format-string
14+
- java/tainted-arithmetic
15+
- java/uncontrolled-arithmetic
16+
- java/extreme-value-arithmetic
17+
- java/local-temp-file-or-directory-information-disclosure
18+
- java/non-https-url
19+
- java/weak-cryptographic-algorithm
20+
- java/potentially-weak-cryptographic-algorithm
21+
- java/missing-jwt-signature-check
22+
- java/sensitive-log
23+
- java/tainted-numeric-cast
24+
- java/hardcoded-credential-api-call
25+
- java/hardcoded-credential-sensitive-call
26+
- java/user-controlled-bypass
27+
- java/tainted-permissions-check
28+
- java/android/sensitive-communication
29+
- java/log4j-injection
30+
- java/spring-view-manipulation
31+
- java/server-side-template-injection
32+
- java/static-initialization-vector
33+
- java/sensitive-android-file-leak
34+
- java/possible-timing-attack-against-signature
35+
- java/timing-attack-against-headers-value
36+
- java/timing-attack-against-signature
37+
- java/disabled-certificate-revocation-checking
38+
- java/unsafe-tls-version
39+
- java/unvalidated-cors-origin-set
40+
- java/ip-address-spoofing
41+
- java/thread-resource-abuse
42+
- java/sensitive-query-with-get
43+
- java/uncaught-servlet-exception
44+
- java/xxe
45+
- java/xxe-with-experimental-sinks
46+
- java/xxe-local-experimental-sinks
47+
- java/android/nfe-local-android-dos
48+
- java/exec-tainted-environment
49+
50+
ruby:
51+
disallowed_patterns:
52+
- '.*-local'
53+
- '.*-experimental'
54+
disallowed_queries:
55+
- rb/clear-text-logging-sensitive-data
56+
- rb/clear-text-storage-sensitive-data
57+
- rb/hardcoded-credentials
58+
- rb/insecure-download
59+
- rb/insecure-randomness
60+
- rb/log-injection
61+
- rb/overly-permissive-file
62+
- rb/sensitive-get-query
63+
- rb/stack-trace-exposure
64+
65+
python:
66+
disallowed_patterns:
67+
- '.*-local'
68+
- '.*-experimental'
69+
disallowed_queries:
70+
- py/timing-attack-against-hash
71+
- py/timing-attack-sensitive-info
72+
- py/timing-attack-against-header-value
73+
- py/clear-text-logging-sensitive-data
74+
- py/ip-address-spoofing
75+
- py/log-injection
76+
- py/possible-timing-attack-against-hash
77+
- py/possible-timing-attack-sensitive-info
78+
- py/polynomial-redos
79+
- py/hardcoded-credentials
80+
- py/clear-text-storage-sensitive-data
81+
- py/untrusted-data-to-external-api
82+
- py/azure-storage/unsafe-client-side-encryption-in-us
83+
- py/xml-bomb # similar to xxe
84+
- py/weak-sensitive-data-hashing
85+
- py/partial-ssrf # similar to full-ssrf
86+
87+
go:
88+
disallowed_patterns:
89+
- '.*-local'
90+
- '.*-experimental'
91+
disallowed_queries:
92+
- go/log-injection
93+
- go/insecure-hostkeycallback
94+
- go/incomplete-hostname-regexp
95+
- go/cookie-httponly-not-set
96+
- go/cookie-httponly-not-set
97+
- go/insecure-tls
98+
- go/constant-oauth2-state
99+
- go/untrusted-data-to-external-api
100+
- go/timing-attack
101+
- go/suspicious-character-in-regex
102+
- go/divide-by-zero
103+
- go/cookie-httponly-not-set
104+
- go/incorrect-integer-conversion
105+
- go/cookie-httponly-not-set
106+
- go/untrusted-data-to-unknown-external-api
107+
- go/allocation-size-overflow
108+
- go/wrong-usage-of-unsafe
109+
110+
javascript:
111+
disallowed_patterns:
112+
- '.*-local'
113+
- '.*-experimental'
114+
115+
cpp:
116+
disallowed_patterns:
117+
- '.*-local'
118+
- '.*-experimental'
119+
120+
csharp:
121+
disallowed_patterns:
122+
- '.*-local'
123+
- '.*-experimental'
124+
disallowed_queries:
125+
- cs/inappropriate-encoding
126+
- cs/hash-without-salt
127+
- cs/stored-command-line-injection
128+

‎ql/hotspots/qlpack.yml

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
library: false
2+
name: githubsecuritylab/hotspots
3+
version: 0.0.1
4+
dependencies:
5+
codeql/ql: '*'

‎ql/hotspots/requirements.txt

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
pyyaml
2+
pandas
3+
Jinja2
4+
beautifulsoup4
5+
PyGithub

0 commit comments

Comments
 (0)