Skip to content
Merged
Changes from 3 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 81 additions & 2 deletions content/docs/static-analysis/codeql/10-advanced.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,71 @@
If you are using the CodeQL VSCode extension to write and run queries, [it can
initialize the query pack and create the `qlpack.yml` file automatically](#running-custom-queries-using-the-vscode-extension).

Finally, you have to create a [workspace file](https://docs.github.com/en/code-security/codeql-cli/using-the-advanced-functionality-of-the-codeql-cli/about-codeql-workspaces) for CodeQL CLI to work correctly.

Most probably you will write at least a few packs. Setup the following directory structure for the easiest development:
```

Check failure on line 44 in content/docs/static-analysis/codeql/10-advanced.md

View workflow job for this annotation

GitHub Actions / markdown-linter

Fenced code blocks should be surrounded by blank lines

content/docs/static-analysis/codeql/10-advanced.md:44 MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "```"] https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md031.md

Check failure on line 44 in content/docs/static-analysis/codeql/10-advanced.md

View workflow job for this annotation

GitHub Actions / markdown-linter

Fenced code blocks should have a language specified

content/docs/static-analysis/codeql/10-advanced.md:44 MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"] https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md040.md
.
├── CODEOWNERS
├── README.md
├── codeql-workspace.yml
├── cpp
│ ├── lib
│ │ ├── qlpack.yml
│ │ └── scope
│ │ └── crypto
│ │ └── someLibrary.qll
│ ├── src
│ │ ├── qlpack.yml
│ │ ├── codeql-suites
│ │ │ ├── scope-cpp-code-scanning.qls
│ │ │ └── scope-cpp-security.qls
│ │ ├── crypto
│ │ │ ├── SomeCryptoAnalysis.ql
│ │ ├── security
│ │ │ ├── AppSecAnalysis
│ │ │ │ ├── AppSecAnalysis.c
│ │ │ │ ├── AppSecAnalysis.qhelp
│ │ │ │ └── AppSecAnalysis.ql
│ │ ├── docs
│ │ │ ├── crypto
│ │ │ │ ├── SomeCryptoAnalysis.md
│ │ │ └── security
│ │ │ └── AppSecAnalysis.md
│ └── test
│ ├── qlpack.yml
│ ├── include
│ │ ├── libc
│ │ │ ├── stubs.h
│ ├── library-tests
│ │ └── crypto
│ │ ├── someLibrary
│ │ │ ├── someLibrary.expected
│ │ │ ├── someLibrary.ql
│ │ │ └── someLibrary.c
│ └── query-tests
│ ├── crypto
│ │ ├── SomeCryptoAnalysis
│ │ │ ├── SomeCryptoAnalysis.expected
│ │ │ ├── SomeCryptoAnalysis.qlref
│ │ │ └── SomeCryptoAnalysis.c
│ └── security
│ └── AppSecAnalysis
│ ├── AppSecAnalysis.c
│ ├── AppSecAnalysis.expected
│ └── AppSecAnalysis.qlref
├── go
│ ├── src
...
```

We divide query packs per-language, but also per-type (security, cryptographic, etc.). This follows GitHub's convention.

For setting-up unit tests continue reading to [Unit testing custom queries](#unit-testing-custom-queries) section.

Finally, you can use [our bash script for generating new queries](https://github.com/trailofbits/codeql-queries/tree/main/scripts/new_query.sh) when you have the structure above.


Check failure on line 105 in content/docs/static-analysis/codeql/10-advanced.md

View workflow job for this annotation

GitHub Actions / markdown-linter

Multiple consecutive blank lines

content/docs/static-analysis/codeql/10-advanced.md:105 MD012/no-multiple-blanks Multiple consecutive blank lines [Expected: 1; Actual: 2] https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md012.md
### Adding dependencies

To be able to define a custom query we need to import the CodeQL standard
Expand Down Expand Up @@ -199,6 +264,16 @@
of the version you want, you can use `"*"` which always resolves to the latest
version.)

### Installing the new packs
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this step still necessary if you've properly set up your workspace with codeql-workspace.yml?

I ask because I've had some trouble between using codeql-workspace.yml vs. .codeqlmanifest.json vs. --search-path vs. --additional-packs. And generally just using codeql-workspace.yml has fixed my problems 🤷

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure? You need codeql to known the location somehow, would it auto-pick locally cloned repo?
Also the pack download is easier than cloning the repo.

Fell free to update this section anyway.


Once you have initialized the new query pack, added dependencies and some sample query, you need to run
`codeql pack install` in every directory that has a qlpack.yml file (including folders with test).

Then, inform the codeql CLI about your new queries by creating `~/.config/codeql/config` file with the following content:
```

Check failure on line 273 in content/docs/static-analysis/codeql/10-advanced.md

View workflow job for this annotation

GitHub Actions / markdown-linter

Fenced code blocks should be surrounded by blank lines

content/docs/static-analysis/codeql/10-advanced.md:273 MD031/blanks-around-fences Fenced code blocks should be surrounded by blank lines [Context: "```"] https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md031.md

Check failure on line 273 in content/docs/static-analysis/codeql/10-advanced.md

View workflow job for this annotation

GitHub Actions / markdown-linter

Fenced code blocks should have a language specified

content/docs/static-analysis/codeql/10-advanced.md:273 MD040/fenced-code-language Fenced code blocks should have a language specified [Context: "```"] https://github.com/DavidAnson/markdownlint/blob/v0.37.4/doc/md040.md
--search-path /full/path/to/your/codeql/root/directory
```

## Writing custom queries

{{< hint info >}}
Expand Down Expand Up @@ -476,8 +551,12 @@
- `MemcpyCall.expected`: A text file containing the expected output from
running the query against the source file

The source file must build cleanly without any external dependencies. To test
the query, run the following command:
The source file must build cleanly without any external dependencies.
This requirement is problematic mostly for C/C++ queries: you need to create
stub files with `extern` declarations for libraries you want to `#include`.
Check [our tests](https://github.com/trailofbits/codeql-queries/blob/d994c7ca05dab30fe195555ef6943f9d51ec38df/cpp/test/query-tests/security/CStrnFinder/test.c#L1) for examples.

To test the query, run the following command:

```sh
codeql test run -- path/to/test/pack/root/directory
Expand Down
Loading