Skip to content

Commit 267e369

Browse files
authored
Merge pull request github#6153 from github/docs-content-4456-multiple-dbs
Add information on multi-db options to the CodeQL CLI docs
2 parents ca4c519 + c4047af commit 267e369

File tree

4 files changed

+79
-55
lines changed

4 files changed

+79
-55
lines changed

docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst

Lines changed: 24 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -56,6 +56,12 @@ You must specify:
5656

5757
You can also specify:
5858

59+
- ``--sarif-category``: an identifying category for the results. Used when
60+
you want to upload more than one set of results for a commit.
61+
For example, when you use ``github upload-results`` to send results for more than one
62+
language to the GitHub code scanning API. For more information about this use case,
63+
see `Configuring CodeQL CLI in your CI system <https://docs.github.com/en/code-security/secure-coding/using-codeql-code-scanning-with-your-existing-ci-system/configuring-codeql-cli-in-your-ci-system>`__ in the GitHub documentation.
64+
5965
- .. include:: ../reusables/threads-query-execution.rst
6066

6167

@@ -85,8 +91,8 @@ repositories.
8591
Running a single query
8692
~~~~~~~~~~~~~~~~~~~~~~
8793

88-
To run a single query over a JavaScript codebase, you could use the following
89-
command from the directory containing your database::
94+
To run a single query over a CodeQL database for a JavaScript codebase,
95+
you could use the following command from the directory containing your database::
9096

9197
codeql database analyze <javascript-database> ../ql/javascript/ql/src/Declarations/UnusedVariable.ql --format=csv --output=js-analysis/js-results.csv
9298

@@ -105,13 +111,23 @@ see ":doc:`Using custom queries with the CodeQL CLI <using-custom-queries-with-t
105111
Running GitHub code scanning suites
106112
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
107113

108-
The CodeQL repository also includes query suites, which can be run over your
109-
code as part of a broader code review. CodeQL query suites are ``.qls`` files
110-
that use directives to select queries to run based on certain metadata
111-
properties.
114+
To run the GitHub code scanning suite of queries over a CodeQL database for a C/C++ codebase,
115+
you could use the following command from the directory containing your database::
116+
117+
codeql database analyze <cpp-database> cpp-code-scanning.qls --format=sarifv2.1.0 --output=cpp-results.sarif
118+
119+
The analysis generates a file in the v2.1.0 SARIF format that is supported by all versions of GitHub.
120+
This file can be uploaded to GitHub using ``github upload-results`` or the code scanning API.
121+
For more information, see `Analyzing a CodeQL database <https://docs.github.com/en/code-security/secure-coding/configuring-codeql-cli-in-your-ci-system#analyzing-a-codeql-database>`__
122+
or `Code scanning API <https://docs.github.com/en/rest/reference/code-scanning>`__ in the GitHub documentation.
112123

113-
The CodeQL repository includes query suites that are used by the CodeQL action on
114-
`GitHub.com <https://github.com>`__. The query suites are located at the following paths in
124+
CodeQL query suites are ``.qls`` files that use directives to select queries to run
125+
based on certain metadata properties. The standard QL packs have metadata that specify
126+
the location of the code scanning suites, so the CodeQL CLI knows where to find these
127+
suite files automatically, and you don't have to specify the full path on the command line.
128+
For more information, see ":ref:`About QL packs <standard-ql-packs>`."
129+
130+
The standard query suites are stored at the following paths in
115131
the CodeQL repository::
116132

117133
ql/<language>/ql/src/codeql-suites/<language>-code-scanning.qls
@@ -120,23 +136,6 @@ and at the following path in the CodeQL for Go repository::
120136

121137
ql/src/codeql-suites/go-code-scanning.qls
122138

123-
These locations are specified in the metadata included in the standard QL packs.
124-
This means that the CodeQL CLI knows where to find the suite files automatically, and
125-
you don't have to specify the full path on the command line when running an
126-
analysis. For more information, see ":ref:`About QL packs <standard-ql-packs>`."
127-
128-
.. pull-quote::
129-
130-
Important
131-
132-
If you plan to upload the results to GitHub, you must generate SARIF results.
133-
For more information, see `Analyzing a CodeQL database <https://docs.github.com/en/code-security/secure-coding/running-codeql-cli-in-your-ci-system#analyzing-a-codeql-database>`__ in the GitHub documentation.
134-
135-
For example, to run the code scanning query suite on a C++ codebase and generate
136-
results in the v2.1 SARIF format supported by all versions of GitHub, you would run::
137-
138-
codeql database analyze <cpp-database> cpp-code-scanning.qls --format=sarifv2.1.0 --output=cpp-analysis/cpp-results.sarif
139-
140139
The repository also includes the query suites used by `LGTM.com <https://lgtm.com>`__.
141140
These are stored alongside the code scanning suites with names of the form: ``<language>-lgtm.qls``.
142141

docs/codeql/codeql-cli/creating-codeql-databases.rst

Lines changed: 49 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,12 @@ Before you generate a CodeQL database, you need to:
1717
- Check out the version of your codebase you want to analyze. The directory
1818
should be ready to build, with all dependencies already installed.
1919

20+
For information about using the CodeQL CLI in a third-party CI system to create results
21+
to display in GitHub as code scanning alerts, see `Configuring CodeQL CLI in your CI system <https://docs.github.com/en/code-security/secure-coding/using-codeql-code-scanning-with-your-existing-ci-system/configuring-codeql-cli-in-your-ci-system>`__
22+
in the GitHub documentation. For information about enabling CodeQL code scanning using GitHub Actions,
23+
see `Setting up code scanning for a repository <https://docs.github.com/en/code-security/secure-coding/automatically-scanning-your-code-for-vulnerabilities-and-errors/setting-up-code-scanning-for-a-repository>`__
24+
in the GitHub documentation.
25+
2026
Running ``codeql database create``
2127
----------------------------------
2228

@@ -33,21 +39,30 @@ You must specify:
3339
be created when you execute the command---you cannot specify an existing
3440
directory.
3541
- ``--language``: the identifier for the language to create a database for.
42+
When used with ``--db-cluster``, the option accepts a comma-separated list,
43+
or can be specified more than once.
3644
CodeQL supports creating databases for the following languages:
3745

3846
.. include:: ../reusables/extractors.rst
3947

40-
Other options may be specified depending on the location of your source file and
41-
the language you want to analyze:
48+
You can specify additional options depending on the location of your source file,
49+
if the code needs to be compiled, and if you want to create CodeQL databases for
50+
more than one language:
4251

4352
- ``--source-root``: the root folder for the primary source files used in
4453
database creation. By default, the command assumes that the current
4554
directory is the source root---use this option to specify a different location.
46-
- ``--command``: for compiled languages only, the build commands that invoke the
47-
compiler. Do not specify ``--command`` options for Python and
48-
JavaScript. Commands will be run from the current folder, or ``--source-root``
55+
- ``--db-cluster``: use for multi-language codebases when you want to create
56+
databases for more than one language.
57+
- ``--command``: used when you create a database for one or more compiled languages,
58+
omit if the only languages requested are Python and JavaScript.
59+
This specifies the build commands needed to invoke the compiler.
60+
Commands are run from the current folder, or ``--source-root``
4961
if specified. If you don't include a ``--command``, CodeQL will attempt to
5062
detect the build system automatically, using a built-in autobuilder.
63+
- ``--no-run-unnecessary-builds``: used with ``--db-cluster`` to suppress the build
64+
command for languages where the CodeQL CLI does not need to monitor the build
65+
(for example, Python and JavaScript/TypeScript).
5166

5267
For full details of all the options you can use when creating databases,
5368
see the `database create reference documentation <../manual/database-create>`__.
@@ -62,31 +77,13 @@ it failed. For compiled languages, the console will display the output of the
6277
build system.
6378

6479
When the database is successfully created, you'll find a new directory at the
65-
path specified in the command. This directory contains a number of
80+
path specified in the command. If you used the ``--db-cluster`` option to create
81+
more than one database, a subdirectory is created for each language.
82+
Each CodeQL database directory contains a number of
6683
subdirectories, including the relational data (required for analysis) and a
6784
source archive---a copy of the source files made at the time the database was
6885
created---which is used for displaying analysis results.
6986

70-
Obtaining databases from LGTM.com
71-
---------------------------------
72-
73-
`LGTM.com <https://lgtm.com>`__ analyzes thousands of open-source projects using
74-
CodeQL. For each project on LGTM.com, you can download an archived CodeQL
75-
database corresponding to the most recently analyzed revision of the code. These
76-
databases can also be analyzed using the CodeQL CLI.
77-
78-
.. include:: ../reusables/download-lgtm-database.rst
79-
80-
Before running an analysis, unzip the databases and try :doc:`upgrading <upgrading-codeql-databases>` the
81-
unzipped databases to ensure they are compatible with your local copy of the
82-
CodeQL queries and libraries.
83-
84-
.. pull-quote::
85-
86-
Note
87-
88-
.. include:: ../reusables/index-files-note.rst
89-
9087
Creating databases for non-compiled languages
9188
---------------------------------------------
9289

@@ -99,9 +96,11 @@ are available.
9996

10097
.. pull-quote:: Important
10198

102-
When running ``database create`` for JavaScript, TypeScript, and Python, you must not
103-
specify a ``--command`` option. If you do, you will override the normal
104-
extractor invocation, which will create an empty database.
99+
When you run ``database create`` for JavaScript, TypeScript, and Python, you should not
100+
specify a ``--command`` option. Otherwise this overrides the normal
101+
extractor invocation, which will create an empty database. If you create
102+
databases for multiple languages and one of them is a compiled language,
103+
use the ``--no-run-unnecessary-builds`` option to skip the command for the languages that don't need to be compiled.
105104

106105
JavaScript and TypeScript
107106
~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -229,6 +228,27 @@ commands that you can specify for compiled languages.
229228
This command runs a custom script that contains all of the commands required
230229
to build the project.
231230

231+
Obtaining databases from LGTM.com
232+
---------------------------------
233+
234+
`LGTM.com <https://lgtm.com>`__ analyzes thousands of open-source projects using
235+
CodeQL. For each project on LGTM.com, you can download an archived CodeQL
236+
database corresponding to the most recently analyzed revision of the code. These
237+
databases can also be analyzed using the CodeQL CLI or used with the CodeQL
238+
extension for Visual Studio Code.
239+
240+
.. include:: ../reusables/download-lgtm-database.rst
241+
242+
Before running an analysis, unzip the databases and try :doc:`upgrading <upgrading-codeql-databases>` the
243+
unzipped databases to ensure they are compatible with your local copy of the
244+
CodeQL queries and libraries.
245+
246+
.. pull-quote::
247+
248+
Note
249+
250+
.. include:: ../reusables/index-files-note.rst
251+
232252

233253
Further reading
234254
---------------

docs/codeql/codeql-cli/getting-started-with-the-codeql-cli.rst

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,11 @@ follow the steps below. For macOS version 10.15 ("Catalina"), steps 1 and 4 are
2222
slightly different---for further details, see the sections labeled **Information
2323
for macOS "Catalina" users**.
2424

25+
For information about installing the CodeQL CLI in a CI system to create results
26+
to display in GitHub as code scanning alerts, see
27+
`Installing CodeQL CLI in your CI system <https://docs.github.com/en/code-security/secure-coding/using-codeql-code-scanning-with-your-existing-ci-system/installing-codeql-cli-in-your-ci-system>`__
28+
in the GitHub documentation.
29+
2530
1. Download the CodeQL CLI zip package
2631
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2732

docs/codeql/writing-codeql-queries/about-codeql-queries.rst

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@ CodeQL includes queries to find the most relevant and interesting problems for e
1313
- **Alert queries**: queries that highlight issues in specific locations in your code.
1414
- **Path queries**: queries that describe the flow of information between a source and a sink in your code.
1515

16-
You can add custom queries to :doc:`QL packs <../codeql-cli/about-ql-packs>` to analyze your projects with "`Code scanning <https://docs.github.com/github/finding-security-vulnerabilities-and-errors-in-your-code>`__", use them to analyze a database with the ":ref:`CodeQL CLI <codeql-cli>`," or you can contribute to the standard CodeQL queries in our `open source repository on GitHub <https://github.com/github/codeql>`__.
16+
You can add custom queries to :doc:`QL packs <../codeql-cli/about-ql-packs>` to analyze your projects with "`Code scanning <https://docs.github.com/en/code-security/secure-coding/automatically-scanning-your-code-for-vulnerabilities-and-errors/about-code-scanning>`__", use them to analyze a database with the ":ref:`CodeQL CLI <codeql-cli>`," or you can contribute to the standard CodeQL queries in our `open source repository on GitHub <https://github.com/github/codeql>`__.
1717

1818
This topic is a basic introduction to query files. You can find more information on writing queries for specific programming languages in the ":ref:`CodeQL language guides <codeql-language-guides>`," and detailed technical information about QL in the ":ref:`QL language reference <ql-language-reference>`."
1919
For more information on how to format your code when contributing queries to the GitHub repository, see the `CodeQL style guide <https://github.com/github/codeql/blob/main/docs/ql-style-guide.md>`__.

0 commit comments

Comments
 (0)