Merge pull request github#7618 from github/aeisenberg/getting-started-docs

Docs: Simplify getting started docs

Author: aeisenberg

docs/codeql/codeql-cli/analyzing-databases-with-the-codeql-cli.rst

@@ -26,6 +26,7 @@ Running ``codeql database analyze``
 
 When you run ``database analyze``, it:
 
+#. Optionally downloads any referenced CodeQL packages that are not available locally.
 #. Executes one or more query files, by running them over a CodeQL database.
 #. Interprets the results, based on certain query metadata, so that alerts can be
    displayed in the correct location in the source code.
@@ -50,23 +51,30 @@ You must specify:
 
 You can also specify:
 
-- ``<queries>``: the queries to run over your database. You can
-  list one or more individual query files, specify a directory that will be
-  searched recursively for query files, or name a query suite that defines a
-  particular set of queries. If omitted, the default query suite for the language
-  of the database being analyzed will be usedFor more information, see the
-  :ref:`examples <database-analyze-examples>` below.
+- ``...<query-specifications>``: a list of queries to run over your database. This
+  is a list of arguments. Where each argument can be:
+
+  - a path to a query file
+  - a path to a directory containing query files
+  - a path to a query suite file
+  - the name of a CodeQL query pack
+    If omitted, the default query suite for the language
+    of the database being analyzed will be used. For more information, see the
+    :ref:`examples <database-analyze-examples>` below.
 
 - ``--sarif-category``: an identifying category for the results. Used when
   you want to upload more than one set of results for a commit.
   For example, when you use  ``github upload-results`` to send results for more than one
   language to the GitHub code scanning API. For more information about this use case,
   see `Configuring CodeQL CLI in your CI system <https://docs.github.com/en/code-security/secure-coding/using-codeql-code-scanning-with-your-existing-ci-system/configuring-codeql-cli-in-your-ci-system>`__ in the GitHub documentation.
 
-- ``--sarif-add-query-help``: (supported in version 2.7.1 onwards) adds any custom query help written 
-   in markdown to SARIF files (v2.1.0 or later) generated by the analysis. Query help stored in ``.qhelp`` files must be 
-   converted to ``.md`` before running the analysis. For further information, 
-   see ":ref:`Including query help for custom CodeQL queries in SARIF files  <including-query-help-for-custom-codeql-queries-in-sarif-files>`."
+- ``--sarif-add-query-help``: (supported in version 2.7.1 onwards) adds any custom query help written
+  in markdown to SARIF files (v2.1.0 or later) generated by the analysis. Query help stored in ``.qhelp`` files must be
+  converted to ``.md`` before running the analysis. For further information,
+  see ":ref:`Including query help for custom CodeQL queries in SARIF files  <including-query-help-for-custom-codeql-queries-in-sarif-files>`."
+
+- ``--download``: a boolean flag that will allow the CLI to download  any referenced CodeQL packages that are not available locally.
+  If this flag is missing and a referenced CodeQL package is not available locally, the command will fail.
 
 - .. include:: ../reusables/threads-query-execution.rst
 
@@ -119,17 +127,16 @@ Running a CodeQL pack
 
 .. include:: ../reusables/beta-note-package-management.rst
 
-To run an existing CodeQL query pack from the GitHub Container registry, you need to download it first::
-
-   codeql pack download microsoft/[email protected]
+To run an existing CodeQL query pack from the GitHub Container registry, you can specify one or more
+pack names and use the ``--download`` flag::
 
-Afterwards, you can run the pack on a specific database::
+   codeql database analyze <database> microsoft/[email protected] github/security-queries --format=sarifv2.1.0 --output=query-results.sarif --download
 
-   codeql database analyze <database> microsoft/[email protected] <scope>/<other-pack> --format=sarifv2.1.0 --output=query-results.sarif
-
-The ``analyze`` command above runs the default suite from ``microsoft/coding-standards v1.0.0`` and the latest version of ``scope/other-pack`` on the specified database.
+The ``analyze`` command above runs the default suite from ``microsoft/coding-standards v1.0.0`` and the latest version of ``github/secutiry-queries`` on the specified database.
 For further information about default suites, see ":ref:`Publishing and using CodeQL packs <publishing-and-using-codeql-packs>`".
 
+For more information about CodeQL packs, see :doc:`About CodeQL Packs <about-codeql-packs>`.
+
 Running query suites
 ~~~~~~~~~~~~~~~~~~~~
 
@@ -176,7 +183,7 @@ Integrating a CodeQL pack into a code scanning workflow in GitHub
 
 .. include:: ../reusables/beta-note-package-management.rst
 
-You can use CodeQL query packs in your Code Scanning setup. This allows you to select query packs published by various sources and use them to analyze your code. 
+You can use CodeQL query packs in your code scanning setup. This allows you to select query packs published by various sources and use them to analyze your code.
 For more information, see "`Using CodeQL query packs in the CodeQL action <https://docs.github.com/en/code-security/secure-coding/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-codeql-query-packs/>`_" or "`Downloading and using CodeQL query packs in your CI system <https://docs.github.com/en/code-security/secure-coding/using-codeql-code-scanning-with-your-existing-ci-system/configuring-codeql-cli-in-your-ci-system#downloading-and-using-codeql-query-packs>`_."
 
 
@@ -206,28 +213,28 @@ A SARIF results file is generated. Specifying ``--format=sarif-latest`` ensures
 that the results are formatted according to the most recent SARIF specification
 supported by CodeQL.
 
-.. _including-query-help-for-custom-codeql-queries-in-sarif-files: 
+.. _including-query-help-for-custom-codeql-queries-in-sarif-files:
 
-Including query help for custom CodeQL queries in SARIF files 
+Including query help for custom CodeQL queries in SARIF files
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 If you use the CodeQL CLI to to run code scanning analyses on third party CI/CD systems,
 you can include the query help for your custom queries in SARIF files generated during an analysis.
-After uploading the SARIF file to GitHub, the query help is shown in the code scanning UI for any 
-alerts generated by the custom queries. 
+After uploading the SARIF file to GitHub, the query help is shown in the code scanning UI for any
+alerts generated by the custom queries.
 
-From CodeQL CLI 2.7.1 onwards, you can include markdown-rendered query help in SARIF files 
+From CodeQL CLI 2.7.1 onwards, you can include markdown-rendered query help in SARIF files
 by providing the ``--sarif-add-query-help`` option when running
-``codeql database analyze``. 
+``codeql database analyze``.
 For more information, see `Configuring CodeQL CLI in your CI system <https://docs.github.com/en/code-security/code-scanning/using-codeql-code-scanning-with-your-existing-ci-system/configuring-codeql-cli-in-your-ci-system#analyzing-a-codeql-database>`__
 in the GitHub documentation.
 
 You can write query help for custom queries directly in a markdown file and save it alongside the
-corresponding query. Alternatively, for consistency with the standard CodeQL queries, 
-you can write query help in the ``.qhelp`` format. Query help written in ``.qhelp`` 
+corresponding query. Alternatively, for consistency with the standard CodeQL queries,
+you can write query help in the ``.qhelp`` format. Query help written in ``.qhelp``
 files can't be included in SARIF files, and they can't be processed by code
-scanning so must be converted to markdown before running 
-the analysis. For more information, see ":ref:`Query help files <query-help-files>`" 
+scanning so must be converted to markdown before running
+the analysis. For more information, see ":ref:`Query help files <query-help-files>`"
 and ":doc:`Testing query help files <testing-query-help-files>`."
 
 Results

docs/codeql/codeql-cli/getting-started-with-the-codeql-cli.rst

@@ -30,6 +30,8 @@ to display in GitHub as code scanning alerts, see
 `Installing CodeQL CLI in your CI system <https://docs.github.com/en/code-security/secure-coding/using-codeql-code-scanning-with-your-existing-ci-system/installing-codeql-cli-in-your-ci-system>`__
 in the GitHub documentation.
 
+.. _download-cli:
+
 1. Download the CodeQL CLI zip package
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -84,6 +86,90 @@ Alternatively, you can download ``codeql.zip``, which contains the CLI for all s
       iv. Ensure the check-box labeled **Open "safe" files after downloading**.
           is unchecked.
 
+2. Extract the zip archive
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For Linux, Windows, and macOS users (version 10.14 "Mojave", and earlier)
+simply extract the zip archive.
+
+.. container:: toggle
+
+   .. container:: name
+
+      **Information for macOS "Catalina" (or newer) users**
+
+   .. pull-quote:: macOS "Catalina"
+
+      macOS "Catalina", "Big Sur", or "Monterey" users should run the following
+      commands in the Terminal, where ``${extraction-root}`` is the path to the
+      directory where you will extract the CodeQL CLI zip archive:
+
+      i. ``mv ~/Downloads/codeql*.zip ${extraction-root}``
+      ii. ``cd ${extraction-root}``
+      iii. ``/usr/bin/xattr -c codeql*.zip``
+      iv. ``unzip codeql*.zip``
+
+.. _launch-codeql-cli:
+
+3. Launch ``codeql``
+~~~~~~~~~~~~~~~~~~~~
+
+Once extracted, you can run CodeQL processes by running the ``codeql``
+executable in a couple of ways:
+
+- By executing ``<extraction-root>/codeql/codeql``, where
+  ``<extraction-root>`` is the folder where you extracted the CodeQL CLI
+  package.
+- By adding ``<extraction-root>/codeql`` to your ``PATH``, so that you
+  can run the executable as just ``codeql``.
+
+At this point, you can execute CodeQL commands. For a full list of the CodeQL
+CLI commands, see the "`CodeQL CLI manual <../manual>`__."
+
+.. pull-quote:: Note
+
+   If you add ``codeql`` to your ``PATH``, it can be accessed by CodeQL
+   for Visual Studio Code to compile and run queries.
+   For more information about configuring VS Code to access the CodeQL CLI, see
+   ":ref:`Setting up CodeQL in Visual Studio Code <setting-up-codeql-in-visual-studio-code>`."
+
+
+4. Verify your CodeQL CLI setup
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+CodeQL CLI has subcommands you can execute to verify that you are correctly set
+up to create and analyze databases:
+
+- Run ``codeql resolve languages`` to show which languages are
+  available for database creation. This will list the languages supported by
+  default in your CodeQL CLI package.
+- (Optional) You can download some ":ref:`CodeQL packs <about-codeql-packs>`" containing pre-compiled queries you would like to run.
+  To do this, run ``codeql pack download <pack-name> [...pack-name]``, where ``pack-name`` is the name of
+  the pack you want to download. The core query packs are a good place to start. They are:
+
+      - ``codeql/cpp-queries``
+      - ``codeql/csharp-queries``
+      - ``codeql/java-queries``
+      - ``codeql/javascript-queries``
+      - ``codeql/python-queries``
+      - ``codeql/ruby-queries``
+
+   Alternatively, you can download query packs during the analysis by using the ``--download`` flag of the ``codeql database analyze``
+   command.
+
+
+Checking out the CodeQL source code directly
+--------------------------------------------
+
+Some users prefer working with CodeQL query sources directly in order to work on or contribute to the Open Source shared queries. In
+order to do this, the following steps are recommended. Note that the following instructions are a slightly more complicated alternative
+to working with CodeQL packages as explained above.
+
+1. Download the CodeQL CLI zip
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Follow :ref:`step 1 from the previous section<download-cli>`.
+
 2. Create a new CodeQL directory
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -164,45 +250,11 @@ For example, if the path to your copy of the CodeQL repository is
 ``$HOME/codeql-home/codeql-repo``, then extract the CLI into
 ``$HOME/codeql-home/``.
 
-.. container:: toggle
-
-   .. container:: name
-
-      **Information for macOS "Catalina" (or newer) users**
-
-   .. pull-quote:: macOS "Catalina"
-
-      macOS "Catalina", "Big Sur", or "Monterey" users should run the following
-      commands in the Terminal, where ``${install_loc}`` is the path to the
-      directory you created in step 2:
-
-      i. ``mv ~/Downloads/codeql*.zip ${install_loc}``
-      ii. ``cd ${install_loc}``
-      iii. ``/usr/bin/xattr -c codeql*.zip``
-      iv. ``unzip codeql*.zip``
 
 5. Launch ``codeql``
 ~~~~~~~~~~~~~~~~~~~~
 
-Once extracted, you can run CodeQL processes by running the ``codeql``
-executable in a couple of ways:
-
-- By executing ``<extraction-root>/codeql/codeql``, where
-  ``<extraction-root>`` is the folder where you extracted the CodeQL CLI
-  package.
-- By adding ``<extraction-root>/codeql`` to your ``PATH``, so that you
-  can run the executable as just ``codeql``.
-
-At this point, you can execute CodeQL commands. For a full list of the CodeQL
-CLI commands, see the "`CodeQL CLI manual <../manual>`__."
-
-.. pull-quote:: Note
-
-   If you add ``codeql`` to your ``PATH``, it can be accessed by CodeQL
-   for Visual Studio Code to compile and run queries.
-   For more information about configuring VS Code to access the CodeQL CLI, see
-   ":ref:`Setting up CodeQL in Visual Studio Code <setting-up-codeql-in-visual-studio-code>`."
-
+See :ref:`step 3 from the previous section<launch-codeql-cli>`.
 
 6. Verify your CodeQL CLI setup
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -227,7 +279,6 @@ up to create and analyze databases:
   - Legacy packs that ensure custom queries and libraries created using older products are
     compatible with your version of CodeQL.
 
-
 .. _using-two-versions-of-the-codeql-cli:
 
 Using two versions of the CodeQL CLI

docs/codeql/codeql-cli/upgrading-codeql-databases.rst

@@ -4,20 +4,21 @@ Upgrading CodeQL databases
 ==========================
 
 As the CodeQL CLI tools and queries evolve, you may find that some of your
-CodeQL databases become out of date. You must upgrade out-of-date databases 
+CodeQL databases become out of date. You must upgrade out-of-date databases
 before you can analyze them.
 
 Databases become out of date when:
 
 - For databases created using the CodeQL CLI, the version of CLI tools used to
-  create them is older than your copy of the CodeQL queries. 
+  create them is older than your copy of the CodeQL queries.
 - For databases downloaded from LGTM.com, the CodeQL tools used by LGTM.com to create
   that revision of the code are older than your copy of the CodeQL queries.
 
-The ``main`` branch of the CodeQL queries is updated more often than both the
+If you have a local checkout of the ``github/codeql`` repository, please note that
+the ``main`` branch of the CodeQL queries is updated more often than both the
 CLI and LGTM.com, so databases are most likely to become out of date if you use
 the queries on this branch. For more information about the different versions of
-the CodeQL queries, see ":ref:`Getting started with the CodeQL CLI <local-copy-codeql-queries>`." 
+the CodeQL queries, see ":ref:`Getting started with the CodeQL CLI <local-copy-codeql-queries>`."
 
 Out-of-date databases must be upgraded before they can be analyzed. This topic
 shows you how to upgrade a CodeQL database using the ``database upgrade``