QL handbook: bring library path documentation up to date

Author: hmakholm

docs/language/ql-handbook/language.rst

@@ -42,6 +42,79 @@ A QL program can be *evaluated* (see `Evaluation <#evaluation>`__) to produce a
 
 For a QL program to be *valid*, it must conform to a variety of conditions that are described throughout this specification; otherwise the program is said to be *invalid*. An implementation of QL must detect all invalid programs and refuse to evaluate them.
 
+Library path
+------------
+
+The library path an ordered list of directory locations that is used
+in for resolving module imports, described below.  It is not strictly
+speaking a core part of the QL language, since different
+implementations of QL construct it in slightly different ways. Most QL
+tooling also allows specifying it explicitly on the command line for a
+particular invocation, though that is that is rarely done, and only
+useful in very special situation. This section describes the default
+construction of the library path.
+
+First, determine the *query directory* of the ``.ql`` file being
+compiled. Starting with the directory containing the ``.ql`` file, and
+walking up the directory structure, each directory is checked for a
+file called ``queries.xml`` or ``qlpack.yml``. The first directory
+where such a file is found is the query directory. If there is no such
+directory, the directory of the ``.ql`` file itself is the query
+directory.
+
+A ``queries.xml`` file that defines a query directory must always
+contain, containing a single top-level tag named
+``queries``, which has a ``language`` attribute set to the identifier
+of the active database scheme (for example, ``<queries
+language="java"/>``).
+
+A ``qlpack.yml`` file defines a `QL pack
+<https://help.semmle.com/codeql/codeql-cli/reference/qlpack-overview.html>`;
+its content is described in the CodeQL CLI documentation. This file
+will not be recognized when using older QL tooling that is not based
+on the CodeQL CLI (that is, LGTM.com, LGTM Enterprise, Odasa, QL for
+Eclipse, and QL for Visual Studio).
+
+If both a ``queries.xml`` and a ``qlpack.yml`` exist in the same
+directory the latter takes precedence (and the former is assumed to
+exist for compatibility with older tooling).
+
+The query directory itself becomes the first element of the library
+path.
+
+In old tooling that doesn't recognize ``qlpack.yml``, the default
+value of the rest of the library path is hard-coded in the tooling for
+each supporting language. It contains directories within the Odasa
+distribution that define the default CodeQL libraries for the selected
+language. Which language to use depends on the ``language`` attibute
+of the ``queries.xml`` file if not overridden with a ``--language``
+option to Odasa.
+
+On the other hand, the CodeQL CLI and newer tooling based on it (e.g.,
+GitHub Code Scanning and the Visual Stidio Code extension for CodeQL)
+constructs a default library path using QL packs. For each QL pack
+added to the language path, the QL packs named in its
+``libraryPathDependencies`` will be subsequently added to the library
+path, and the process continues until all packs have been
+resolved. The actual library path consists of the root directories of
+the selected QL packs. This process depends on a mechanism for finding
+QL packs by pack name, as described in the CodeQL CLI documentation.
+
+When the query directory contains a ``queries.xml`` file but no
+``qlpack.yml``, the QL pack resolution behaves as if it defines a QL
+pack with no name and a single library-path dependency named
+``legacy-libraries-LANGUAGE`` where ``LANGUAGE`` is taken from
+``queries.xml``. The ``github/codeql`` repository provides packs with
+names following this pattern, which themselves depend on the actual
+CodeQL libraries for each language.
+
+When the query directory contains neither ``queries.xml`` nor
+``qlpack.yml`` it will be considered to be a QL pack with no name and
+no library dependencies. This causes the library path to consist of
+*only* the query directory itself, which is not generally useful --
+but will suffice to run toy examples of QL code that don't actually
+use information from the database.
+
 Name resolution
 ---------------
 
@@ -162,11 +235,9 @@ For selection identifiers (``a::b``):
 
 For qualified identifiers (``a.b``):
 
--  Define the *current file* as the file the import directive occurs in.
-
--  Determine the current file's *query directory*, if any. Starting with the directory containing the current file, and walking up the directory structure, each directory is checked for a file called ``queries.xml``, containing a single top-level tag named ``queries``, which has a ``language`` attribute set to the identifier of the active database scheme (for example, ``<queries language="java"/>``). The closest enclosing directory is taken as the current file's query directory.
-
--  Build up a list of *candidate search paths*, consisting of the current file's directory, the current file's query directory (if one was determined in the previous step), and the list of directories making up the library path (in order).
+-  Build up a list of *candidate search paths*, consisting of the
+   current file's directory, and each of the directories on the
+   *library path* (in order).
 
 -  Determine the first candidate search path that has a *matching* QLL file for the import directive's qualified name. A QLL file in a candidate search path is said to match a qualified name if, starting from the candidate search path, there is a subdirectory for each successive qualifier in the qualified name, and the directory named by the final qualifier contains a file whose base name matches the qualified name's base name, with the addition of the file extension ``.qll``. The file and directory names are matched case-sensitively, regardless of whether the filesystem is case-sensitive or not.