Merge pull request github#5425 from yoff/tausbn-python-document-api-graphs

Python: document api graphs

Author: erik-krogh

docs/codeql/codeql-language-guides/analyzing-data-flow-in-python.rst

@@ -99,6 +99,8 @@ Python has builtin functionality for reading and writing files, such as the func
 
 ➤ `See this in the query console on LGTM.com <https://lgtm.com/query/8635258505893505141/>`__. Two of the demo projects make use of this low-level API.
 
+Notice the use of the ``API`` module for referring to library functions. For more information, see ":doc:`Using API graphs in Python <using-api-graphs-in-python>`."
+
 Unfortunately this will only give the expression in the argument, not the values which could be passed to it. So we use local data flow to find all expressions that flow into the argument:
 
 .. code-block:: ql

docs/codeql/codeql-language-guides/codeql-for-python.rst

@@ -11,6 +11,7 @@ Experiment and learn how to write effective and efficient queries for CodeQL dat
    basic-query-for-python-code
    codeql-library-for-python
    analyzing-data-flow-in-python
+   using-api-graphs-in-python
    functions-in-python
    expressions-and-statements-in-python
    analyzing-control-flow-in-python
@@ -21,6 +22,8 @@ Experiment and learn how to write effective and efficient queries for CodeQL dat
 
 -  :doc:`Analyzing data flow in Python <analyzing-data-flow-in-python>`: You can use CodeQL to track the flow of data through a Python program to places where the data is used.
 
+-  :doc:`Using API graphs in Python <using-api-graphs-in-python>`: API graphs are a uniform interface for referring to functions, classes, and methods defined in external libraries.
+
 -  :doc:`Functions in Python <functions-in-python>`: You can use syntactic classes from the standard CodeQL library to find Python functions and identify calls to them.
 
 -  :doc:`Expressions and statements in Python <expressions-and-statements-in-python>`: You can use syntactic classes from the CodeQL library to explore how Python expressions and statements are used in a codebase.

docs/codeql/codeql-language-guides/codeql-library-for-python.rst

@@ -23,10 +23,9 @@ The CodeQL library for Python incorporates a large number of classes. Each class
 -  **Data flow** - classes that represent entities from the data flow graphs.
 -  **API graphs** - classes that represent entities from the API graphs.
 
-The first two categories are described below. See ":doc:`Analyzing data flow in Python <analyzing-data-flow-in-python>`" for a description of data flow and associated classes.
-
-..
-   and [TO COME IN FUTURE PR] for a description of API graphs and their use.
+The first two categories are described below. 
+For a description of data flow and associated classes, see ":doc:`Analyzing data flow in Python <analyzing-data-flow-in-python>`".
+For a description of API graphs and their use, see ":doc:`Using API graphs in Python <using-api-graphs-in-python>`."
 
 Syntactic classes
 -----------------

docs/codeql/codeql-language-guides/using-api-graphs-in-python.rst

@@ -0,0 +1,164 @@
+.. _using-api-graphs-in-python:
+
+Using API graphs in Python
+==========================
+
+API graphs are a uniform interface for referring to functions, classes, and methods defined in
+external libraries.
+
+About this article
+------------------
+
+This article describes how to use API graphs to reference classes and functions defined in library
+code. You can use API graphs to conveniently refer to external library functions when defining things like
+remote flow sources.
+
+
+Module imports
+--------------
+
+The most common entry point into the API graph will be the point where an external module or package is
+imported. For example, you can access the API graph node corresponding to the ``re`` library
+by using the ``API::moduleImport`` method defined in the ``semmle.python.ApiGraphs`` module, as the
+following snippet demonstrates.
+
+.. code-block:: ql
+
+    import python
+    import semmle.python.ApiGraphs
+
+    select API::moduleImport("re")
+
+➤ `See this in the query console on LGTM.com <https://lgtm.com/query/1876172022264324639/>`__.
+
+This query only selects the API graph node corresponding to the ``re`` module. To find
+where this module is referenced, you can use the ``getAUse`` method. The following query selects
+all references to the ``re`` module in the current database.
+
+.. code-block:: ql
+
+    import python
+    import semmle.python.ApiGraphs
+
+    select API::moduleImport("re").getAUse()
+
+➤ `See this in the query console on LGTM.com <https://lgtm.com/query/8072356519514905526/>`__.
+
+Note that the ``getAUse`` method accounts for local flow, so that ``my_re_compile``
+in the following snippet is
+correctly recognized as a reference to the ``re.compile`` function.
+
+.. code-block:: python
+
+    from re import compile as re_compile
+
+    my_re_compile = re_compile
+
+    r = my_re_compile(".*")
+
+If you only require immediate uses, without taking local flow into account, then you can use
+the ``getAnImmediateUse`` method instead.
+
+Note that the given module name *must not* contain any dots. Thus, something like
+``API::moduleImport("flask.views")`` will not do what you expect. Instead, this should be decomposed
+into an access of the ``views`` member of the API graph node for ``flask``, as described in the next
+section.
+
+Accessing attributes
+--------------------
+
+Given a node in the API graph, you can access its attributes by using the ``getMember`` method. Using
+the above ``re.compile`` example, you can now find references to ``re.compile``.
+
+.. code-block:: ql
+
+    import python
+    import semmle.python.ApiGraphs
+
+    select API::moduleImport("re").getMember("compile").getAUse()
+
+➤ `See this in the query console on LGTM.com <https://lgtm.com/query/7970570434725297676/>`__.
+
+In addition to ``getMember``, you can use the ``getUnknownMember`` method to find references to API
+components where the name is not known statically. You can use the ``getAMember`` method to
+access all members, both known and unknown.
+
+Calls and class instantiations
+------------------------------
+
+To track instances of classes defined in external libraries, or the results of calling externally
+defined functions, you can use the ``getReturn`` method. The following snippet finds all places
+where the return value of ``re.compile`` is used:
+
+.. code-block:: ql
+
+    import python
+    import semmle.python.ApiGraphs
+
+    select API::moduleImport("re").getMember("compile").getReturn().getAUse()
+
+➤ `See this in the query console on LGTM.com <https://lgtm.com/query/4346050399960356921/>`__.
+
+Note that this includes all uses of the result of ``re.compile``, including those reachable via
+local flow. To get just the *calls* to ``re.compile``, you can use ``getAnImmediateUse`` instead of
+``getAUse``. As this is a common occurrence, you can use ``getACall`` instead of
+``getReturn`` followed by ``getAnImmediateUse``. 
+
+➤ `See this in the query console on LGTM.com <https://lgtm.com/query/8143347716552092926/>`__.
+
+Note that the API graph does not distinguish between class instantiations and function calls. As far
+as it's concerned, both are simply places where an API graph node is called.
+
+Subclasses
+----------
+
+For many libraries, the main mode of usage is to extend one or more library classes. To track this
+in the API graph, you can use the ``getASubclass`` method to get the API graph node corresponding to
+all the immediate subclasses of this node. To find *all* subclasses, use ``*`` or ``+`` to apply the
+method repeatedly, as in ``getASubclass*``.
+
+Note that ``getASubclass`` does not account for any subclassing that takes place in library code
+that has not been extracted. Thus, it may be necessary to account for this in the models you write.
+For example, the ``flask.views.View`` class has a predefined subclass ``MethodView``. To find
+all subclasses of ``View``, you must explicitly include the subclasses of ``MethodView`` as well.
+
+.. code-block:: ql
+
+    import python
+    import semmle.python.ApiGraphs
+
+    API::Node viewClass() {
+      result =
+        API::moduleImport("flask").getMember("views").getMember(["View", "MethodView"]).getASubclass*()
+    }
+    
+    select viewClass().getAUse()
+
+➤ `See this in the query console on LGTM.com <https://lgtm.com/query/288293322319747121/>`__.
+
+Note the use of the set literal ``["View", "MethodView"]`` to match both classes simultaneously.
+
+Built-in functions and classes
+------------------------------
+
+You can access built-in functions and classes using the ``API::builtin`` method, giving the name of
+the built-in as an argument.
+
+For example, to find all calls to the built-in ``open`` function, you can use the following snippet.
+
+.. code-block:: ql
+
+    import python
+    import semmle.python.ApiGraphs
+
+    select API::builtin("open").getACall()
+
+
+
+
+Further reading
+---------------
+
+
+.. include:: ../reusables/python-further-reading.rst
+.. include:: ../reusables/codeql-ref-tools-further-reading.rst