Merge branch 'robadams-cicd305-refdoc' into 'linux-6.6.y-memorizer-dev'

Robadams cicd305 refdoc

See merge request ring0/memorizer!45

Author: illinoisrobert

Documentation/memorizer/index.rst

@@ -15,7 +15,7 @@ run the memorizer kernel, along with collecting and sending kmap data.
 
 
 .. toctree::
-    :maxdepth: 1
+    :maxdepth: 2
 
     quick_start
     using_memorizer

Documentation/memorizer/reference.rst

@@ -153,21 +153,29 @@ Config Variables
   Boolean, enables/disables the Memorizer tool. Memorizer traces the memory allocations of kernel objects to track patterns across an object's lifetime.
 
 ``MEMORIZER_STATS``
-  Boolean, enables/disables the Memorizer statistics summary. The statistics summary includes the number of accesses and shadow objects allocated for Memorizer which will slow down the performance of the system.
+  Boolean, enables/disables the Memorizer statistics summary. The
+  statistics summary includes the number of accesses and shadow objects
+  allocated for Memorizer which will slow down the performance of
+  the system.
 
 ``MEMORIZER_TRACKPIDS``
-  Boolean, enables/disables the segregation of memory access counts by process id (PID) within Memorizer data.
+  Boolean, enables/disables the segregation of memory access counts by
+  process id (PID) within Memorizer data.
 
 ``MEMORIZER_DEBUGFS_RAM``
-  Boolean, enables/disables the exposure of Memorizer's buffer via a debugfs file.
+  Boolean, enables/disables the exposure of Memorizer's buffer via a
+  debugfs file.
 
 ``INLINE_LIBS``
-  Boolean, forces gcc to use inline calls for some library functions. This must be enabled to run Memorizer.
+  Boolean, forces gcc to use inline calls for some library functions. This
+  must be enabled to run Memorizer.
 
 Dependencies
 ~~~~~~~~~~~~
 ``KASAN``
-  Boolean, enables/disables Kernel Address Sanitizer (KASAN). This is an error detector designed to find out-of-bounds and use-after-free bugs in dynamic memory. This must be enabled to run Memorizer.
+  Boolean, enables/disables Kernel Address Sanitizer (KASAN). This is
+  an error detector designed to find out-of-bounds and use-after-free
+  bugs in dynamic memory. This must be enabled to run Memorizer.
 
 .. _`debugfs-files`:
 
@@ -333,24 +341,29 @@ Status Files
 
 ``stats``
   Reading this file generates human-readable statistical data
-  about the current state of Memorizer. For more information,
-  see :ref:`debugfs-stat`.
+  about the current state of Memorizer. The format is intended
+  to be self-evident and is subject to change from release to release.
+
+  External scripts should not rely upon the format of this file.
 
 
 File Formats
 ============
 
-.. _`debugfs-stat`:
-
 ``stat``
 ~~~~~~~~
 
-blah.
+Reading this file generates human-readable statistical data
+about the current state of Memorizer. The format is intended
+to be self-evident and is subject to change from release to release.
 
-.. _`debugfs-kmap`:
+External scripts should not rely upon the format of this file.
 
-``kmap``
-~~~~~~~~
+.. _debugfs-kmap-stream:
+.. _debugfs-kmap:
+
+``kmap`` and ``kmap_stream``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 .. note::
 
@@ -365,16 +378,73 @@ Memorizer outputs data as text. The format of the ``kmap`` file is as follows::
     ...
   ...
 
-The longer line represents the allocation and destruction of a kernel object.
-The shorter, indented, line represents the memory accesses of that same object.
-Each shorter line refers to the immediately preceding long line. There may be
-any number of shorter lines per long line. There may be any number of long lines
-in a kmap file.
+The longer line represents the allocation and destruction of a kernel
+object.  The shorter, indented, line represents the memory accesses of
+that same object.  Each shorter line refers to the immediately preceding
+long line. There may be any number of shorter lines per long line. There
+may be any number of long lines in a kmap file.
+
+The following fields are printed as unsigned hexadecimal values:
+
+* ``alloc_ip``
+* ``obj_va_ptr``
+* ``free_ip``
+
+The following fields are printed as unsigned decimal values:
+
+* ``size``
+* ``alloc_index``
+* ``free_index``
+
+The following fields are printed as (possibly blank) character strings:
+
+* ``alloc_type``
+* ``command``
+* ``slabname``
+* ``new_alloc_type``
+
+* The following field is printed as a signed decimal value:
+
+* ``pid``
+
+
 
 ``alloc_ip``
-  The instruction pointer of the ``call`` instruction which resulted
+  The return instruction pointer of the ``call`` instruction which resulted
   in the allocation of the object.
 
+  For some allocations, it is either not useful or not possible
+  to describe the actual ``call`` instruction. For example, a
+  statically-declared object does not have the same vmalloc-use-vfree
+  life cycle as a dynamically-allocated object. In those cases synthetic
+  values are used, each with a special meaning.
+
+  ``MEMORIZER_PREALLOCATED(0xfeedbeef)``
+    This flag is used when Slub allocates new objects for a cache.
+    Memorizer preallocates objects here so any accesses from constructors
+    are captured correctly. This value is subsequently overwritten with
+    the actual instruction pointer of the ``call`` instruction which
+    resulted in the allocation of the object.
+
+  ``0``
+    Memorizer maintains a set of general-purpose object descriptors,
+    each with ``alloc_ip`` set to zero and with ``alloc_type`` set to
+    a valid :ref:alloc_type value. These are used when a memory access
+    occurs and the memory address is not described by an existing
+    object descriptor.
+    
+  Other ``alloc_type`` values
+    Some memory accesses which cannot be associated with existing
+    object descriptors result in the creation of new descriptors
+    with ``alloc_ip`` set equal to :ref:alloc_type. These
+    types include UFO_MEMBLOCK(0x14), STACK_PAGE(0x3),
+    UFO_GLOBAL(0x18), and UFO_NONE(0x19), 
+
+    .. note::
+      ``alloc_ip`` is printed as a hexadecimal value without a ``0x``
+      prefix. The printed value ``14``, for example is ``0x14`` or
+      decimal 20.
+
 ``pid``
   The process ID of the process that allocated the object.
 
@@ -394,7 +464,7 @@ in a kmap file.
   `memorizer_index_type`_ for a description.
 
 ``free_ip``
-  The instruction pointer of the `call` instruction which destroyed the object.
+  The instruction pointer of the ``call`` instruction which destroyed the object.
 
   There are a few special cases:
 
@@ -403,57 +473,114 @@ in a kmap file.
     the free, then ``free_ip`` is also zero.
 
   - If a subsequently allocated object exists in the same virtual addresses
-    as a previously allocated, not freed, object, then Memorizer probably
-    did not observe the intervening free.
-
-    In this case, ``free_ip`` of the previous object is ``0xdeadbeef`` and
-    the ``free_index`` of the previous object is set equal to the
-    ``alloc_ip`` of the subsequent object.
-    ``new_alloc_type`` of the previous
-    allocation is set to the ``alloc_type`` of the subsequent allocation.
-
-  - If a subsequently allocated object has exactly the same virtual address
-    as the immediately preceding allocation, this represents a
-    nested allocation. In this case, ``free_ip`` is set to ``0xfedbeef``.
-    ``new_alloc_type`` of the previous
-    allocation is set to the ``alloc_type`` of the subsequent allocation.
-    ``free_index`` of the previous allocation is set to ``alloc_index``
-    of the subsequent allocation.
+    as a previously allocated, not freed, object, then either Memorizer
+    did not observe the intervening free, or there was no interveneing free.
+
+    * If this is the result of nested allocations, then ``free_ip`` of
+      the previous allocation will have ``0xfedbeef``.
+
+    * If this occurs for some other, unknown, reason, then ``free_ip`` of
+      the previous allocation will have ``0xdeadbeef``.
+
+    In either case, the ``free_index`` of the previous allocation is
+    set equal to the ``alloc_index`` of the subsequent allocation.
 
 ``alloc_type``
   Memorizer tracks various sorts of object allocation. This field
   gives an indication of which type this is.
 
   This field has several possible values. Consult the source code
-  for information on each of these::
-
-    STACK
-    STACK_FRAME
-    STACK_ARGS
-    STACK_PAGE
-    GEN_HEAP
-    UFO_HEAP
-    GLOBAL
-    KMALLOC
-    KMALLOC_ND
-    KMEM_CACHE
-    KMEM_CACHE_ND
-    KMEM_CACHE_BULK
-    ALLOC_PAGES
-    ALLOC_PAGES_EXACT
-    ALLOC_PAGES_GETFREEPAGES
-    ALLOC_PAGES_FOLIO
-    VMALLOC
-    INDUCED_ALLOC
-    BOOTMEM
-    MEMBLOCK
-    UFO_MEMBLOCK
-    MEMORIZER
-    USER
-    BUG
-    UFO_GLOBAL
-    UFO_NONE
-    NONE
+  for more information on each of these:
+
+  STACK (MEM_STACK, 0x0)
+    Unused.
+
+  STACK_FRAME (MEM_STACK_FRAME, 0x1)
+    Used in the generation of call frame graphs (CFGs).This value should
+    not appear in kmap.
+
+  STACK_ARGS (MEM_STACK_ARGS, 0x2)
+    Used in the generation of call frame graphs (CFGs).This value should
+    not appear in kmap.
+
+  STACK_PAGE (MEM_STACK_PAGE, 0x3)
+    Used to describe a kernel object allocated in a stack frame.
+
+  GEN_HEAP (MEM_HEAP, 0x4)
+    Unused.
+
+  UFO_HEAP (MEM_UFO_HEAP, 0x5)
+    Unused.
+
+  GLOBAL (MEM_GLOBAL, 0x6)
+    The object is globally declared.
+
+  KMALLOC (MEM_KMALLOC, 0x7)
+    The subject allocation was from either the ``kmalloc`` or
+    ``kmalloc_node`` family of allocators.
+
+  KMALLOC_ND (MEM_KMALLOC_ND, 0x8)
+    The subject allocation was from either the ``kmalloc`` or
+    ``kmalloc_node`` family of allocators.
+
+  KMEM_CACHE (MEM_KMEM_CACHE, 0x9)
+    The subject allocation was from the kmem_cache_alloc family of allocators.
+
+  KMEM_CACHE_ND (MEM_KMEM_CACHE_ND, 0xA)
+    The subject allocation was from the kmem_cache_alloc family of allocators.
+
+  KMEM_CACHE_BULK (MEM_KMEM_CACHE_BULK, 0xB)
+    The subject allocation was from the kmem_cache_alloc family of allocators.
+
+  VMALLOC (MEM_VMALLOC, 0xC)
+    The subject allocation was from ``vmalloc``.
+
+  ALLOC_PAGES (MEM_ALLOC_PAGES, 0xD)
+    The subject allocation was from ``__alloc_pages``., et al.
+
+  ALLOC_PAGES_EXACT (MEM_ALLOC_PAGES_EXACT, 0xE)
+    The subject allocation was from ``alloc_pages_exact``.
+
+  ALLOC_PAGES_GETFREEPAGES (MEM_ALLOC_PAGES_GETFREEPAGES, 0xF)
+    The subject allocation was from ``__get_free_page``.
+
+  ALLOC_PAGES_FOLIO (MEM_ALLOC_PAGES_FOLIO, 0x10)
+    The subject allocation was from ``__folio_alloc``.
+
+  INDUCED_ALLOC (MEM_INDUCED_ALLOC, 0x11)
+    This object was allocated during (and, presumably as a result of)
+    Memorizer itself. In order to avoid either deadlock or infinite
+    recursion, we mark this object as INDUCED.
+
+  BOOTMEM (MEM_BOOTMEM, 0x12)
+    Unused.
+
+  MEMBLOCK (MEM_MEMBLOCK, 0x13)
+    This object is inside a region returned by ``memblock_insert_region``.
+
+  UFO_MEMBLOCK (MEM_UFO_MEMBLOCK, 0x14)
+    This object is inside a ``memblock_insert_region``,
+    but is (incorrectly) not in the Memorizer tracking system.
+
+  MEMORIZER (MEM_MEMORIZER, 0x15)
+    This object is inside Memorizer's private memory pool.
+
+  USER (MEM_MZ_USER, 0x16)
+   This object exists in user space.
+
+  BUG (MEM_BUG, 0x17)
+    This object exists within the first page or is an address
+    from which the software can infer no meaning.
+
+  UFO_GLOBAL (MEM_UFO_GLOBAL, 0x18)
+    The object is globally declared, 
+    but is (incorrectly) not in the Memorizer tracking system.
+
+  UFO_NONE (MEM_UFO_NONE, 0x19)
+    Memorizer can infer no information about the object.
+
+  NONE (MEM_NONE, 0x1A)
+    Memorizer can infer no information about the object.
 
 ``command``
   The executable name, excluding the path, of the program running
@@ -468,7 +595,7 @@ in a kmap file.
 
 ``new_alloc_type``
   Every allocator must, itself, be a client of a more generic
-  allocator.  For example, ``kmalloc`` might gets its memory from
+  allocator.  For example, ``kmalloc`` might get its memory from
   ``__alloc_pages``. When that happens, the allocation kmap
   entry for the more generic allocation will include the
   ``alloc_type`` of the more specific allocation in this