📝 Document expected_outputs.log (#273)

Author: shnizzedy

docs/_sources/_static/flowcharts/getLogger.svg

@@ -575,3 +575,15 @@ def _unireplace(release_note, unireplace):
 """.format(
     versions=', '.join(gh_tags[:5])
 )
+
+def setup(app):
+    from CPAC.utils.monitoring import custom_logging
+
+    # initilaize class to make factory functions available to Sphinx
+    ml = custom_logging.MockLogger('test', 'test.log', 0, '/tmp')
+    for method in [
+        method for method in
+        set(dir(ml)) - set(dir(custom_logging.MockLogger)) if
+        method not in ['name', 'handlers']
+    ]:
+        setattr(custom_logging.MockLogger, method, getattr(ml, method))

docs/_sources/conf.py

@@ -19,6 +19,7 @@ Contents:
    pipeline
    nodes
    random_state
+   logging
    workflow_documentation
    workflows/index
    xcpqc

docs/_sources/developer/index.rst

@@ -0,0 +1,34 @@
+Logging
+=======
+
+Most C-PAC logging is handled by the built-in Python ``logging`` library. When logging, take care to choose an appropriate ``getLogger`` function or method. While :py:func:`CPAC.utils.monitoring.custom_logging.getLogger` and ``nipype.utils.logger.Logging.getLogger`` each fall back on ``logging.getLogger``,
+
+.. raw:: html
+
+    <div class="flowchart-container"><object data="../_static/flowcharts/getLogger.svg" type="image/svg+xml"></object></div>
+
+
+each have their own intended use cases.
+
+    Loggers are singletons that are never freed during a script execution, and so creating lots of loggers will use up memory which can’t then be freed.
+
+    -- `Logging Cookbook: Creating a lot of loggers <https://docs.python.org/3/howto/logging-cookbook.html#creating-a-lot-of-loggers>`_
+
+``CPAC.utils.monitoring.custom_logging.getLogger`` will look for a :py:class:`CPAC.utils.monitoring.custom_logging.MockLogger` before falling back on ``logging.getLogger``.
+
+A ``CPAC.utils.monitoring.custom_logging.MockLogger`` can be used in place of a :py:class:`logging.Logger` and deleted from memory when no longer needed. For an example, see how ``missing_log`` is created and deleted in :py:func:`CPAC.pipeline.check_outputs.check_outputs`.
+
+``nipype.utils.logger.Logging.getLogger`` will only consider loggers that are part of the class instance calling the method.
+
+.. autofunction:: CPAC.utils.monitoring.custom_logging.getLogger
+
+.. autoclass:: CPAC.utils.monitoring.custom_logging.MockLogger
+   :special-members: __init__
+   :members:
+   :inherited-members:
+
+.. automethod:: nipype.utils.logger.Logging.getLogger
+
+.. automethod:: nipype.utils.logger.Logging.__init__
+
+.. autofunction:: logging.getLogger

docs/_sources/developer/logging.rst

@@ -12,14 +12,18 @@ The output structure of C-PAC ≥ 1.8.0 is based on `the Brain Imaging Data Stru
 
    └── pipeline-name
        ├── log
-       │   └── pipeline_pipeline-name
-       │       └── subject_session
+       │   └── pipeline_pipeline-name
+       │       └── subject_session
        └── output
            └── cpac_pipeline-name
                └── subject_session
                    ├── anat
                    └── func
 
+
+Beginning with C-PAC v1.8.4, C-PAC generates a YAML file called ``expected_outputs.yml`` near the beginning of each participant run. This file is stored in the participant's log directory and lists derivative modalities (e.g., ``anat``, ``func``) and partial BIDS filenames for outputs expected to be generated and stored for each modality given the provided data and pipeline configuration. At the end of the run (or when crashing gracefully), C-PAC will check the output directory for the presence of all expected outputs. If any expected outputs are missing, C-PAC will generate a ``missing_ouputs.yml`` file listing those missing outputs, also in the participant's log directory.
+
+
 .. note::
 
    If any nodes use more memory at runtime than C-PAC estimates, C-PAC will report those instances near the end of the log in the terminal and in a log file called ``callback.log.resource_overusage.txt`` (beginning in v1.8.0). Please `report excessive memory usage <https://github.com/FCP-INDI/C-PAC/issues/new>`_ to the C-PAC team.

docs/_sources/user/output_dir.rst

@@ -1,3 +1,4 @@
+docutils<0.18
 m2r
 mistune==0.8.4
 numpydoc