doc update

Author: jfennick

.gitignore

@@ -18,10 +18,6 @@ gromacs_mdp.html
 NCI*
 error_*.txt
 validation_*.txt
-cwl_dirs.txt
-yml_dirs.txt
-inference_rules.txt
-renaming_conventions.txt
 .hypothesis/
 .env
 node_modules/

BPS_poster.svg

@@ -2,8 +2,62 @@
 
 This repository contains workflows for various molecular modeling tasks. The workflows can be compiled & executed using the [Workflow Inference Compiler](https://github.com/PolusAI/workflow-inference-compiler).
 
+![BPS Poster](BPS_poster.svg)
+
 ## Quick Start
-Follow the [installation instructions](https://github.com/PolusAI/workflow-inference-compiler#quick-start) here.
+First, follow the [installation instructions](https://github.com/PolusAI/workflow-inference-compiler#quick-start) for WIC.
+
+Then, clone this repository and run the following commands:
+```
+cd install
+./install_biobb_adapters.sh
+cd ..
+mm-workflows --generate_schemas
+wic --generate_schemas
+```
+
+Some of the workflows require an Nvidia CUDA GPU. Please see the Nvidia [installation guides](https://docs.nvidia.com/cuda/#installation-guides) for more information.
+(Moreover, you'll also need to configure it to work with docker. Good luck!)
+
+If all goes well, then you can try running the tutorial, which is based on the following [gromacs tutorial](https://mmb.irbbarcelona.org/biobb/availability/tutorials/cwl).
+```
+wic --yaml ../mm-workflows/examples/gromacs/tutorial.wic --graphviz --run_local --quiet
+```
+
+That last command will infer edges, compile to CWL, generate a GraphViz diagram of the root workflow, and run it locally.
+
+```yaml
+label: Conjugate Gradient
+steps:
+  - grompp:
+      in:
+        config: !ii
+          mdp:
+            integrator: cg
+            nsteps: 1000
+  - mdrun:
+      in:
+        # Use GPU by default
+        bonded_terms: !ii cpu
+        pme_terms: !ii cpu
+  - gmx_energy:
+      in:
+        config: !ii
+          terms: [Potential]
+        output_xvg_path: !ii energy_min_cg.xvg
+```
+The subworkflow [`examples/gromacs/cg.wic`](https://github.com/PolusAI/mm-workflows/blob/main/examples/gromacs/cg.wic) in `mm-workflows` is shown above, and the GraphViz diagram of the root workflow [`examples/gromacs/tutorial.wic`](https://github.com/PolusAI/mm-workflows/blob/main/examples/gromacs/tutorial.wic) in `mm-workflows` is shown below.
+
+![Workflow](examples/gromacs/tutorial.wic.gv.png)
+
+If you add the --parallel flag to the above command then, in another terminal, you can view the plots in real-time:
+```
+conda activate wic
+cd install && ./install_timeseriesplots.sh && cd ..
+timeseriesplots
+```
+
+![Plots](examples/gromacs/plots.png)
 
 ## Jupyter notebook visualization
 
@@ -18,3 +72,10 @@ pip install -e ".[all]"
 ```
 
 ![Plots](docs/tree_viewer.png)
+
+
+## Visualizing the results
+
+This particular workflow creates files which represent 3D coordinates, so we can view them in the Jupyter notebook `src/vis/viewer.ipynb`. Make sure you are using the `vis` conda environment as mentioned in the installation guide.
+
+![Multistep](protein.png)

README.md

@@ -0,0 +1,116 @@
+# Advanced Features
+
+## Static dispatch
+
+Here is an example that shows how to swap out constant pressure implementations.
+
+```yaml
+wic:
+  default_backend: gromacs
+  backends:
+    gromacs:
+      steps:
+        - npt_gromacs.wic:
+    amber:
+      steps:
+        - npt_amber.wic:
+  graphviz:
+    label: Constant Pressure
+```
+
+Then you just need to choose a specific implementation at the call site:
+
+```yaml
+steps:
+  - nvt.wic:
+  - npt.wic:
+
+wic:
+  graphviz:
+    label: Equilibration
+  steps:
+    (2, npt.wic):
+      wic:
+        backend: amber
+```
+This will override the default implementation of `gromacs` and use `amber`. This really just means that `npt_amber.wic` is called instead of `npt_gromacs.wic` (If `--insert_steps_automatically` is enabled, the compiler will attempt to automatically insert the necessary file format conversions as determined below.)
+
+## Subinterpreters
+
+A portion of [`examples/gromacs/nvt.wic`](https://github.com/PolusAI/mm-workflows/blob/main/examples/gromacs/nvt.wic) in `mm-workflows` is shown below. You can see that the `in:` tag of gmx_energy is identical to the `config:` tag of cwl_watcher. This currently needs to be manually copy & pasted (and indented), but it should be possible to automatically do this in the future.
+
+```yaml
+...
+  - mdrun:
+      out:
+      - output_edr_path: !& nvt.edr # Explicit edge reference / anchor
+        # (This edge can be inferred, but made explicit for demonstration purposes.)
+  - gmx_energy:
+      in:
+        input_energy_path: !* nvt.edr # Explicit edge dereference / alias
+        config: !ii
+          terms: [Temperature]
+        output_xvg_path: temperature.xvg
+# NOTE: explicit edges are not supported with cwl_watcher, and all filenames
+# must be globally unique!
+  - cwl_watcher:
+      in:
+        #cachedir_path: /absolute/path/to/cachedir/ (automatically filled in by wic)
+        file_pattern: '*nvt.edr'  # Any strings that start with & or * need to be escaped in quotes
+        cwl_tool: gmx_energy # This can also be an arbitrary subworkflow!
+        max_times: '5'
+        config: !ii
+          in:
+            input_energy_path: '*nvt.edr' # This * is automatically removed.
+            config: !ii
+              terms: [Temperature]
+            output_xvg_path: temperature.xvg
+...
+```
+
+Note that although gmx_energy appears before cwl_watcher in the YAML file, gmx_energy is independent of cwl_watcher in the DAG and thus not considered to be a previous step. We include gmx_energy simply to guarantee that the analysis gets run one more time in the main workflow, when all the files are known to be in their final state.
+
+### Known Issues
+
+Since the two runtimes are not linked, there is not currently a reliable way to determine if the previous steps have finished. Thus, to guarantee termination of the second runtime, we simply execute `cwl_tool` upto `max_times`. We also waive any guarantees about the files, so the subworkflow in the second runtime may of course fail for any number of reasons. Thus, we do not propagate speculative failures up to the main workflow.
+
+The runtime system intentionally hides the working sub-directories of each step. Thus, we are forced to use a file watcher (hence the name cwl_watcher) recursively starting from `cachedir_path`. This is why all filenames used with cwl_watcher must be globally unique. (Actually, for technical reasons we cannot use a file watching library; we simply use a good old fashioned polling loop.)
+
+## Real-time plots
+
+It is assumed that the real-time analysis takes care of the complex log file parsing, etc and produces simple tabular data files (i.e. csv files separated by whitespace instead of a comma). We need to use the same file watching / polling trick as above to locate these tabular data files. The first argument to the following command is the directory in which to look for the files. (By default it is `cachedir` because that is the default value of the  `--cachedir` wic command line argument.) You can also optionally supply the file patterns, which by default are `*.xvg` and `*.dat`.
+
+```
+timeseriesplots cachedir <pat1> <pat2> <...>
+```
+
+## YAML Metadata Annotations
+
+### Overloading / Parameter Passing
+
+This example shows how we can recursively pass in parameters / recursively overload metadata.
+
+Suppose we want to do a very careful minimization, first in vacuum and then in solvent (i.e. [`examples/gromacs/setup_vac_min.wic`](https://github.com/PolusAI/mm-workflows/blob/main/examples/gromacs/setup_vac_min.wic) in `mm-workflows`). We would like to re-use the abstract minimization protocol from `min.wic`. However, our stability analysis requires an explicit edge definition from the final minimized coordinates (i.e. in solvent). If we try to simply add `- output_tpr_path: !& min.tpr` directly to `min.wic`, there will be duplicate definitions! This is not allowed (it will generate an exception).
+
+The solution is to pass in this parameter to only the second instance of `min.wic`.
+
+A portion of [`examples/gromacs/basic.wic`](https://github.com/PolusAI/mm-workflows/blob/main/examples/gromacs/basic.wic) is shown below.
+
+```yaml
+...
+# Put everything under one top-level wic: tag to facilitate easy merging and removal.
+wic:
+  graphviz:
+    label: Molecular Dynamics
+  steps:
+    (1, min.wic):
+      wic:
+        steps:
+          (2, cg.wic):
+            wic:
+              steps:
+                (1, grompp):
+                  out:
+                  - output_tpr_path: !& min.tpr
+...
+```

…s/biosimspace/conversion_amb_gro_zip.cwl …sert_steps_automatically_amb_gro_zip.cwlcwl_adapters/file_format_conversions/biosimspace/conversion_amb_gro_zip.cwl renamed to cwl_adapters/file_format_conversions/biosimspace/insert_steps_automatically_amb_gro_zip.cwl cwl_adapters/file_format_conversions/biosimspace/conversion_amb_gro_zip.cwl renamed to cwl_adapters/file_format_conversions/biosimspace/insert_steps_automatically_amb_gro_zip.cwl

@@ -9,6 +9,7 @@ Molecular Modeling Workflows documentation
    installguide.md
    tutorials/tutorials.rst
    userguide.md
+   advanced.md
    dev/api.rst
 
 Indices and tables

…s/biosimspace/conversion_amb_gro_zip.wic …sert_steps_automatically_amb_gro_zip.wiccwl_adapters/file_format_conversions/biosimspace/conversion_amb_gro_zip.wic renamed to cwl_adapters/file_format_conversions/biosimspace/insert_steps_automatically_amb_gro_zip.wic cwl_adapters/file_format_conversions/biosimspace/conversion_amb_gro_zip.wic renamed to cwl_adapters/file_format_conversions/biosimspace/insert_steps_automatically_amb_gro_zip.wic

@@ -1,3 +1,21 @@
-# Overview
+# Main Features / Design Overview
 
-TODO
+* Subworkflows
+
+Subworkflows are used extensively to create reusable building blocks.
+
+For example, a basic molecular dynamics workflow is composed of minimization, equilibration, and production steps, the equilibration step is composed of constant volume (NVT) and constant pressure (NPT) steps, and each of those are composed of primitive backend-specific steps. If we then want to do a stability analysis, we should be able to incorporate the molecular dynamics workflow as a black box, and we should only have to append the stability analysis subworkflow. See [here](https://github.com/PolusAI/mm-workflows/blob/main/examples/gromacs/tutorial.wic)!
+
+* Multiple backends
+
+There are often several backend engines that implement the same algorithm (e.g. amber / gromacs / namd). In principle, each backend ought to be exactly interchangeable, but in practice backends may randomly crash, etc. For this reason, we want the ability to arbitrarily switch backends at any step. Moreover, different users may be familiar with different backends, but we still want to compose together their workflows.
+
+For example, we should be able to compose system setup using amber/tleap, equilibration using namd, and metadynamics using gromacs. File format conversions should be automatically inserted between steps. This is not possible with other 'backend-independent' software packages which require a single backend to be fixed at the beginning and used throughout.
+
+See [static dispatch](advanced.md#static-dispatch)
+
+* Automated Real-time Analysis & Plots
+
+For quality control purposes, it is highly desirable to have fully automated analyses. This is particularly important for Virtual Screening, where the number of simulations is so large that a user cannot possibly manually inspect each simulation and intervene. For example, a stability analysis ought to be done before an expensive binding free energy calculation is performed.
+
+We support iteratively running an arbitrary analysis workflow in real-time (i.e. while the simulation is still running) and plotting the results. Timeseries data can be automatically segmented and clustered into stastically separate probability distributions, which are beautifully reflected in the associated histograms.