Profiling docs (#2325)

* profiling docs

* clean up debug option, move csv info

* Imitation Learning -> Behavioral Cloning

Author: Chris Elion

docs/Profiling.md

@@ -0,0 +1,53 @@
+# Profiling ML-Agents in Python
+
+ML-Agents provides a lightweight profiling system, in order to identity hotspots in the training process and help spot
+regressions from changes.
+
+Timers are hierarchical, meaning that the time tracked in a block of code can be further split into other blocks if
+desired. This also means that a function that is called from multiple places in the code will appear in multiple
+places in the timing output.
+
+All timers operate using a "global" instance by default, but this can be overridden if necessary (mainly for testing).
+
+## Adding Profiling
+
+There are two ways to indicate code should be included in profiling. The simplest way is to add the `@timed`
+decorator to a function or method of interested.
+
+```python
+class TrainerController:
+    # ....
+    @timed
+    def advance(self, env: EnvManager) -> int:
+        # do stuff
+```
+
+You can also used the `hierarchical_timer` context manager.
+
+
+``` python
+with hierarchical_timer("communicator.exchange"):
+    outputs = self.communicator.exchange(step_input)
+```
+
+The context manager may be easier than the `@timed` decorator for profiling different parts of a large function, or
+profiling calls to abstract methods that might not use decorator.
+
+## Output
+By default, at the end of training, timers are collected and written in json format to
+`{summaries_dir}/{run_id}_timers.json`. The output consists of node objects with the following keys:
+ * name (string): The name of the block of code.
+ * total (float): The total time in seconds spent in the block, including child calls.
+ * count (int): The number of times the block was called.
+ * self (float): The total time in seconds spent in the block, excluding child calls.
+ * children (list): A list of child nodes.
+ * is_parallel (bool): Indicates that the block of code was executed in multiple threads or processes (see below). This
+ is optional and defaults to false.
+
+### Parallel execution
+For code that executes in multiple processes (for example, SubprocessEnvManager), we periodically send the timer
+information back to the "main" process, aggregate the timers there, and flush them in the subprocess. Note that
+(depending on the number of processes) this can result in timers where the total time may exceed the parent's total
+time. This is analogous to the difference between "real" and "user" values reported from the unix `time` command. In the
+timer output, blocks that were run in parallel are indicated by the `is_parallel` flag.
+

docs/Training-ML-Agents.md

@@ -144,10 +144,7 @@ environment, you can set the following command line options when invoking
   training doesn't involve visual observations (reading from Pixels). See
   [here](https://docs.unity3d.com/Manual/CommandLineArguments.html) for more
   details.
-* `--debug` - Specify this option to run ML-Agents in debug mode and log Trainer
-  Metrics to a CSV stored in the `summaries` directory. The metrics  stored are:
-  brain name, time to update policy, time since start of training, time for last experience collection, number of experiences used for training, mean return. This
-  option is not available currently for Imitation Learning.
+* `--debug` - Specify this option to enable debug-level logging for some parts of the code.
 
 ### Training config file
 
@@ -204,3 +201,16 @@ You can also compare the
 to the corresponding sections of the `config/trainer_config.yaml` file for each
 example to see how the hyperparameters and other configuration variables have
 been changed from the defaults.
+
+### Output metrics
+Trainer Metrics are logged to a CSV stored in the `summaries` directory. The metrics stored are:
+  * brain name
+  * time to update policy
+  * time since start of training
+  * time for last experience collection
+  * number of experiences used for training
+  * mean return 
+  
+This option is not available currently for Behavioral Cloning.
+
+[Profiling](Profiling.md) information is also saved in the `summaries` directory.