Merge branch 'master' into development

Author: awjuliani

docs/Getting-Started-with-Balance-Ball.md

@@ -2,9 +2,9 @@
 
 ![Balance Ball](../images/balance.png)
 
-This tutorial will walk through the end-to-end process of installing Unity Agents, building an example environment, training an agent in it, and finally embedding the trained model into the Unity environment. 
+This tutorial will walk through the end-to-end process of installing Unity Agents, building an example environment, training an agent in it, and finally embedding the trained model into the Unity environment.
 
-Unity ML Agents contains a number of example environments which can be used as templates for new environments, or as ways to test a new ML algorithm to ensure it is functioning correctly. 
+Unity ML Agents contains a number of example environments which can be used as templates for new environments, or as ways to test a new ML algorithm to ensure it is functioning correctly.
 
 In this walkthrough we will be using the **3D Balance Ball** environment. The environment contains a number of platforms and balls. Platforms can act to keep the ball up by rotating either horizontally or vertically. Each platform is an agent which is rewarded the longer it can keep a ball balanced on it, and provided a negative reward for dropping the ball. The goal of the training process is to have the platforms learn to never drop the ball.
 
@@ -15,7 +15,7 @@ Let's get started!
 In order to install and set-up the Python and Unity environments, see the instructions [here](installation.md).
 
 ## Building Unity Environment
-Launch the Unity Editor, and log in, if necessary. 
+Launch the Unity Editor, and log in, if necessary.
 
 1. Open the `unity-environment` folder using the Unity editor.  *(If this is not first time running Unity, you'll be able to skip most of these immediate steps, choose directly from the list of recently opened projects)*
     - On the initial dialog, choose `Open` on the top options
@@ -38,24 +38,24 @@ Launch the Unity Editor, and log in, if necessary.
 
 To launch jupyter, run in the command line:
 
-`jupyter notebook` 
+`jupyter notebook`
 
 Then navigate to `localhost:8888` to access the notebooks. If you're new to jupyter, check out the [quick start guide](https://jupyter-notebook-beginner-guide.readthedocs.io/en/latest/execute.html) before you continue.
 
 To ensure that your environment and the Python API work as expected, you can use the `python/Basics` Jupyter notebook. This notebook contains a simple walkthrough of the functionality of the API. Within `Basics`, be sure to set `env_name` to the name of the environment file you built earlier.
 
 ### Training with PPO
-In order to train an agent to correctly balance the ball, we will use a Reinforcement Learning algorithm called Proximal Policy Optimization (PPO). This is a method that has been shown to be safe, efficient, and more general purpose than many other RL algorithms, as such we have chosen it as the example algorithm for use with ML Agents. For more information on PPO, OpenAI has a recent [blog post](https://blog.openai.com/openai-baselines-ppo/) explaining it. 
+In order to train an agent to correctly balance the ball, we will use a Reinforcement Learning algorithm called Proximal Policy Optimization (PPO). This is a method that has been shown to be safe, efficient, and more general purpose than many other RL algorithms, as such we have chosen it as the example algorithm for use with ML Agents. For more information on PPO, OpenAI has a recent [blog post](https://blog.openai.com/openai-baselines-ppo/) explaining it.
 
 In order to train the agents within the Ball Balance environment:
 
 1. Open `python/PPO.ipynb` notebook from Jupyter.
-2. Set `env_name` to whatever you named your environment file.
-3. (optional) Set `run_path` directory to your choice. 
+2. Set `env_name` to the name of your environment file earlier.
+3. (optional) Set `run_path` directory to your choice.
 4. Run all cells of notebook with the exception of the last one under "Export the trained Tensorflow graph."
 
 ### Observing Training Progress
-In order to observe the training process in more detail, you can use Tensorboard. 
+In order to observe the training process in more detail, you can use Tensorboard.
 In your command line, enter into `python` directory and then run :
 
 `tensorboard --logdir=summaries`
@@ -73,28 +73,28 @@ From Tensorboard, you will see the summary statistics of six variables:
 ## Embedding Trained Brain into Unity Environment _[Experimental]_
 Once the training process displays an average reward of ~75 or greater, and there has been a recently saved model (denoted by the `Saved Model` message) you can choose to stop the training process by stopping the cell execution. Once this is done, you now have a trained TensorFlow model. You must now convert the saved model to a Unity-ready format which can be embedded directly into the Unity project by following the steps below.
 
-### Setting up TensorFlowSharp Support 
+### Setting up TensorFlowSharp Support
 Because TensorFlowSharp support is still experimental, it is disabled by default. In order to enable it, you must follow these steps. Please note that the `Internal` Brain mode will only be available once completing these steps.
 
 1. Make sure you are using Unity 2017.1 or newer.
-2. Make sure the TensorFlowSharp plugin is in your Asset folder. A Plugins folder which includes TF# can be downloaded [here](https://s3.amazonaws.com/unity-agents/TFSharpPlugin.unitypackage).
+2. Make sure the TensorFlowSharp plugin is in your `Assets` folder. A Plugins folder which includes TF# can be downloaded [here](https://s3.amazonaws.com/unity-agents/TFSharpPlugin.unitypackage). Double click and import it once downloaded.
 3. Go to `Edit` -> `Project Settings` -> `Player`
 4. For each of the platforms you target (**`PC, Mac and Linux Standalone`**, **`iOS`** or **`Android`**):   
 	1. Go into `Other Settings`.
-	2. Select `Scripting Runtime Version` to `Experimental (.NET 4.6 Equivalent)` 
+	2. Select `Scripting Runtime Version` to `Experimental (.NET 4.6 Equivalent)`
 	3. In `Scripting Defined Symbols`, add the flag `ENABLE_TENSORFLOW`
 5. Restart the Unity Editor.
 
 ### Embedding the trained model into Unity
 
-1. Run the final cell of the notebook under "Export the trained TensorFlow graph" to produce an `<env_name >.bytes` file. 
-2. Move `<env_name>.bytes` from `python/models/...` into `unity-environment/Assets/ML-Agents/Examples/3DBall/TFModels/`.
-3. Open the Unity Editor, and select the `3DBall` scene as described above. 
-4. Select the `3DBallBrain` object from the Scene hierarchy. 
+1. Run the final cell of the notebook under "Export the trained TensorFlow graph" to produce an `<env_name >.bytes` file.
+2. Move `<env_name>.bytes` from `python/models/ppo/` into `unity-environment/Assets/ML-Agents/Examples/3DBall/TFModels/`.
+3. Open the Unity Editor, and select the `3DBall` scene as described above.
+4. Select the `Ball3DBrain` object from the Scene hierarchy.
 5. Change the `Type of Brain` to `Internal`.
 6. Drag the `<env_name>.bytes` file from the Project window of the Editor to the `Graph Model` placeholder in the `3DBallBrain` inspector window.
-7. Set the `Graph Placeholder` size to 1 (_Note that step 7 and 8 are done because 3DBall is a continuous control environment, and the TensorFlow model requires a noise parameter to decide actions. In cases with discrete control, epsilon is not needed_). 
-8. Add a placeholder called `epsilon` with a type of `floating point` and a range of values from 0 to 0.
+7. Set the `Graph Placeholder` size to 1 (_Note that step 7 and 8 are done because 3DBall is a continuous control environment, and the TensorFlow model requires a noise parameter to decide actions. In cases with discrete control, epsilon is not needed_).
+8. Add a placeholder called `epsilon` with a type of `floating point` and a range of values from `0` to `0`.
 9. Press the Play button at the top of the editor.
 
-If you followed these steps correctly, you should now see the trained model being used to control the behavior of the balance ball within the Editor itself. From here you can re-build the Unity binary, and run it standalone with your agent's new learned behavior built right in. 
+If you followed these steps correctly, you should now see the trained model being used to control the behavior of the balance ball within the Editor itself. From here you can re-build the Unity binary, and run it standalone with your agent's new learned behavior built right in.

docs/Limitations-&-Common-Issues.md

@@ -40,7 +40,7 @@ If you receive a file-not-found error while attempting to launch an environment,
 
 If you receive an exception `"Couldn't launch new environment because communication port {} is still in use. "`, you can change the worker number in the python script when calling 
 
-`UnityEnvironment(file_name=filename, worker_num=X)`
+`UnityEnvironment(file_name=filename, worker_id=X)`
 
 ### Mean reward : nan
 

docs/Unity-Agents---Python-API.md

@@ -20,7 +20,7 @@ env = UnityEnvironment(file_name=filename, worker_num=0)
 A BrainInfo object contains the following fields:
 
 * **`observations`** : A list of 4 dimensional numpy arrays. Matrix n of the list corresponds to the n<sup>th</sup> observation of the brain. 
-* **`states`** : A two dimensional numpy array of dimension `(batch size, state size)` if the state space is continuous and `(batch size, state size)` if the state space is discrete.
+* **`states`** : A two dimensional numpy array of dimension `(batch size, state size)` if the state space is continuous and `(batch size, 1)` if the state space is discrete.
 * **`memories`** : A two dimensional numpy array of dimension `(batch size, memory size)` which corresponds to the memories sent at the previous step.
 * **`rewards`** : A list as long as the number of agents using the brain containing the rewards they each obtained at the previous step. 
 * **`local_done`** : A list as long as the number of agents using the brain containing  `done` flags (wether or not the agent is done). 

docs/best-practices-ppo.md

@@ -1,6 +1,6 @@
 # Best Practices when training with PPO
 
-The process of training a Reinforcement Learning model can often involve the need to tune the hyperparameters in order to achieve 
+The process of training a Reinforcement Learning model can often involve the need to tune the hyperparameters in order to achieve
 a level of performance that is desirable. This guide contains some best practices for tuning the training process when the default
 parameters don't seem to be giving the level of performance you would like.
 
@@ -72,7 +72,7 @@ Typical Range: `5e5 - 1e7`
 
 ## Training Statistics
 
-To view training statistics, use Tensorboard. For information on launching and using Tensorboard, see [here](../Getting-Started-with-Balance-Ball.md#observing-training-progress).
+To view training statistics, use Tensorboard. For information on launching and using Tensorboard, see [here](./Getting-Started-with-Balance-Ball.md#observing-training-progress).
 
 ### Cumulative Reward
 

docs/installation.md

@@ -13,7 +13,7 @@ In order to train an agent within the framework, you will need to install Python
 
 ### Windows Users
 
-If you are a Windows user who is new to Python/TensorFlow, follow [this guide](https://nitishmutha.github.io/tensorflow/2017/01/22/TensorFlow-with-gpu-for-windows.html) to set up your Python environment.
+If you are a Windows user who is new to Python/TensorFlow, follow [this guide](https://unity3d.college/2017/10/25/machine-learning-in-unity3d-setting-up-the-environment-tensorflow-for-agentml-on-windows-10/) to set up your Python environment.
 
 ### Requirements
 * Jupyter