Merge branch 'development-0.3' into docs/random-fixes

# Conflicts:
#	docs/Learning-Environment-Create-New.md

Author: Marwan Mattar

docs/Getting-Started-with-Balance-Ball.md

@@ -325,7 +325,7 @@ the `Internal` Brain mode will only be available once completing these steps.
 
 1. Make sure the TensorFlowSharp plugin is in your `Assets` folder. A Plugins 
 folder which includes TF# can be downloaded 
-[here](https://s3.amazonaws.com/unity-agents/0.2/TFSharpPlugin.unitypackage). 
+[here](https://s3.amazonaws.com/unity-agents/0.3/TFSharpPlugin.unitypackage). 
 Double click and import it once downloaded.  You can see if this was
 successfully installed by checking the TensorFlow files in the Project tab
 under `Assets` -> `ML-Agents` -> `Plugins` -> `Computer`

docs/Learning-Environment-Best-Practices.md

@@ -15,6 +15,7 @@ complexity over time. This can either be done manually, or via Curriculum Learni
 
 ## Vector Observations
 * Vector Observations should include all variables relevant to allowing the agent to take the optimally informed decision.
+* In cases where Vector Observations need to be remembered or compared over time, increase the `Stacked Vectors` value to allow the agent to keep track of multiple observations into the past. 
 * Categorical variables such as type of object (Sword, Shield, Bow) should be encoded in one-hot fashion (i.e. `3` -> `0, 0, 1`).
 * Besides encoding non-numeric values, all inputs should be normalized to be in the range 0 to +1 (or -1 to 1). For example, the `x` position information of an agent where the maximum possible value is `maxValue` should be recorded as `AddVectorObs(transform.position.x / maxValue);` rather than `AddVectorObs(transform.position.x);`. See the equation below for one approach of normalization. 
 * Positional information of relevant GameObjects should be encoded in relative coordinates wherever possible. This is often relative to the agent position.

docs/Learning-Environment-Create-New.md

@@ -57,7 +57,7 @@ Next, we will create a very simple scene to act as our ML-Agents environment. Th
 2. Name the GameObject "Target"
 3. Select Target to view its properties in the Inspector window.
 4. Set Transform to Position = (3,0.5,3), Rotation = (0,0,0), Scale = (1,1,1).
-5. On the Cube's Mesh Renderer, expand the Materials property and change the default-material to *block*.
+5. On the Cube's Mesh Renderer, expand the Materials property and change the default-material to *Block*.
 
 ![The Target Cube in the Inspector window](images/mlagents-NewTutBlock.png)
 
@@ -116,15 +116,13 @@ The default settings for the Academy properties are also fine for this environme
 
 ## Add a Brain
 
-The Brain object encapsulates the decision making process. An Agent sends its observations to its Brain and expects a decision in return. The Brain Type setting determines how the Brain makes decisions. Unlike the Academy and Agent classes, you don't make your own Brain subclasses. (You can extend CoreBrain to make your own *types* of Brain, but the four built-in brain types should cover almost all scenarios.)
+The Brain object encapsulates the decision making process. An Agent sends its observations to its Brain and expects a decision in return. The Brain Type setting determines how the Brain makes decisions. Unlike the Academy and Agent classes, you don't make your own Brain subclasses. 
 
 To create the Brain:
 
-1. Right-click the Academy GameObject in the Hierarchy window and choose *Create Empty* to add a child GameObject.
-2. Name the new GameObject, "Brain".
-3. Select the Brain GameObject to show its properties in the Inspector window.
-4. Click **Add Component**.
-5. Select the **Scripts/Brain** component to add it to the GameObject.
+1. Select the Brain GameObject created earlier to show its properties in the Inspector window.
+2. Click **Add Component**.
+3. Select the **Scripts/Brain** component to add it to the GameObject.
 
 We will come back to the Brain properties later, but leave the Brain Type as **Player** for now.
 
@@ -258,9 +256,9 @@ The final part of the Agent code is the Agent.AgentAction() function, which rece
 
 **Actions**
 
-The decision of the Brain comes in the form of an action array passed to the `AgentAction()` function. The number of elements in this array is determined by the `Vector Action Space Type` and `Vector Action Space Size` settings of the agent's Brain. The RollerAgent uses the continuous vector action space and needs two continuous control signals from the brain. Thus, we will set the Brain `Vector Action Size` to 2. The first element,`action[0]` determines the force applied along the x axis; `action[1]` determines the force applied along the z axis. (If we allowed the agent to move in three dimensions, then we would need to set `Vector Action Size` to 3. Note the Brain really has no idea what the values in the action array mean. The training process adjust the action values in response to the observation input and then sees what kind of rewards it gets as a result. 
+The decision of the Brain comes in the form of an action array passed to the `AgentAction()` function. The number of elements in this array is determined by the `Vector Action Space Type` and `Vector Action Space Size` settings of the agent's Brain. The RollerAgent uses the continuous vector action space and needs two continuous control signals from the brain. Thus, we will set the Brain `Vector Action Size` to 2. The first element,`action[0]` determines the force applied along the x axis; `action[1]` determines the force applied along the z axis. (If we allowed the agent to move in three dimensions, then we would need to set `Vector Action Size` to 3. Note the Brain really has no idea what the values in the action array mean. The training process just adjusts the action values in response to the observation input and then sees what kind of rewards it gets as a result. 
 
-Before we can add a force to the agent, we need a reference to its Rigidbody component. A [Rigidbody](https://docs.unity3d.com/ScriptReference/Rigidbody.html) is Unity's primary element for physics simulation. (See [Physics](https://docs.unity3d.com/Manual/PhysicsSection.html) for full documentation of Unity physics.) A good place to set references to other components of the same GameObject is in the standard Unity `Start()` method:
+The RollerAgent applies the values from the action[] array to its Rigidbody component, `rBody`, using the `Rigidbody.AddForce` function:
 
 
 With the reference to the Rigidbody, the agent can apply the values from the action[] array using the `Rigidbody.AddForce` function:
@@ -383,10 +381,10 @@ Also, drag the Target GameObject from the Hierarchy window to the RollerAgent Ta
 
 Finally, select the Brain GameObject so that you can see its properties in the Inspector window. Set the following properties:
 
+* `Vector Observation Space Type` = **Continuous**
 * `Vector Observation Space Size` = 8
-* `Vector Action Space Size` = 2
 * `Vector Action Space Type` = **Continuous**
-* `Vector Observation Space Type` = **Continuous**
+* `Vector Action Space Size` = 2
 * `Brain Type` = **Player**
 
 Now you are ready to test the environment before training.

docs/Learning-Environment-Design-Brains.md

@@ -23,7 +23,7 @@ The Brain Inspector window in the Unity Editor displays the properties assigned
     * `Vector Observation` 
     	* `Space Type` - Corresponds to whether the observation vector contains a single integer (Discrete) or a series of real-valued floats (Continuous).
     	* `Space Size` - Length of vector observation for brain (In _Continuous_ space type). Or number of possible values (in _Discrete_ space type).
-		* `Stacked Vectors` - The number of previous vector observations that will be stacked before being sent to the brain.
+		* `Stacked Vectors` - The number of previous vector observations that will be stacked and used collectively for decision making. This results in the effective size of the vector observation being passed to the brain being: _Space Size_ x _Stacked Vectors_.
 	* `Visual Observations`	- Describes height, width, and whether to grayscale visual observations for the Brain.
 	* `Vector Action`
 		* `Space Type` - Corresponds to whether action vector contains a single integer (Discrete) or a series of real-valued floats (Continuous).

docs/Training-on-Amazon-Web-Service.md

@@ -1,53 +1,68 @@
 # Training on Amazon Web Service
 
-This page contains instructions for setting up an EC2 instance on Amazon Web Service for use in training ML-Agents environments. Current limitations of the Unity Engine require that a screen be available to render to. In order to make this possible when training on a remote server, a virtual screen is required. We can do this by installing Xorg and creating a virtual screen. Once installed and created, we can display the Unity environment in the virtual environment, and train as we would on a local machine. 
+This page contains instructions for setting up an EC2 instance on Amazon Web Service for training ML-Agents environments. You can run "headless" training if none of the agents in the environment use visual observations. 
 
 ## Pre-Configured AMI
 A public pre-configured AMI is available with the ID: `ami-30ec184a` in the `us-east-1` region. It was created as a modification of the Amazon Deep Learning [AMI](https://aws.amazon.com/marketplace/pp/B01M0AXXQB). 
 
 ## Configuring your own Instance
-Instructions here are adapted from this [Medium post](https://medium.com/towards-data-science/how-to-run-unity-on-amazon-cloud-or-without-monitor-3c10ce022639) on running general Unity applications in the cloud.
 
 1. To begin with, you will need an EC2 instance which contains the latest Nvidia drivers, CUDA8, and cuDNN.  There are a number of external tutorials which describe this, such as:
     * [Getting CUDA 8 to Work With openAI Gym on AWS and Compiling TensorFlow for CUDA 8 Compatibility](https://davidsanwald.github.io/2016/11/13/building-tensorflow-with-gpu-support.html)
     * [Installing TensorFlow on an AWS EC2 P2 GPU Instance](http://expressionflow.com/2016/10/09/installing-tensorflow-on-an-aws-ec2-p2-gpu-instance/)
     * [Updating Nvidia CUDA to 8.0.x in Ubuntu 16.04 – EC2 Gx instance](https://aichamp.wordpress.com/2016/11/09/updating-nvidia-cuda-to-8-0-x-in-ubuntu-16-04-ec2-gx-instance/)
-2. Move `python` to remote instance.
+
+## Installing ML-Agents
+
+2. Move `python` sub-folder of this ml-agents repo to the remote ECS instance, and set it as the working directory.
 2. Install the required packages with `pip3 install .`.
-3. Run the following commands to install Xorg:
+
+## Testing
+
+To verify that all steps worked correctly:
+
+1. In the Unity Editor, load a project containing an ML-Agents environment (you can use one of the example environments if you have not created your own).
+2. Open the Build Settings window (menu: File > Build Settings).
+3. Select Linux as the Target Platform, and x64_86 as the target architecture.
+4. Check Headless Mode (unless you have enabled a virtual screen following the instructions below).
+5. Click Build to build the Unity environment executable.
+6. Upload the executable to your EC2 instance.
+7. Test the instance setup from Python using:
+
+```python
+from unityagents import UnityEnvironment
+
+env = UnityEnvironment(<your_env>)
+```
+Where `<your_env>` corresponds to the path to your environment executable.
+ 
+You should receive a message confirming that the environment was loaded successfully.
+
+## (Optional) Enabling a virtual screen
+
+_Instructions here are adapted from this [Medium post](https://medium.com/towards-data-science/how-to-run-unity-on-amazon-cloud-or-without-monitor-3c10ce022639) on running general Unity applications in the cloud._
+
+Current limitations of the Unity Engine require that a screen be available to render to when using visual observations. In order to make this possible when training on a remote server, a virtual screen is required. We can do this by installing Xorg and creating a virtual screen. Once installed and created, we can display the Unity environment in the virtual environment, and train as we would on a local machine. Ensure that `headless` mode is disabled when building linux executables which use visual observations.
+
+1. Run the following commands to install Xorg:
+
     ```
     sudo apt-get update
     sudo apt-get install -y xserver-xorg mesa-utils
     sudo nvidia-xconfig -a --use-display-device=None --virtual=1280x1024
     ```
-4. Restart the EC2 instance.
 
-## Launching your instance
+2. Restart the EC2 instance.
 
-1. Make sure there are no Xorg processes running. To kill the Xorg processes, run `sudo killall Xorg`.  
+3. Make sure there are no Xorg processes running. To kill the Xorg processes, run `sudo killall Xorg`.  
 Note that you might have to run this command multiple times depending on how Xorg is configured.  
 If you run `nvidia-smi`, you will have a list of processes running on the GPU, Xorg should not be in the list. 
 
-2. Run:
+4. Run:
+
     ```
     sudo /usr/bin/X :0 &
     export DISPLAY=:0
     ```
-3. To ensure the installation was successful, run `glxgears`. If there are no errors, then Xorg is correctly configured.
-4. There is a bug in _Unity 2017.1_ which requires the uninstallation of `libxrandr2`, which can be removed with :
-```
-sudo apt-get remove --purge libwxgtk3.0-0v5
-sudo apt-get remove --purge libxrandr2
-```
-This is scheduled to be fixed in 2017.3.
-
-## Testing
-
-If all steps worked correctly, upload an example binary built for Linux to the instance, and test it from Python with:
-```python
-from unityagents import UnityEnvironment
-
-env = UnityEnvironment(your_env)
-```
-
-You should receive a message confirming that the environment was loaded successfully.
+ 
+5. To ensure the installation was successful, run `glxgears`. If there are no errors, then Xorg is correctly configured.

python/setup.py

@@ -7,13 +7,13 @@
     required = f.read().splitlines()
 
 setup(name='unityagents',
-      version='0.2.0',
+      version='0.3.0',
       description='Unity Machine Learning Agents',
       license='Apache License 2.0',
       author='Unity Technologies',
       author_email='[email protected]',
       url='https://github.com/Unity-Technologies/ml-agents',
-      packages=find_packages(exclude = ['ppo']),
+      packages=find_packages(),
       install_requires = required,
       long_description= ("Unity Machine Learning Agents allows researchers and developers "
        "to transform games and simulations created using the Unity Editor into environments "