You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[machine learning background page](Background-Machine-Learning.md), many of the
5
-
algorithms we provide in the ML-Agents Toolkit leverage some form of deep
6
-
learning. More specifically, our implementations are built on top of the
7
-
open-source library [PyTorch](https://pytorch.org/). In this page we
8
-
provide a brief overview of PyTorch and TensorBoard
9
-
that we leverage within the ML-Agents Toolkit.
3
+
As discussed in our [machine learning background page](Background-Machine-Learning.md), many of the algorithms we provide in the ML-Agents Toolkit leverage some form of deep learning. More specifically, our implementations are built on top of the open-source library [PyTorch](https://pytorch.org/). In this page we provide a brief overview of PyTorch and TensorBoard that we leverage within the ML-Agents Toolkit.
10
4
11
5
## PyTorch
12
6
13
-
[PyTorch](https://pytorch.org/) is an open source library for
14
-
performing computations using data flow graphs, the underlying representation of
15
-
deep learning models. It facilitates training and inference on CPUs and GPUs in
16
-
a desktop, server, or mobile device. Within the ML-Agents Toolkit, when you
17
-
train the behavior of an agent, the output is a model (.onnx) file that you can
18
-
then associate with an Agent. Unless you implement a new algorithm, the use of
19
-
PyTorch is mostly abstracted away and behind the scenes.
7
+
[PyTorch](https://pytorch.org/) is an open source library for performing computations using data flow graphs, the underlying representation of deep learning models. It facilitates training and inference on CPUs and GPUs in a desktop, server, or mobile device. Within the ML-Agents Toolkit, when you train the behavior of an agent, the output is a model (.onnx) file that you can then associate with an Agent. Unless you implement a new algorithm, the use of PyTorch is mostly abstracted away and behind the scenes.
20
8
21
9
## TensorBoard
22
10
23
-
One component of training models with PyTorch is setting the values of
24
-
certain model attributes (called _hyperparameters_). Finding the right values of
25
-
these hyperparameters can require a few iterations. Consequently, we leverage a
One component of training models with PyTorch is setting the values of certain model attributes (called _hyperparameters_). Finding the right values of these hyperparameters can require a few iterations. Consequently, we leverage a visualization tool called [TensorBoard](https://www.tensorflow.org/tensorboard). It allows the visualization of certain agent attributes (e.g. reward) throughout training which can be helpful in both building intuitions for the different hyperparameters and setting the optimal values for your Unity environment. We provide more details on setting the hyperparameters in the [Training ML-Agents](Training-ML-Agents.md) page. If you are unfamiliar with TensorBoard we recommend our guide on [using TensorBoard with ML-Agents](Using-Tensorboard.md) or this [tutorial](https://github.com/dandelionmane/tf-dev-summit-tensorboard-tutorial).
is a fantastic resource to learn all the basic concepts of Unity to get started
8
-
with the ML-Agents Toolkit:
3
+
If you are not familiar with the [Unity Engine](https://unity3d.com/unity), we highly recommend the [Unity Manual](https://docs.unity3d.com/Manual/index.html) and [Tutorials page](https://unity3d.com/learn/tutorials). The [Roll-a-ball tutorial](https://learn.unity.com/project/roll-a-ball) is a fantastic resource to learn all the basic concepts of Unity to get started with the ML-Agents Toolkit:
Copy file name to clipboardExpand all lines: com.unity.ml-agents/Documentation~/Custom-GridSensors.md
+5-10Lines changed: 5 additions & 10 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -8,23 +8,19 @@ One extra feature with Grid Sensors is that you can derive from the Grid Sensor
8
8
To create a custom grid sensor, you'll need to derive from two classes: `GridSensorBase` and `GridSensorComponent`.
9
9
10
10
## Deriving from `GridSensorBase`
11
-
This is the implementation of your sensor. This defines how your sensor process detected colliders,
12
-
what the data looks like, and how the observations are constructed from the detected objects.
13
-
Consider overriding the following methods depending on your use case:
11
+
This is the implementation of your sensor. This defines how your sensor process detected colliders, what the data looks like, and how the observations are constructed from the detected objects. Consider overriding the following methods depending on your use case:
14
12
*`protected virtual int GetCellObservationSize()`: Return the observation size per cell. Default to `1`.
15
13
*`protected virtual void GetObjectData(GameObject detectedObject, int tagIndex, float[] dataBuffer)`: Constructs observations from the detected object. The input provides the detected GameObject and the index of its tag (0-indexed). The observations should be written to the given `dataBuffer` and the buffer size is defined in `GetCellObservationSize()`. This data will be gathered from each cell and sent to the trainer as observation.
16
14
*`protected virtual bool IsDataNormalized()`: Return whether the observation is normalized to 0~1. This affects whether you're able to use compressed observations as compressed data only supports normalized data. Return `true` if all the values written in `GetObjectData` are within the range of (0, 1), otherwise return `false`. Default to `false`.
17
15
18
-
There might be cases when your data is not in the range of (0, 1) but you still wish to use compressed data to speed up training. If your data is naturally bounded within a range, normalize your data first to the possible range and fill the buffer with normalized data. For example, since the angle of rotation is bounded within `0 ~ 360`, record an angle `x` as `x/360` instead of `x`. If your data value is not bounded (position, velocity, etc.), consider setting a reasonable min/max value and use that to normalize your data.
16
+
There might be cases when your data is not in the range of (0, 1) but you still wish to use compressed data to speed up training. If your data is naturally bounded within a range, normalize your data first to the possible range and fill the buffer with normalized data. For example, since the angle of rotation is bounded within `0 ~ 360`, record an angle `x` as `x/360` instead of `x`. If your data value is not bounded (position, velocity, etc.), consider setting a reasonable min/max value and use that to normalize your data.
19
17
*`protected internal virtual ProcessCollidersMethod GetProcessCollidersMethod()`: Return the method to process colliders detected in a cell. This defines the sensor behavior when multiple objects with detectable tags are detected within a cell.
20
-
Currently two methods are provided:
18
+
Currently, two methods are provided:
21
19
*`ProcessCollidersMethod.ProcessClosestColliders` (Default): Process the closest collider to the agent. In this case each cell's data is represented by one object.
22
20
*`ProcessCollidersMethod.ProcessAllColliders`: Process all detected colliders. This is useful when the data from each cell is additive, for instance, the count of detected objects in a cell. When using this option, the input `dataBuffer` in `GetObjectData()` will contain processed data from other colliders detected in the cell. You'll more likely want to add/subtract values from the buffer instead of overwrite it completely.
23
21
24
22
## Deriving from `GridSensorComponent`
25
-
To create your sensor, you need to override the sensor component and add your sensor to the creation.
26
-
Specifically, you need to override `GetGridSensors()` and return an array of grid sensors you want to use in the component.
27
-
It can be used to create multiple different customized grid sensors, or you can also include the ones provided in our package (listed in the next section).
23
+
To create your sensor, you need to override the sensor component and add your sensor to the creation. Specifically, you need to override `GetGridSensors()` and return an array of grid sensors you want to use in the component. It can be used to create multiple different customized grid sensors, or you can also include the ones provided in our package (listed in the next section).
28
24
29
25
Example:
30
26
```csharp
@@ -38,8 +34,7 @@ public class CustomGridSensorComponent : GridSensorComponent
38
34
```
39
35
40
36
## Grid Sensor Types
41
-
Here we list out two types of grid sensor provided in the package: `OneHotGridSensor` and `CountingGridSensor`.
42
-
Their implementations are also a good reference for making you own ones.
37
+
Here we list two types of grid sensor provided in the package: `OneHotGridSensor` and `CountingGridSensor`. Their implementations are also a good reference for making you own ones.
43
38
44
39
### OneHotGridSensor
45
40
This is the default sensor used by `GridSensorComponent`. It detects objects with detectable tags and the observation is the one-hot representation of the detected tag index.
Copy file name to clipboardExpand all lines: com.unity.ml-agents/Documentation~/Custom-SideChannels.md
+18-53Lines changed: 18 additions & 53 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -1,77 +1,55 @@
1
1
# Custom Side Channels
2
2
3
-
You can create your own side channel in C# and Python and use it to communicate
4
-
custom data structures between the two. This can be useful for situations in
5
-
which the data to be sent is too complex or structured for the built-in
3
+
You can create your own side channel in C# and Python and use it to communicate custom data structures between the two. This can be useful for situations in which the data to be sent is too complex or structured for the built-in
6
4
`EnvironmentParameters`, or is not related to any specific agent, and therefore
7
5
inappropriate as an agent observation.
8
6
9
7
## Overview
10
8
11
-
In order to use a side channel, it must be implemented as both Unity and Python
12
-
classes.
9
+
In order to use a side channel, it must be implemented as both Unity and Python classes.
13
10
14
11
### Unity side
15
12
16
-
The side channel will have to implement the `SideChannel` abstract class and the
17
-
following method.
13
+
The side channel will have to implement the `SideChannel` abstract class and the following method.
18
14
19
15
-`OnMessageReceived(IncomingMessage msg)` : You must implement this method and
20
-
read the data from IncomingMessage. The data must be read in the order that it
21
-
was written.
16
+
read the data from IncomingMessage. The data must be read in the order that it was written.
22
17
23
18
The side channel must also assign a `ChannelId` property in the constructor. The
24
19
`ChannelId` is a Guid (or UUID in Python) used to uniquely identify a side
25
-
channel. This Guid must be the same on C# and Python. There can only be one side
26
-
channel of a certain id during communication.
20
+
channel. This Guid must be the same on C# and Python. There can only be one side channel of a certain id during communication.
27
21
28
-
To send data from C# to Python, create an `OutgoingMessage` instance, add data
29
-
to it, call the `base.QueueMessageToSend(msg)` method inside the side channel,
30
-
and call the `OutgoingMessage.Dispose()` method.
22
+
To send data from C# to Python, create an `OutgoingMessage` instance, add data to it, call the `base.QueueMessageToSend(msg)` method inside the side channel, and call the `OutgoingMessage.Dispose()` method.
31
23
32
24
To register a side channel on the Unity side, call
33
25
`SideChannelManager.RegisterSideChannel` with the side channel as only argument.
34
26
35
27
### Python side
36
28
37
-
The side channel will have to implement the `SideChannel` abstract class. You
38
-
must implement :
29
+
The side channel will have to implement the `SideChannel` abstract class. You must implement :
39
30
40
31
-`on_message_received(self, msg: "IncomingMessage") -> None` : You must
41
-
implement this method and read the data from IncomingMessage. The data must be
42
-
read in the order that it was written.
32
+
implement this method and read the data from IncomingMessage. The data must be read in the order that it was written.
43
33
44
-
The side channel must also assign a `channel_id` property in the constructor.
45
-
The `channel_id` is a UUID (referred in C# as Guid) used to uniquely identify a
46
-
side channel. This number must be the same on C# and Python. There can only be
47
-
one side channel of a certain id during communication.
34
+
The side channel must also assign a `channel_id` property in the constructor. The `channel_id` is a UUID (referred in C# as Guid) used to uniquely identify a side channel. This number must be the same on C# and Python. There can only be one side channel of a certain id during communication.
48
35
49
-
To assign the `channel_id` call the abstract class constructor with the
50
-
appropriate `channel_id` as follows:
36
+
To assign the `channel_id` call the abstract class constructor with the appropriate `channel_id` as follows:
51
37
52
38
```python
53
39
super().__init__(my_channel_id)
54
40
```
55
41
56
-
To send a byte array from Python to C#, create an `OutgoingMessage` instance,
57
-
add data to it, and call the `super().queue_message_to_send(msg)` method inside
58
-
the side channel.
42
+
To send a byte array from Python to C#, create an `OutgoingMessage` instance, add data to it, and call the `super().queue_message_to_send(msg)` method inside the side channel.
59
43
60
-
To register a side channel on the Python side, pass the side channel as argument
61
-
when creating the `UnityEnvironment` object. One of the arguments of the
62
-
constructor (`side_channels`) is a list of side channels.
44
+
To register a side channel on the Python side, pass the side channel as argument when creating the `UnityEnvironment` object. One of the arguments of the constructor (`side_channels`) is a list of side channels.
63
45
64
46
## Example implementation
65
47
66
-
Below is a simple implementation of a side channel that will exchange ASCII
67
-
encoded strings between a Unity environment and Python.
48
+
Below is a simple implementation of a side channel that will exchange ASCII encoded strings between a Unity environment and Python.
68
49
69
50
### Example Unity C# code
70
51
71
-
The first step is to create the `StringLogSideChannel` class within the Unity
72
-
project. Here is an implementation of a `StringLogSideChannel` that will listen
73
-
for messages from python and print them to the Unity debug log, as well as send
74
-
error messages from Unity to python.
52
+
The first step is to create the `StringLogSideChannel` class within the Unity project. Here is an implementation of a `StringLogSideChannel` that will listen for messages from python and print them to the Unity debug log, as well as send error messages from Unity to python.
75
53
76
54
```csharp
77
55
usingUnityEngine;
@@ -108,14 +86,7 @@ public class StringLogSideChannel : SideChannel
108
86
}
109
87
```
110
88
111
-
Once we have defined our custom side channel class, we need to ensure that it is
112
-
instantiated and registered. This can typically be done wherever the logic of
113
-
the side channel makes sense to be associated, for example on a MonoBehaviour
114
-
object that might need to access data from the side channel. Here we show a
115
-
simple MonoBehaviour object which instantiates and registers the new side
116
-
channel. If you have not done it already, make sure that the MonoBehaviour which
117
-
registers the side channel is attached to a GameObject which will be live in
118
-
your Unity scene.
89
+
Once we have defined our custom side channel class, we need to ensure that it is instantiated and registered. This can typically be done wherever the logic of the side channel makes sense to be associated, for example on a MonoBehaviour object that might need to access data from the side channel. Here we show a simple MonoBehaviour object which instantiates and registers the new side channel. If you have not done it already, make sure that the MonoBehaviour which registers the side channel is attached to a GameObject which will be live in your Unity scene.
119
90
120
91
```csharp
121
92
usingUnityEngine;
@@ -160,8 +131,7 @@ public class RegisterStringLogSideChannel : MonoBehaviour
160
131
161
132
### Example Python code
162
133
163
-
Now that we have created the necessary Unity C# classes, we can create their
164
-
Python counterparts.
134
+
Now that we have created the necessary Unity C# classes, we can create their Python counterparts.
165
135
166
136
```python
167
137
from mlagents_envs.environment import UnityEnvironment
@@ -196,9 +166,7 @@ class StringLogChannel(SideChannel):
196
166
super().queue_message_to_send(msg)
197
167
```
198
168
199
-
We can then instantiate the new side channel, launch a `UnityEnvironment` with
200
-
that side channel active, and send a series of messages to the Unity environment
201
-
from Python using it.
169
+
We can then instantiate the new side channel, launch a `UnityEnvironment` with that side channel active, and send a series of messages to the Unity environment from Python using it.
202
170
203
171
```python
204
172
# Create the channel
@@ -223,7 +191,4 @@ for i in range(1000):
223
191
env.close()
224
192
```
225
193
226
-
Now, if you run this script and press `Play` the Unity Editor when prompted, the
227
-
console in the Unity Editor will display a message at every Python step.
228
-
Additionally, if you press the Space Bar in the Unity Engine, a message will
229
-
appear in the terminal.
194
+
Now, if you run this script and press `Play` the Unity Editor when prompted, the console in the Unity Editor will display a message at every Python step. Additionally, if you press the Space Bar in the Unity Engine, a message will appear in the terminal.
0 commit comments