ML-Agents Release 13

ML-Agents Release 13

Package Versions

NOTE: It is strongly recommended that you use packages from the same release together for the best experience.

Package	Version
com.unity.ml-agents (C#)	v1.8.0
com.unity.ml-agents.extensions (C#)	v0.1.0-preview
ml-agents (Python)	v0.24.0
ml-agents-envs (Python)	v0.24.0
gym-unity (Python)	v0.24.0
Communicator (C#/Python)	v1.4.0

Major Features and Improvements

com.unity.ml-agents / com.unity.ml-agents.extensions

• Add an InputActuatorComponent to allow the generation of Agent action spaces from an InputActionAsset.
  Projects wanting to use this feature will need to add the
  Input System Package at version 1.1.0-preview.3 or later. (#4881)

ml-agents / ml-agents-envs / gym-unity (Python)

• TensorFlow trainers have been removed, please use the Torch trainers instead. (#4707)
• A plugin system for mlagents-learn has been added. You can now define custom
  StatsWriter implementations and register them to be called during training.
  More types of plugins will be added in the future. (#4788)

Minor Changes

com.unity.ml-agents / com.unity.ml-agents.extensions (C#)

• The ActionSpec constructor is now public. Previously, it was not possible to create an
  ActionSpec with both continuous and discrete actions from code. (#4896)
• StatAggregationMethod.Sum can now be passed to StatsRecorder.Add(). This
  will result in the values being summed (instead of averaged) when written to
  TensorBoard. Thanks to @brccabral for the contribution! (#4816)
• The upper limit for the time scale (by setting the --time-scale parameter in mlagents-learn) was
  removed when training with a player. The Editor still requires it to be clamped to 100. (#4867)
• Added the IHeuristicProvider interface to allow IActuators as well as Agent implement the Heuristic function to generate actions.
  Updated the Basic example and the Match3 Example to use Actuators.
  Changed the namespace and file names of classes in com.unity.ml-agents.extensions. (#4849)
• Added VectorSensor.AddObservation(IList<float>). VectorSensor.AddObservation(IEnumerable<float>)
  is deprecated. The IList version is recommended, as it does not generate any
  additional memory allocations. (#4887)
• Added ObservationWriter.AddList() and deprecated ObservationWriter.AddRange().
  AddList() is recommended, as it does not generate any additional memory allocations. (#4887)
• The Barracuda dependency was upgraded to 1.3.0. (#4898)
• Added ActuatorComponent.CreateActuators, and deprecated ActuatorComponent.CreateActuator. The
  default implementation will wrap ActuatorComponent.CreateActuator in an array and return that. (#4899)
• InferenceDevice.Burst was added, indicating that Agent's model will be run using Barracuda's Burst backend.
  This is the default for new Agents, but existing ones that use InferenceDevice.CPU should update to
  InferenceDevice.Burst. (#4925)

ml-agents / ml-agents-envs / gym-unity (Python)

• Tensorboard now logs the Environment Reward as both a scalar and a histogram. (#4878)
• Added a --torch-device commandline option to mlagents-learn, which sets the default
  torch.device used for training. (#4888)
• The --cpu commandline option had no effect and was removed. Use --torch-device=cpu to force CPU training. (#4888)
• The mlagents_env API has changed, BehaviorSpec now has a observation_specs property containing a list of ObservationSpec. For more information on ObservationSpec see here. (#4763, #4825)

Bug Fixes

com.unity.ml-agents (C#)

• Fix a compile warning about using an obsolete enum in GrpcExtensions.cs. (#4812)
• CameraSensor now logs an error if the GraphicsDevice is null. (#4880)
• Removed unnecessary memory allocations in ActuatorManager.UpdateActionArray() (#4877)
• Removed unnecessary memory allocations in SensorShapeValidator.ValidateSensors() (#4879)
• Removed unnecessary memory allocations in SideChannelManager.GetSideChannelMessage() (#4886)
• Removed several memory allocations that happened during inference. On a test scene, this
  reduced the amount of memory allocated by approximately 25%. (#4887)
• Removed several memory allocations that happened during inference with discrete actions. (#4922)
• Properly catch permission errors when writing timer files. (#4921)
• Unexpected exceptions during training initialization and shutdown are now logged. If you see
  "noisy" logs, please let us know! (#4930, #4935)

ml-agents / ml-agents-envs / gym-unity (Python)

• Fixed a bug that would cause an exception when RunOptions was deserialized via pickle. (#4842)
• Fixed a bug that can cause a crash if a behavior can appear during training in multi-environment training. (#4872)
• Fixed the computation of entropy for continuous actions. (#4869)
• Fixed a bug that would cause UnityEnvironment to wait the full timeout
  period and report a misleading error message if the executable crashed
  without closing the connection. It now periodically checks the process status
  while waiting for a connection, and raises a better error message if it crashes. (#4880)
• Passing a -logfile option in the --env-args option to mlagents-learn is
  no longer overwritten. (#4880)
• The load_weights function was being called unnecessarily often in the Ghost Trainer leading to training slowdowns. (#4934)

ML-Agents Release 12

ML-Agents Release 12

Package Versions

NOTE: It is strongly recommended that you use packages from the same release together for the best experience.

Package	Version
com.unity.ml-agents (C#)	v1.7.2
ml-agents (Python)	v0.23.0
ml-agents-envs (Python)	v0.23.0
gym-unity (Python)	v0.23.0
Communicator (C#/Python)	v1.3.0

Bug Fixes

com.unity.ml-agents (C#)

• Add analytics package dependency to the package manifest. (#4794)

ml-agents / ml-agents-envs / gym-unity (Python)

• Fixed the docker build process. (#4791)

ML-Agents Release 11

ML-Agents Release 11

Package Versions

NOTE: It is strongly recommended that you use packages from the same release together for the best experience.

Package	Version
com.unity.ml-agents (C#)	v1.7.0
ml-agents (Python)	v0.23.0
ml-agents-envs (Python)	v0.23.0
gym-unity (Python)	v0.23.0
Communicator (C#/Python)	v1.3.0

Major Features and Improvements

com.unity.ml-agents / com.unity.ml-agents.extensions (C#)

• An individual agent can now take both continuous and discrete actions. You can specify both continuous and discrete action sizes in Behavior Parameters. (#4702, #4718)

ml-agents / ml-agents-envs / gym-unity (Python)

• PyTorch trainers now support training agents with both continuous and discrete action spaces. (#4702)

Bug Fixes and Minor Changes

com.unity.ml-agents / com.unity.ml-agents.extensions (C#)

• In order to improve the developer experience for Unity ML-Agents Toolkit, we have added in-editor analytics. Please refer to "Information that is passively collected by Unity" in the Unity Privacy Policy. (#4677)
• The FoodCollector example environment now uses continuous actions for moving and discrete actions for shooting. (#4746)
• Removed noisy warnings about API minor version mismatches in both the C# and python code. (#4688)

ml-agents / ml-agents-envs / gym-unity (Python)

• ActionSpec._validate_action() now enforces that UnityEnvironment.set_action_for_agent() receives a 1D np.array.

ML-Agents Release 10

ML-Agents Release 10

Package Versions

NOTE: It is strongly recommended that you use packages from the same release together for the best experience.

Package	Version
com.unity.ml-agents (C#)	v1.6.0
ml-agents (Python)	v0.22.0
ml-agents-envs (Python)	v0.22.0
gym-unity (Python)	v0.22.0
Communicator (C#/Python)	v1.2.0

Major Features and Improvements

New Demo Environment

• The Match3 environment was added to the Project, it uses the new utilities added in com.unity.ml-agents.extensions.

ml-agents / ml-agents-envs / gym-unity (Python)

• PyTorch trainers are now the default. See the installation docs for more information on installing PyTorch. For the time being, TensorFlow is still available; you can use the TensorFlow backend by adding --tensorflow to the CLI, or adding framework: tensorflow in the configuration YAML. (#4517)

Bug Fixes and Minor Changes

com.unity.ml-agents / com.unity.ml-agents.extensions (C#)

• The Barracuda dependency was upgraded to 1.1.2 (#4571)
• Utilities were added to com.unity.ml-agents.extensions to make it easier to integrate with match-3 games. See the readme for more details. (#4515)
• Agent.CollectObservations() and Agent.EndEpisode() will now throw an exception if they are called recursively (for example, if they call Agent.EndEpisode()). Previously, this would result in an infinite loop and cause the editor to hang. (#4573)

ml-agents / ml-agents-envs / gym-unity (Python)

• The action_probs node is no longer listed as an output in TensorFlow models (#4613).

ML-Agents Release 9

ML-Agents Release 9

Package Versions

NOTE: It is strongly recommended that you use packages from the same release together for the best experience.

Package	Version
com.unity.ml-agents (C#)	v1.5.0
ml-agents (Python)	v0.21.1
ml-agents-envs (Python)	v0.21.1
gym-unity (Python)	v0.21.1
Communicator (C#/Python)	v1.2.0

Bug Fixes and Minor Changes

ml-agents (Python)

• Fixed an issue where runs could not be resumed when using TensorFlow and Ghost Training. (#4593)
• Capped cattrs version to 1.0.x. (#4613)
• Capped PyTorch version to 1.6.x. (#4617)

ML-Agents Release 8

ML-Agents Release 8

Package Versions

NOTE: It is strongly recommended that you use packages from the same release together for the best experience.

Package	Version
com.unity.ml-agents (C#)	v1.5.0
ml-agents (Python)	v0.21.0
ml-agents-envs (Python)	v0.21.0
gym-unity (Python)	v0.21.0
Communicator (C#/Python)	v1.2.0

Major Features and Improvements

com.unity.ml-agents (C#)

• Stacking for compressed observations is now supported. An additional setting option Observation Stacks is added to the sensor components that support compressed observations. A new class ISparseChannelSensor with an additional method GetCompressedChannelMapping()is added to generate a mapping of the channels in compressed data to the actual channel after decompression for the python side to decompress correctly. (#4476)

ml-agents (Python)

• Added the Random Network Distillation (RND) intrinsic reward signal to the Pytorch trainers. To use RND, add a rnd section to the reward_signals section of your yaml configuration file. More information here. (#4473)
• The Communication API was changed to 1.2.0 to indicate support for stacked compressed observation. A new entry compressed_channel_mapping is added to the proto to handle decompression correctly. Newer versions of the package that wish to make use of this will also need a compatible version of the Python trainers. (#4476)

Bug Fixes and Minor Changes

com.unity.ml-agents (C#)

• Fixed a bug where accessing the Academy outside of play mode would cause the Academy to get stepped multiple times when in play mode. (#4532)

ml-agents (Python)

• In the VisualFoodCollector scene, a vector flag representing the frozen state of the agent is added to the input observations in addition to the original first-person camera frame. The scene is able to train with the provided default config file. (#4511)
• Added a new visual scene Visual3DBall in the 3DBall example. (#4513)
• Added conversion to string for sampler classes to increase the verbosity of the curriculum lesson changes. The lesson updates would now output the sampler stats in addition to the lesson and parameter name to the console. (#4484)
• Localized documentation in Russian is added. (#4529)

Acknowledgements

Thank you @SergeyMatrosov for your contributions to this release.

ML-Agents Release 7

ML-Agents Release 7

Package Versions

NOTE: It is strongly recommended that you use packages from the same release together for the best experience.

Package	Version
com.unity.ml-agents (C#)	v1.4.0
ml-agents (Python)	v0.20.0
ml-agents-envs (Python)	v0.20.0
gym-unity (Python)	v0.20.0
Communicator (C#/Python)	v1.1.0

Major Features and Improvements

com.unity.ml-agents (C#)

• The IActuator interface and ActuatorComponent abstract class were added. These are analogous to ISensor and SensorComponent, but for applying actions for an Agent. They allow you to control the action space more programmatically than defining the actions in the Agent's Behavior Parameters. See BasicActuatorComponent.cs for an example of how to use them. (#4297, #4315)

ml-agents (Python)

• Experimental PyTorch support has been added. Use --torch when running mlagents-learn, or add framework: pytorch to your trainer configuration (under the behavior name) to enable it. Note that PyTorch 1.6.0 or greater should be installed to use this feature; see see the PyTorch website for installation instructions and the relevant ML-Agents docs for usage. (#4335)

Breaking Changes

ml-agents (Python)

The minimum supported version of TensorFlow was increased to 1.14.0. (#4411)

Known Issues

ml-agents (Python)

• Soft-Actor Critic (SAC) runs considerably slower when using the PyTorch backend than when using TensorFlow.

Bug Fixes and Minor Changes

com.unity.ml-agents (C#)

• Updated Barracuda to 1.1.1-preview (#4482)
• Enabled C# formatting using dotnet-format. (#4362)
• GridSensor was added to the com.unity.ml-agents.extensions package. Thank you to Jaden Travnik from Eidos Montreal for the contribution! (#4399)
• Added Agent.EpisodeInterrupted(), which can be used to reset the agent when it has reached a user-determined maximum number of steps. This behaves similarly to Agent.EndEpsiode() but has a slightly different effect on training (#4453).
• Previously, com.unity.ml-agents was not declaring built-in packages as dependencies in its package.json. The relevant dependencies are now listed. (#4384)
• Fixed the sample code in the custom SideChannel example. (#4466)

ml-agents (Python)

• Compressed visual observations with >3 channels are now supported. In ISensor.GetCompressedObservation(), this can be done by writing 3 channels at a time to a PNG and concatenating the resulting bytes. (#4399)
• The Communication API was changed to 1.1.0 to indicate support for concatenated PNGs (see above). Newer versions of the package that wish to make use of this will also need a compatible version of the trainer. (#4462)
• A CNN (vis_encode_type: match3) for smaller grids, e.g. board games, has been added. (#4434)
• You can now again specify a default configuration for your behaviors. Specify default_settings in your trainer configuration to do so. (#4448)
• A bug in the observation normalizer that would cause rewards to decrease when using --resume was fixed. (#4463)

Acknowledgements

Thank you @NeonMika , @armando-fandango, @Sebastian-Schuchmann , and everyone at Unity for their contributions to this release.

ML-Agents Release 6

ML-Agents Release 6

Package Versions

NOTE: It is strongly recommended that you use packages from the same release together for the best experience.

Package	Version
com.unity.ml-agents (C#)	v1.3.0
ml-agents (Python)	v0.19.0
ml-agents-envs (Python)	v0.19.0
gym-unity (Python)	v0.19.0
Communicator (C#/Python)	v1.0.0

Breaking Changes

ml-agents (Python)

• The minimum supported Python version for ml-agents-envs was changed to 3.6.1. (#4244)

Bug Fixes and Minor Changes

com.unity.ml-agents (C#)

• Academy.EnvironmentStep() will now throw an exception if it is called recursively (for example, by an Agent's CollectObservations method). Previously, this would result in an infinite loop and cause the editor to hang. (#1551)

ml-agents (Python)

• The supported versions of numpy ml-agents-envs was changed to not allow 1.19.0 or later. This was done to reflect a similar change in TensorFlow's requirements. (#4274)
• Model checkpoints are now also saved as .nn files during training. (#4127)
• Model checkpoint info is saved in TrainingStatus.json after training is concluded (#4127)
• The CSV statistics writer was removed (#4300).
• StatsSideChannel now stores multiple values per key. This means that multiple calls to StatsRecorder.Add() with the same key in the same step will no longer overwrite each other. (#4236)

ml-agents-envs (Python)

• The interaction between EnvManager and TrainerController was changed; EnvManager.advance() was split into two stages, and TrainerController now uses the results from the first stage to handle new behavior names. This change speeds up Python training by approximately 5-10%. (#4259)

Known Issues

com.unity.ml-agents (C#)

• On macOS 10.15, if the ML-Agents package is installed from a website download, you may receive a “file cannot be opened issue” when running scenes that use ML-Agents. Workarounds include installing the package using the Unity Package Manager, or following the instructions here. (specifically, the section titled “How to open an app that hasn’t been notarized or is from an unidentified developer”).

Acknowledgements

Thank you ChristianCoenen, niskander, and everyone at Unity for their contributions to this release.

ML-Agents Release 5

Package Versions

As part of ML-Agents Release 5, we will be versioning the different packages that make up the release.

NOTE: It is strongly recommended that you use packages from the same release together for the best experience.

Package	Version
com.unity.ml-agents (C#)	v1.2.0
ml-agents (Python)	v0.18.1
ml-agents-envs (Python)	v0.18.1
gym-unity (Python)	v0.18.1
Communicator (C#/Python)	v1.0.0

Bug Fixes

ml-agents / ml-agents-envs / gym-unity (Python)

• Summary writer no longer crashes if Hyperparameters could not be written (#4265)
• Reduce numpy version in ml-agents-envs setup (#4274)
• Fix curriculum lesson incrementing past final lesson (#4279)

ML-Agents Release 4

Package Versions

NOTE: It is strongly recommended that you use packages from the same release together for the best experience.

Package	Version
com.unity.ml-agents (C#)	v1.2.0
ml-agents (Python)	v0.18.0
ml-agents-envs (Python)	v0.18.0
gym-unity (Python)	v0.18.0
Communicator (C#/Python)	v1.0.0

Major Features and Improvements

ml-agents (Python)

• The Parameter Randomization feature has been refactored and merged with the Curriculum. This enables sampling of new parameters per episode to improve robustness. The resampling-interval parameter has been removed and the config structure updated. More information here. (#4065)
• Curriculum has been refactored and is now specified at the level of the parameter, not the behavior. It is also now possible to specify a sampler in the lesson of a Curriculum. More information here.(#4160)

Breaking Changes

ml-agents (Python)

• The training configuration yaml file format has changed. In order to use curriculum or environment randomization, you must update your configuration file with the new section environment_parameters. We provide an update script to help this migration. Run python -m mlagents.trainers.upgrade_config -h with the latest version to see the script usage.

Bug Fixes and Minor Changes

com.unity.ml-agents (C#)

• SideChannelsManager was renamed to SideChannelManager. The old name is still supported, but deprecated. (#4137)
• RayPerceptionSensor.Perceive() now additionally stores the GameObject that was hit by the ray. (#4111)
• Fixed an issue where RayPerceptionSensor would raise an exception when the list of tags was empty, or a tag in the list was invalid (unknown, null, or empty string). (#4155)
• The Barracuda dependency was upgraded to 1.0.1 (#4188)

ml-agents-envs (Python)

• Added new Google Colab notebooks to show how to use UnityEnvironment. (#4117)

ml-agents (Python)

• Fixed an error when setting initialize_from in the trainer configuration YAML to null. (#4175)
• Fixed a rare crash in StatsReporter when using threaded trainers (#4201)

Example Environments (C#)

• Fixed issue with FoodCollector, Soccer, and WallJump when playing with keyboard. (#4147, #4174)

Known Issues

com.unity.ml-agents (C#)

• On macOS 10.15, if the ML-Agents package is installed from a website download, you may receive a “file cannot be opened issue” when running scenes that use ML-Agents. Workarounds include installing the package using the Unity Package Manager, or following the instructions here.

Acknowledgements

Thank you @StefanoCecere, @yongjun823, @poehlerflorian, @ChristianCoenen, @textarcana, @furkan-celik and everyone at Unity for their contributions to this release.