Fix hooks for multiplayer case (#1071)

* Fix devcontainer

* Fix hooks for multiplayer case

* split off NEWS.md into per-package files

* Bump version, fix hooks

* Fix syntax error

* Fix syntax

* Fix syntax

* Fix linted issues

* Fix type signatures

* Fix test

* Fix hooks

* Add RL.jl to devcontainer

* Fix spellcheck error

* Fix syntax

* Add RLFarm news

Author: jeremiahpslewis

.devcontainer/devcontainer.json

@@ -2,13 +2,14 @@
   "customizations": {
       "vscode": {
           "extensions": [
-              "julialang.language-julia"
+            "julialang.language-julia",
+            "ms-azuretools.vscode-docker"
           ]
       }
   },
   "runArgs": [
       "--privileged"
   ],
   "dockerFile": "Dockerfile",
-  "updateContentCommand": "julia -e 'using Pkg; Pkg.develop(path=\"src/ReinforcementLearningBase\"); Pkg.develop(path=\"src/ReinforcementLearningEnvironments\"); Pkg.develop(path=\"src/ReinforcementLearningCore\"); Pkg.develop(path=\"src/ReinforcementLearningZoo\");'"
+  "updateContentCommand": "julia -e 'using Pkg; Pkg.develop(path=\"src/ReinforcementLearningBase\"); Pkg.develop(path=\"src/ReinforcementLearningEnvironments\"); Pkg.develop(path=\"src/ReinforcementLearningCore\"); Pkg.develop(path=\"src/ReinforcementLearningFarm\"); Pkg.develop(path=\"src/ReinforcementLearning\");'"
 }

NEWS.md

@@ -1,252 +1,13 @@
 # ReinforcementLearning.jl Release Notes
 
-## ReinforcementLearning.jl@v0.10.2
+#### v0.11.0
 
-- Pin sub-packages to pre-refactor versions
-- Agent calls now accept keyword arguments that will be passed to the policy. E.g. if the policy accepts a testmode.
-
-### ReinforcementLearningExperiments.jl
-
-#### v0.3
-
-- Transition to `RLCore.forward`, `RLBase.act!`, `RLBase.plan!` and `Base.push!` syntax instead of functional objects for hooks, policies and environments
-
-#### v0.2
-
-- Drop `ReinforcementLearning.jl` from dependencies, use `ReinforcementLearningCore.jl` instead
-
-#### v0.1.4
-
-- Support `device_rng` in SAC [#606](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/606)
-
-#### v0.1.3
-
-- Test experiments on GPU by default [#549](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/549)
-
-#### v0.1.2
-
-- Added an experiment for DQN training on discrete `PendulumEnv` (#537)
-
-### ReinforcementLearningEnvironments.jl
-
-#### v0.8
-
-- Transition to `RLCore.forward`, `RLBase.act!`, `RLBase.plan!` and `Base.push!` syntax instead of functional objects for hooks, policies and environments
-
-#### v0.7.2
-
-- Reduce allocations, improve performance of `RandomWalk1D`
-- Add tests to `RandomWalk1D`
-- Chase down JET.jl errors, fix
-- Update `TicTacToeEnv` and `RockPaperScissorsEnv` to support new `MultiAgentPolicy` setup
-
-#### v0.6.12
-
-- Bugfix bug with `is_discrete_space` [#566](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/issues/566)
-
-#### v0.6.11
-
-- Bugfix of CartPoleEnv with keyword arguments
-
-#### v0.6.10
-
-- Bugfix of CartPoleEnv with Float32
-
-#### v0.6.9
-
-- Added a continuous option for CartPoleEnv [#543](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/543).
-
-#### v0.6.8
-
-- Support `action_space(::TicTacToeEnv, player)`.
-
-#### v0.6.7
-
-- Fixed bugs in plotting `MountainCarEnv` (#537)
-- Implemented plotting for `PendulumEnv` (#537)
-
-#### v0.6.6
-
-- Bugfix with `ZeroTo` [#534](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/534)
-
-#### v0.6.4
-
-- Add `GraphShortestPathEnv`. [#445](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/445)
-
-#### v0.6.3
-
-- Add `StockTradingEnv` from the paper [Deep Reinforcement Learning for
-  Automated Stock Trading: An Ensemble
-  Strategy](https://github.com/AI4Finance-Foundation/FinRL-Trading).
-  This environment is a good testbed for multi-continuous action space
-  algorithms. [#428](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/428)
-
-#### v0.6.2
-
-- Add `SequentialEnv` environment wrapper to turn a simultaneous environment
-  into a sequential one.
-
-#### v0.6.1
-
-- Drop GR in RLEnvs and lazily load ploting functions.[#309](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/309), [#310](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/310)
-
-#### v0.6.0
-
-- Set `AcrobotEnv` into lazy loading to reduce the dependency of `OrdinaryDiffEq`.
-
-### ReinforcementLearningZoo.jl
-
-#### v0.7.0
-
-- Transition to `RLCore.forward`, `RLBase.act!`, `RLBase.plan!` and `Base.push!` syntax instead of functional objects for hooks, policies and environments
-- Reduce excess `TDLearner` allocations by using Tuple instead of Array
-
-#### v0.4.1
-
-- Make keyword argument `n_actions` in `TabularPolicy` optional. [#300](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/300)
-
-#### v0.6.0
-
-- Extensive refactor based on RLBase.jl `v0.11`, most components not **yet** ported
-
-#### v0.5.11
-
-- Fix multi-dimension action space in TD3. [#624](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/issues/624)
-
-#### v0.5.10
-
-- Support `device_rng` in SAC [#606](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/606)
-
-#### v0.5.7
+- Complete major refactor, API consistency improvements and incorporate ReinforcementLearningTrajectories.jl
 
-- Fix warning about `vararg.data` in [email protected] [#560](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/560)
+#### v0.10.2
 
-#### v0.5.6
-
-- Make BC GPU compatible [#553](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/553)
-
-#### v0.5.5
-
-- Make most algorithms GPU compatible [#549](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/549)
-
-#### v0.5.4
-
-- Support `length` method for `VectorWSARTTrajectory`.
-
-#### v0.5.3
-
-- Revert part of the unexpected change of PPO in the last PR.
-
-#### v0.5.2
-
-- Fixed the bug with MaskedPPOTrajectory reported [here](https://discourse.julialang.org/t/using-ppopolicy-with-custom-environment-with-action-masking-in-reinforcementlearning-jl/69625/6)
-
-#### v0.5.0
-
-- Update the complete SAC implementation and modify some details based on the
-  original paper. [#365](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/365)
-- Add some extra keyword parameters for `BehaviorCloningPolicy` to use it
-  online. [#390](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/390)
-
-#### v0.4.0
-
-- Moved all the experiments into a new package `ReinforcementLearningExperiments.jl`. The related dependencies are also removed (`BSON.jl`, `StableRNGs.jl`, `TensorBoardLogger.jl`).
-
-### ReinforcementLearningDatasets.jl
-
-#### v0.1.0
-
-- Add functionality for fetching d4rl datasets as an iterable DataSet. Credits: https://arxiv.org/abs/2004.07219
-- This supports d4rl and d4rl-pybullet and Google Research DQN atari datasets.
-- Uses DataDeps for data dependency management.
-- This package also supports RL Unplugged Datasets.
-- Support for [google-research/deep_ope](https://github.com/google-research/deep_ope) added.
-
-## [email protected]
-
-### ReinforcementLearningBase.jl
-
-#### v0.12.0
-
-- Transition to `RLCore.forward`, `RLBase.act!`, `RLBase.plan!` and `Base.push!` syntax instead of functional objects for hooks, policies and environments
-
-#### v0.9.7
-
-- Update POMDPModelTools -> POMDPTools
-- Add `next_player!` method to support `Sequential` `MultiAgent` environments
-
-#### v0.9.6
-
-- Implement `Base.:(==)` for `Space`. [#428](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/428)
-
-#### v0.9.5
-
-- Add default `Base.:(==)` and `Base.hash` method for `AbstractEnv`. [#348](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/348)
-
-### ReinforcementLearningCore.jl
-
-#### v0.10.1
-
-- Fix hook issue with 'extra' call; always run `push!` at end of episode, regardless of whether stopped or terminated
-
-#### v0.10.0
-
-- Transition to `RLCore.forward`, `RLBase.act!`, `RLBase.plan!` and `Base.push!` syntax instead of functional objects for hooks, policies and environments
-
-#### v0.9.3
-
-- Add back multi-agent support with `MultiAgentPolicy` and `MultiAgentHook`
-
-#### v0.9.2
-
-- Use correct Flux.stack function signature
-- Reduce allocations, improve performance of `RandomPolicy`
-- Chase down JET.jl errors, fix
-- Add tests for `StopAfterStep`, `StopAfterEpisode`
-- Add tests, improve performance of `RewardsPerEpisode`
-- Refactor `Agent` for speedup
-
-#### v0.8.11
-
-- When sending a `CircularArrayBuffer` to GPU devices, convert `CircularArrayBuffer` into `CuArray` instead of the adapted `CircularArrayBuffer` of `CuArray`. [#606](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/606)
-
-#### v0.8.10
-
-- Update dependency of `CircularArrayBuffers` to `v0.1.9`. [#602](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/602)
-- Add `CovGaussianNetwork`. [#597](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/597)
-#### v0.8.8
-
-- Fix warning about `vararg.data` in [email protected] [#560](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/560)
-
-#### v0.8.7
-
-- Make `GaussianNetwork` differentiable. [#549](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/549)
-
-#### v0.8.6
-
-- Fixed a bug [1] with the `DoOnExit` hook (#537)
-- Added some convenience hooks for rendering rollout episodes (#537)
-
-#### v0.8.5
-
-- Fixed the method overwritten warning of `device` from `CUDA.jl`.
-
-#### v0.8.3
-
-- Add extra two optional keyword arguments (`min_σ` and `max_σ`) in
-  `GaussianNetwork` to clip the output of `logσ`. [#428](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/428)
-
-#### v0.8.2
-
-- Add GaussianNetwork and DuelingNetwork into ReinforcementLearningCore.jl as general components. [#370](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/370)
-- Export `WeightedSoftmaxExplorer`.
-  [#382](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/382)
-
-#### v0.8.1
-
-- Minor bug & typo fixes
+- Pin sub-packages to pre-refactor versions
+- Agent calls now accept keyword arguments that will be passed to the policy. E.g. if the policy accepts a testmode.
 
-#### v0.8.0
 
-- Removed `ResizeImage` preprocessor to reduce the dependency of `ImageTransformations`. 
-- Show unicode plot at the end of an experiment in the `TotalRewardPerEpisode` hook.
+#### v0.9.0

src/ReinforcementLearningBase/NEWS.md

@@ -0,0 +1,18 @@
+### ReinforcementLearningBase.jl Release Notes
+
+#### v0.12.0
+
+- Transition to `RLCore.forward`, `RLBase.act!`, `RLBase.plan!` and `Base.push!` syntax instead of functional objects for hooks, policies and environments
+
+#### v0.9.7
+
+- Update POMDPModelTools -> POMDPTools
+- Add `next_player!` method to support `Sequential` `MultiAgent` environments
+
+#### v0.9.6
+
+- Implement `Base.:(==)` for `Space`. [#428](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/428)
+
+#### v0.9.5
+
+- Add default `Base.:(==)` and `Base.hash` method for `AbstractEnv`. [#348](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/348)

src/ReinforcementLearningCore/NEWS.md

@@ -0,0 +1,74 @@
+#### v0.15.1
+
+- Fix MultiPlayer hook bugs
+- Clarify that the correct `push!` syntax is `push!(hook, stage, policy, env)` or `push!(hook, stage, policy, env, player)`; `push!(hook)` or other permutations now error as not implemented.
+
+#### v0.15.0
+
+- First version released with ReinforcementLearning v0.11
+
+#### v0.10.1
+
+- Fix hook issue with 'extra' call; always run `push!` at end of episode, regardless of whether stopped or terminated
+
+#### v0.10.0
+
+- Transition to `RLCore.forward`, `RLBase.act!`, `RLBase.plan!` and `Base.push!` syntax instead of functional objects for hooks, policies and environments
+
+#### v0.9.3
+
+- Add back multi-agent support with `MultiAgentPolicy` and `MultiAgentHook`
+
+#### v0.9.2
+
+- Use correct Flux.stack function signature
+- Reduce allocations, improve performance of `RandomPolicy`
+- Chase down JET.jl errors, fix
+- Add tests for `StopAfterStep`, `StopAfterEpisode`
+- Add tests, improve performance of `RewardsPerEpisode`
+- Refactor `Agent` for speedup
+
+#### v0.8.11
+
+- When sending a `CircularArrayBuffer` to GPU devices, convert `CircularArrayBuffer` into `CuArray` instead of the adapted `CircularArrayBuffer` of `CuArray`. [#606](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/606)
+
+#### v0.8.10
+
+- Update dependency of `CircularArrayBuffers` to `v0.1.9`. [#602](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/602)
+- Add `CovGaussianNetwork`. [#597](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/597)
+#### v0.8.8
+
+- Fix warning about `vararg.data` in [email protected] [#560](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/560)
+
+#### v0.8.7
+
+- Make `GaussianNetwork` differentiable. [#549](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/549)
+
+#### v0.8.6
+
+- Fixed a bug [1] with the `DoOnExit` hook (#537)
+- Added some convenience hooks for rendering rollout episodes (#537)
+
+#### v0.8.5
+
+- Fixed the method overwritten warning of `device` from `CUDA.jl`.
+
+#### v0.8.3
+
+- Add extra two optional keyword arguments (`min_σ` and `max_σ`) in
+  `GaussianNetwork` to clip the output of `logσ`. [#428](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/428)
+
+#### v0.8.2
+
+- Add GaussianNetwork and DuelingNetwork into ReinforcementLearningCore.jl as general components. [#370](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/370)
+- Export `WeightedSoftmaxExplorer`.
+  [#382](https://github.com/JuliaReinforcementLearning/ReinforcementLearning.jl/pull/382)
+
+#### v0.8.1
+
+- Minor bug & typo fixes
+
+#### v0.8.0
+
+- Removed `ResizeImage` preprocessor to reduce the dependency of `ImageTransformations`. 
+- Show unicode plot at the end of an experiment in the `TotalRewardPerEpisode` hook.

src/ReinforcementLearningCore/Project.toml

@@ -1,6 +1,6 @@
 name = "ReinforcementLearningCore"
 uuid = "de1b191a-4ae0-4afa-a27b-92d07f46b2d6"
-version = "0.15.0"
+version = "0.15.1"
 
 [deps]
 AbstractTrees = "1520ce14-60c1-5f80-bbc7-55ef81b5835c"