Questions regarding SB3 implementation

[2BH]

Hi Onno,

I'm very interested in the idea that you proposed and recently started to work on the code.

I have a question regarding the implementation of your SB3 for SAC with colored noise.

Since in SB3, the update of policy is done every N steps. This means that most likely, your policy will be updated multiple times within one episode. Each update will call the sampling function from your colored gaussian distribution and thus, draw samples from your colored process generator's buffer. This would affect the correlation also during the interaction with the environments as some "samples" in between are taken for updating policy.

Would this be a problem then? Or you think it would not have huge impact anyway.

Best,

Baohe

[onnoeberhard]

Hi Baohe, thanks for your question. You are right that if the training procedure samples from the same process that is used for collecting rollouts, then the correlations can be impacted. However, I can't find any sampling in the training steps of the SB3 implementation of SAC (https://github.com/DLR-RM/stable-baselines3/blob/master/stable_baselines3/sac/sac.py). Could you please point out where the sampling during updates is happening? We tested the pink action noise with the SB3 SAC implementation and found a significant difference compared to the uncorrelated noise, so if sampling is indeed happening during updates, then it seems to not affect the correlation during rollouts too much.
-- Onno