question about loss calculation 。

[sarsigmadelta]

thanks for sharing 。 confusion about update_weights function in the agents/base.py ，about line 480 ， value_prefix_loss += kl_loss(value_prefixes, target_value_prefixes_support[:, step_i]) * mask policy_loss_i = kl_loss(policies, target_policies[:, step_i + 1]) * mask the target data indexed by step_i and step_i + 1 ，why is different index ， thanks very much

[Shengjiewang-Jason]

This is because when we calculate the reward loss (value prefix loss in the code) within the unroll steps, the reward is conditioned on the state and action. However, the action is only conditioned on the current state. The number of actions is over the number of rewards within the unroll steps.