New Algorithm: MAPO implementation #388
Conversation
ZiyiTsang
had a problem deploying
to
AReaL-unittests
GitHub Actions
Error
ZiyiTsang
had a problem deploying
to
AReaL-unittests
GitHub Actions
Error
ZiyiTsang
had a problem deploying
to
AReaL-unittests
GitHub Actions
Error
ZiyiTsang
had a problem deploying
to
AReaL-unittests
GitHub Actions
Error
ZiyiTsang
had a problem deploying
to
AReaL-unittests
GitHub Actions
Error
ZiyiTsang
had a problem deploying
to
AReaL-unittests
GitHub Actions
Error
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
Co-authored-by: Copilot <[email protected]>
ZiyiTsang
had a problem deploying
to
AReaL-unittests
GitHub Actions
Error
ZiyiTsang
had a problem deploying
to
AReaL-unittests
GitHub Actions
Error
ZiyiTsang
had a problem deploying
to
AReaL-unittests
GitHub Actions
Error
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
ZiyiTsang
had a problem deploying
to
AReaL-unittests
GitHub Actions
Error
ZiyiTsang
had a problem deploying
to
AReaL-unittests
GitHub Actions
Error
ZiyiTsang
had a problem deploying
to
AReaL-unittests
GitHub Actions
Error
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
This comment was marked as resolved.
This comment was marked as resolved.
There was a problem hiding this comment.
Pull Request Overview
Copilot reviewed 11 out of 12 changed files in this pull request and generated 8 comments.
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
|
Ready for review. |
| # since the advantages is same within same trajectory, we can get the trajectory_level advantage from first token | ||
| # base on assumption that the advantage on last dim are totally same | ||
|
|
||
| advantages_ = advantages[:, 0] # advantages shape [batch_size*group_size] |
There was a problem hiding this comment.
This line does not take any effect and should be removed.
There was a problem hiding this comment.
No, useful. Please see the code comment.
There was a problem hiding this comment.
The adv of first token is extract and use for below logic.
| return ( | ||
| 1 - trajectory_reweight | ||
| ) * deviation_base_norm + trajectory_reweight * mean_base_norm |
There was a problem hiding this comment.
double-check the formula. Since the trajectory_weight is computed as 4p(1-p) rather than 1-4p(1-p), should we reverse the weighting of these two norms?
There was a problem hiding this comment.
my mistake. thank you
ZiyiTsang
had a problem deploying
to
AReaL-unittests
GitHub Actions
Failure
ZiyiTsang
had a problem deploying
to
AReaL-unittests
GitHub Actions
Failure
ZiyiTsang
had a problem deploying
to
AReaL-unittests
GitHub Actions
Failure
ZiyiTsang
had a problem deploying
to
AReaL-unittests
GitHub Actions
Failure
ZiyiTsang
had a problem deploying
to
AReaL-unittests
GitHub Actions
Failure
ZiyiTsang
had a problem deploying
to
AReaL-unittests
GitHub Actions
Failure
ZiyiTsang
had a problem deploying
to
AReaL-unittests
GitHub Actions
Error
ZiyiTsang
had a problem deploying
to
AReaL-unittests
GitHub Actions
Error
ZiyiTsang
had a problem deploying
to
AReaL-unittests
GitHub Actions
Error
ZiyiTsang
had a problem deploying
to
AReaL-unittests
GitHub Actions
Failure
ZiyiTsang
had a problem deploying
to
AReaL-unittests
GitHub Actions
Failure
ZiyiTsang
had a problem deploying
to
AReaL-unittests
GitHub Actions
Failure
ZiyiTsang
had a problem deploying
to
AReaL-unittests
GitHub Actions
Failure
ZiyiTsang
had a problem deploying
to
AReaL-unittests
GitHub Actions
Failure
ZiyiTsang
had a problem deploying
to
AReaL-unittests
GitHub Actions
Failure
|
This pull request has been automatically marked as stale because it has not had recent activity within the last 14 days. Please add a comment or push new commits to keep it active. Thank you for your contribution! |
2 participants
This is the implementation to paper MAPO.
It doesn't require much code refactoring, still in the process.....
More "reay-to-go" algorithms and examples are the key to making a repo popular, especially RL-related.