Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

#

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Jump to bottom

[RLlib] Fix action masking example. #47817

Merged
merged 4 commits into from
Sep 25, 2024
Merged

[RLlib] Fix action masking example. #47817

merged 4 commits into from
Sep 25, 2024
+6 −12

Conversation

simonsays1980
Copy link
Collaborator

@simonsays1980 simonsays1980 commented Sep 25, 2024 •
edited
Loading

Why are these changes needed?

Due to changes of the value computation (now in a ConnectorV2) manipulation of the batch as done by the ActionMaskingTorchRLModule the action mask was missing in the batch when _forward_train was entered. This PR fixes this error by adding the action_mask manually during value computation.

Related issue number

Closes #47361

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@simonsays1980
Removed parts of docstrings for LSTMEncoder(s) b/c no linear output l…
58cdc1b 
…ayer is used.

Signed-off-by: simonsays1980 <simon.zehnder@gmail.com>
@simonsays1980
Removed part of docstring from 'RecurrentEncoderConfig' b/c no linear…
eb7fc19 
… layer in recurrent encoders..

Signed-off-by: simonsays1980 <simon.zehnder@gmail.com>
@simonsays1980
Removed parts of 'GRUEncoder(s)' b/c linear output layer is not used.
1c7ede5 
Signed-off-by: simonsays1980 <simon.zehnder@gmail.com>
@simonsays1980
Fixed a bug in 'ActionMaskingTorchRLModule' due to changes in value c…
e6db184 
…omputation.

Signed-off-by: simonsays1980 <simon.zehnder@gmail.com>
@simonsays1980 simonsays1980 added rllib RLlib related issues rllib-models An issue related to RLlib (default or custom) Models. labels Sep 25, 2024
@sven1977 sven1977 changed the title [RLlib] - Fix action masking example [RLlib] Fix action masking example. Sep 25, 2024
@sven1977 sven1977 marked this pull request as ready for review September 25, 2024 17:29
@sven1977 sven1977 self-assigned this Sep 25, 2024
sven1977
sven1977 approved these changes Sep 25, 2024
View reviewed changes
Copy link
Contributor

@sven1977 sven1977 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome fix. Thanks @simonsays1980 !!

@sven1977 sven1977 enabled auto-merge (squash) September 25, 2024 17:30
@github-actions github-actions bot added the go add ONLY when ready to merge, run all tests label Sep 25, 2024
@sven1977 sven1977 merged commit 7966130 into ray-project:master Sep 25, 2024
5 of 6 checks passed
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024
@simonsays1980 @ujjawal-khare
[RLlib] Fix action masking example. (ray-project#47817)
0b3691a 
Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024
@simonsays1980 @ujjawal-khare
[RLlib] Fix action masking example. (ray-project#47817)
ec07568 
Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024
@simonsays1980 @ujjawal-khare
[RLlib] Fix action masking example. (ray-project#47817)
4a42773 
Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024
@simonsays1980 @ujjawal-khare
[RLlib] Fix action masking example. (ray-project#47817)
cff5b18 
Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024
@simonsays1980 @ujjawal-khare
[RLlib] Fix action masking example. (ray-project#47817)
eaa0426 
Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024
@simonsays1980 @ujjawal-khare
[RLlib] Fix action masking example. (ray-project#47817)
3e7cbd1 
Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024
@simonsays1980 @ujjawal-khare
[RLlib] Fix action masking example. (ray-project#47817)
0d3ad92 
Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024
@simonsays1980 @ujjawal-khare
[RLlib] Fix action masking example. (ray-project#47817)
ce75400 
Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Oct 15, 2024
@simonsays1980 @ujjawal-khare
[RLlib] Fix action masking example. (ray-project#47817)
e5c2bd4 
Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>
@simonsays1980 simonsays1980 deleted the fix-action-masking-example branch November 22, 2024 10:55
# for free to join this conversation on GitHub. Already have an account? # to comment
Reviewers

@sven1977 sven1977 sven1977 approved these changes

@ArturNiederfahrenhorst ArturNiederfahrenhorst Awaiting requested review from ArturNiederfahrenhorst

Assignees

@sven1977 sven1977

Labels
go add ONLY when ready to merge, run all tests rllib RLlib related issues rllib-models An issue related to RLlib (default or custom) Models. rllib-newstack
Projects
None yet
Milestone
No milestone
2 participants
@simonsays1980 @sven1977