[RLlib] APPO accelerate vol 02: Bug fix for > 1 Learner actor.

[sven1977]

APPO accelerate vol 02: Bug fix for > 1 Learner actor.

The device detection on the Aggregation actors was faulty, leading to all aggregation actors to be placed on cuda:0.

Why are these changes needed?

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[simonsays1980]

LGTM. Working via the placement group is the way to go for letting Ray scheduling workers and learners as intended. There might be a problem when running on multi-node - specifically if nodes are (as usual when scaling out) too small for the complete scheduling group. Maybe we should test this.

[simonsays1980]

My guess is that this does not work as intended on a multi-node cluster b/c you want the learners/workers then be placed by Ray on the same nodes. As soon as the number of learner x number of GPUs per learner becomes greater than the number of GPUs per node the placement group might not be scheduled anywhere.

[sven1977]

Yeah, I'm still trying to figure this one out. This is still in a hacky state and needs more testing (especially with >1 Learners and >0 agg actors per Learner) before I'll merge this. Thanks for your help thus far.