RPC Communication in Distributed RL Training

[Sharad24]

There's three ways that I can think of having distributed training:

1. Use of Pytorch's Distributed Training infrastructure. Would require establishing communication protocols specific to the case of Deep RL. This would all be in Python (most likely) unless we find a way around.
2. Use of Reverb
   • Use TF based Datasets (@threewisemonkeys-as )
   • Pytorch wrapper for the conversion of NumPy arrays, etc (that are received) (Short-term, up for grabs)

[threewisemonkeys-as]

I agree that we should target 2 to begin with. We will still need python multiprocessing over here to run actors and learners seperately right?

As for the structure and fitting it into the rest of the library I was thinking of having DistributedOnPolicyTrainer and DistributedOffPolicyTrainer which will act as the main process and spawn the multile actors while maintaining and updating the central weights. In this case, the agent would only need to implement update_params (to be called in the main process) and select_action (to be called for each actor). The trajectories and weights would be transported through reverb.

I am holding off on #233 since a reverb buffer wrapper would heavily depend on the structure we go with. Plus it is not really useful in the non-distributed case.

[github-actions]

Stale issue message