We have added full parameter fine-tuning support in fft branch. The implementation is alpha quality and have undergone limited testing. We are very excited about out approach because it allows running simultaneous RL jobs on the same set of GPUs. It also leverages delta weight sync to ensure faster weight propagation between trainers and samplers over commodity shared filesystem.
Here is a list of items to be completed before it is ready to be merged in main.
P0:
- Verify
fft is opt-in.
- LoRA works as advertised
P1:
- A user guide that walks users how to deploy
fft enabled OpenRL on a GKE cluster. It should also configure the DRA configuration mapping.
- Update one of the existing recipe to demonstrate full fine-tuning in action
- Ensure default settings for
weight_sync_strategy, apply_method, delta_format is sensible for common models that we support with documentation on how to use advanced settings to optimize weight transfer etc.
- Admin Dashboard
The idea is to make MVP available to users without getting bogged down by implementing everything that is needed.
/cc @ShubyM
We have added full parameter fine-tuning support in
fftbranch. The implementation is alpha quality and have undergone limited testing. We are very excited about out approach because it allows running simultaneous RL jobs on the same set of GPUs. It also leverages delta weight sync to ensure faster weight propagation between trainers and samplers over commodity shared filesystem.Here is a list of items to be completed before it is ready to be merged in
main.P0:
fftis opt-in.P1:
fftenabled OpenRL on a GKE cluster. It should also configure the DRA configuration mapping.weight_sync_strategy,apply_method,delta_formatis sensible for common models that we support with documentation on how to use advanced settings to optimize weight transfer etc.The idea is to make MVP available to users without getting bogged down by implementing everything that is needed.
/cc @ShubyM