Skip to content

Changes to make DPO run faster #1175

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 56 commits into
base: main
Choose a base branch
from
Draft

Changes to make DPO run faster #1175

wants to merge 56 commits into from

Conversation

@finbarrtimbers
Copy link
Collaborator

@finbarrtimbers finbarrtimbers commented Nov 11, 2025

Here, we make the following changes:

  1. We support setting the ZeRO stage for DPO.
  2. We add support for caching the reference logprobs to disk to avoid recomputing them
  3. Changes DPO to log the grad norms
  4. Changes DPO so that we don't crash when wandb tags aren't available

@finbarrtimbers
Set nccl flags
606f7db
@finbarrtimbers
Changed cluster
7319bdc
@finbarrtimbers
testing with a stego HF checkpoint
96bc3ed
@finbarrtimbers
uses olmo3 model
130d591
@finbarrtimbers
uses hf checkpoint
6145d82
@finbarrtimbers
now uses olmo-instruct
cdac2de
@finbarrtimbers
Updated script
11401a8
@finbarrtimbers
now uses a new image
60ed4b2
@finbarrtimbers
Removed unnused flags
906369e
@finbarrtimbers
Added scott's cript
f7c2736
@finbarrtimbers
changes to smoke test
6ad7889
@finbarrtimbers
removed nccl flags
f37a6fb
@finbarrtimbers
Added back nccl flags
d09ce77
@finbarrtimbers
Set LR to the one we're about to preempt
7f259d2
@finbarrtimbers
Now uses 16
e868a51
@finbarrtimbers
matched scott's config
1cd94c1
@finbarrtimbers
Added a cache and plumbed the deepspeed config through
744c48d
@finbarrtimbers
set zero_stage
a6e7fad
@finbarrtimbers
Added flags to both
1fa8827
@finbarrtimbers
updated scripts to use new image
31d85b7
@finbarrtimbers
removed problematic logging statements
d87f7c1
@finbarrtimbers
Set concatenated forward False
ad47c7b
@finbarrtimbers
updated script
a490a51
@finbarrtimbers
fixed script with newline
5faba63
@finbarrtimbers
Added zero hpz partition size
ae90ada
@finbarrtimbers
use 8 nodes
5cfb972
@finbarrtimbers
update code
cf17aff
@finbarrtimbers
now uses train numbers in the filename
e3a6f7c
@finbarrtimbers
set concat forward false on scottg script
850b9f3
@finbarrtimbers
cleaned up caching logic
7788513
finbarrtimbers and others added 26 commits November 10, 2025 20:51
@finbarrtimbers
now get_wandb_tags won't fail
ad239ed
@finbarrtimbers
trying with stage 3
21042a7
@finbarrtimbers
now we save per rank caches
1f08436
@finbarrtimbers
set preemptible flag
f166603
@finbarrtimbers
now 16 nodes
eede995
@scottgeng00
add seed to ref logprobs cache file, and fail if using more than one …
ef419d6 
…epoch
@scottgeng00
launch script on unmerged sft
acd3dcd
@scottgeng00
vibe coded upload ref cache to gs bucket so that other jobs can pull
c819dea
@scottgeng00
Revert "vibe coded upload ref cache to gs bucket so that other jobs c…
3b1f449 
…an pull"

This reverts commit c819dea.
@saurabh111233212
add grad norm logging
a4536d8
@scottgeng00
scripts
415de99
@saurabh111233212
fix grad norm
f437ba0
@scottgeng00
Merge branch 'finbarr/dpo-faster' of github.com:allenai/open-instruct…
b2b075f 
… into finbarr/dpo-faster
@scottgeng00
add tyler implementation of grad norm
705dd2b
@saurabh111233212
just one more grad norm fix bro....
cb59e86
@scottgeng00
more debug scripts
f780eae
@scottgeng00
hardcoded eval prio, update later
c44dcf3
@scottgeng00
eval workspace fix
3556184
@finbarrtimbers
updated code
368ca70
@scottgeng00
fix eval launch
8ceaf5f
@finbarrtimbers
Merge branch 'main' into finbarr/dpo-faster
2834f85
@finbarrtimbers
updated code to run with zro_hpz_partition size
d5adbba
@finbarrtimbers
set offload optimizer
092ae95
@finbarrtimbers
set partition size 1
ccb9aaa
@scottgeng00
dpo eval workspace, gpu, prio flags, and update utils
ad2a9e6
@finbarrtimbers
Merge branch 'main' into finbarr/dpo-faster
daf52ee
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@finbarrtimbers @scottgeng00 @saurabh111233212