Skip to content

Add gradient variance logging as a callback#89

Open
mariannearr wants to merge 2 commits intomainfrom
msa/grad_var_callback
Open

Add gradient variance logging as a callback#89
mariannearr wants to merge 2 commits intomainfrom
msa/grad_var_callback

Conversation

@mariannearr
Copy link
Collaborator

@mariannearr mariannearr commented Nov 18, 2025

  • Gradient variance computed as per Block Diffusion eq. 10, where the variance is taken over gradient vectors from accumulation_steps consecutive gradient updates
Screenshot 2025-11-18 at 5 27 52 PM
  • Gradients are collected after gradient accumulation and before the optimizer step at after_train_batch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants