Zipformer performance suddenly drops after epoch 14

[1215thebqtic]

Hi,

My Zipformer model is trained on over 200k hours of speech data.
Between epoch 11 and 13, the training curve plateaus with no visible gains.
However, performance suddenly degrades heavily once epoch 14 begins.
This problem has never occurred in my prior training runs with identical configurations.
How should I fix and debug this abrupt performance drop?

thanks.

[danpovey]

For debugging, if you dump diagnostics from before and before after the bad stuff happens using the --print-diagnostics=True option (note, you have to turn off fp16 and reduce the batch size), you could compare the diagnostics using compare_importance.pl from
https://github.com/k2-fsa/analyze_diagnostics
which would indicate if any modules suddenly decreased or increased a lot in importance,
also compare_epochs.py from that same repo can show you how much different weight params changed in importance from epoch to epoch and you might see something unusual there if you process its output with a script to look for large changes.

I usually process the output of compare_importance.pl with something like:

~/bin/compare_importance.pl <(~/bin/param_importance.pl  diagnostics1.log) <(~/bin/param_importance.pl diagnostics2.log) | sort  -gr -k4 

and look at the head and tail of that output

However, this won't necessarily produce something actionable that you can use to recover your model's performance. What I would recommend is to get the version of the code from this PR:
#2082
and run the version of the script train_newoptim.py which uses a new optimizer that should be more robust; it needs a smaller base-LR, try 0.023.
It may be OK to train it from a checkpoint that you already trained (pick one that was OK), but in that case be careful to set --batches-per-epoch to the number of batches that are processed in 1 epoch of training with your dataset, or it will mess up the LR schedule. (The schedules are different anyway.) Because the optimizer state_dict will be incompatible you will have to modify the code to not try to pick up the state-dict from the checkpoint; there is an option in the code, something like load_optimizer_state, that should be easy enough to identify.

[1215thebqtic]

@danpovey Thank you very much for your detailed explanation! I expanded training from 3 machines to 8 machines at epoch 10. Could this change be why the model performs worse and worse with further training?

[1215thebqtic]

In addition, what kinds of possible reasons may cause this problem? and I'll be more careful next time.

[danpovey]

It's hard to say what might have happened without narrowing it down to a particular submodule. In general one might expect parameters getting too large is the cause, but that should not really happen in this recipe because it uses ScaledAdam, that is more an issue for Adam without weight decay.